CN110674108A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN110674108A CN110674108A CN201910818776.7A CN201910818776A CN110674108A CN 110674108 A CN110674108 A CN 110674108A CN 201910818776 A CN201910818776 A CN 201910818776A CN 110674108 A CN110674108 A CN 110674108A
- Authority
- CN
- China
- Prior art keywords
- data
- data table
- target
- migration
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 20
- 230000005012 migration Effects 0.000 claims abstract description 158
- 238000013508 migration Methods 0.000 claims abstract description 158
- 239000012634 fragment Substances 0.000 claims abstract description 112
- 238000004519 manufacturing process Methods 0.000 claims abstract description 57
- 238000012545 processing Methods 0.000 claims abstract description 57
- 238000003860 storage Methods 0.000 claims abstract description 32
- 238000000034 method Methods 0.000 claims description 62
- 230000008569 process Effects 0.000 claims description 33
- 238000013467 fragmentation Methods 0.000 claims description 22
- 238000006062 fragmentation reaction Methods 0.000 claims description 22
- 238000011217 control strategy Methods 0.000 claims description 7
- 230000001360 synchronised effect Effects 0.000 claims description 7
- 230000001174 ascending effect Effects 0.000 claims description 6
- 230000005059 dormancy Effects 0.000 claims description 3
- 230000006870 function Effects 0.000 description 15
- 238000004590 computer program Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 11
- 230000011218 segmentation Effects 0.000 description 9
- 238000013500 data storage Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data processing method and a data processing device, wherein the data processing method comprises the following steps: based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library; the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments; migrating the plurality of data fragments to a target data table according to a preset migration strategy; and deleting the record corresponding to the data fragment successfully migrated to the target data table in the source data table. By the embodiment of the invention, incremental data in the service data production library of the system can be efficiently transferred to the service data historical library, and sufficient storage space is vacated in time for storing subsequently generated service data in the production library, so that the system can be ensured to normally operate and has stable performance.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data processing method and device.
Background
At present, for some scenarios requiring analysis of a large amount of service data, due to system characteristics, the processed service data will not participate in calculation, but will be stored in a backup table as historical data.
Over time, the data in the backup table increases dramatically, which results in a drastic reduction in data backup efficiency, and thus may have some adverse effects on system performance. Therefore, in order to ensure that the system can normally operate and the performance is stable, the data stored in the backup table must be migrated to the historical database, so as to improve the backup efficiency of the business data. Due to the 'interruption-free' requirement of system functions, the process of data migration is required to be carried out when the system runs, but meanwhile, the structure of a backup table has a change requirement, and the table structures in a production library and a history library of business data are required to be kept synchronous.
For migration of data in the backup table, although an Extract-Transform-Load (ETL) tool used at present has a strong function, the configuration is complex, and operations such as breakpoint resume and migration progress control are not supported. In addition, there are data migration scripting tools that, while fast in processing, require system shutdown and require manual attendance. Therefore, once the table structure is changed, the tool/code needs to be rewritten, and the tables that have not completed migration need to be re-migrated.
Therefore, a data processing scheme capable of supporting dynamic table building, controllable relocation progress, breakpoint resume and no need of manual watch is needed, so that interference on normal functions of the system is reduced, and intervention of labor cost is reduced.
Disclosure of Invention
The embodiment of the invention provides a data processing method and a data processing device, aiming at efficiently transferring incremental data in a service data production library of a system to a service data historical library and vacating sufficient storage space for storing subsequently generated service data in the production library, so that the system can normally run and the performance is stable.
The embodiment of the invention adopts the following technical scheme:
in a first aspect, an embodiment of the present invention provides a data processing method, where the method includes:
based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library;
the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments;
migrating the plurality of data fragments to the target data table according to a preset migration strategy;
and deleting the record corresponding to the data fragment which is successfully migrated to the target data table in the source data table.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, where the apparatus includes:
the system comprises a grouping module, a data processing module and a data processing module, wherein the grouping module is used for grouping records to be migrated to a target data table in a source data table based on a target index field in the source data table of a service data production library, and the target data table is a backup table corresponding to the source data table in a service data historical library;
the fragmentation module is used for fragmenting the records in each group from small to large according to the increasing field sequence to obtain a plurality of data fragments;
the migration module is used for migrating the plurality of data fragments to the target data table according to a preset migration strategy;
and the deleting module is used for deleting the record corresponding to the data fragment which is successfully migrated to the target data table in the source data table.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library;
the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments;
migrating the plurality of data fragments to the target data table according to a preset migration strategy;
and deleting the record corresponding to the data fragment which is successfully migrated to the target data table in the source data table.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing one or more programs which, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform operations of:
based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library;
the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments;
migrating the plurality of data fragments to the target data table according to a preset migration strategy;
and deleting the record corresponding to the data fragment which is successfully migrated to the target data table in the source data table.
The embodiment of the invention adopts at least one technical scheme which can achieve the following technical effects:
in the embodiment of the invention, when data generated in a service data production library of a system needs to be migrated into a service data historical library for storage so as to reduce the data storage pressure of the service data production library, records in a data table to be migrated into the service data historical library in the source data table can be grouped according to a target index field in a source data table of the service data production library, and then the records in each group are fragmented from small to large according to the sequence of self-increment fields after being grouped, that is, the records in the source data table are sequentially arranged according to an ascending sequence, so that the data fragmentation of which the whole part is zero is realized through multiple times of fragmentation, the data fragmentation efficiency can be improved, and the breakpoint continuous processing in the data migration process is facilitated; furthermore, a plurality of data fragments obtained through multiple segmentation can be efficiently migrated into the business data historical library according to a preset migration strategy, dynamic division of data records to be migrated is not needed in the migration implementation process, and records corresponding to the data fragments successfully migrated into the target data table of the business data historical library are deleted from the source data table, so that sufficient storage space is vacated for storing business data subsequently generated by a business data production library of the system, and the system can be ensured to operate normally and have stable performance.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention without limiting the embodiments of the invention to the right. In the drawings:
fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a method for generating a migration configuration table according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating a method for migrating a plurality of data segments according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the present invention will be described clearly and completely with reference to the embodiments of the present invention and the accompanying drawings. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
For the incremental data migration scheme stated in the background technology, the incremental data migration scheme is very commonly applied today in large information explosion, and can effectively solve the pressure of full backup in the aspects of system storage, network transmission and manual watching cost. In the migration process of incremental data, the incremental data migration scheme based on time field segmentation has the characteristics of clear data processing logic, high speed, low cost, simple flow and the like, determines the incremental data to be backed up in a production library by recording or comparing the time difference of table data of the production library and the historical library, and then migrates the incremental data into the historical library by an extraction tool. Although the scheme can solve the problem that other incremental data migration schemes based on log files, triggers and the like need to occupy production system resources and reduce system performance, the scheme requires that a base table structure must contain time fields, so that the application range of the scheme is limited, and the scheme needs manual synchronization of a history table structure, so that the operation and maintenance cost is increased.
Therefore, a data processing scheme capable of supporting dynamic table building, controllable relocation progress, breakpoint resume and no need of manual watch is needed, so that interference on normal functions of the system is reduced, and intervention of labor cost is reduced.
The technical solutions provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.
Referring to fig. 1, an embodiment of the present invention provides a data processing method, which may include:
step 101: and grouping records to be migrated to a target data table in the source data table based on a target index field in the source data table of the service data production library, wherein the target data table is a backup table corresponding to the source data table in the service data history library.
Optionally, the source data table may be a backup table in a service data production library of the system, where the backup table is used to store service data that has undergone analysis processing and no longer participates in calculation. For example, for a source data table of a business data production library of the fund preparation system, which may include an organization code, a policy number, a policy fee, a policy date, etc., the organization code field may be selected as a target index field for implementing the data grouping.
Step 103: and (4) segmenting the records in each group from small to large according to the increasing field sequence to obtain a plurality of data segments.
Optionally, the auto-increment field is used to identify sequence numbers of the records in the source data table, auto-increment the sequence numbers, and correspond to the data records one by one.
Step 105: and migrating the plurality of data fragments to a target data table according to a preset migration strategy.
Step 107: and deleting the record corresponding to the data fragment successfully migrated to the target data table in the source data table.
In the embodiment of the invention, when data generated in a service data production library of a system needs to be migrated into a service data historical library for storage so as to reduce the data storage pressure of the service data production library, records in a data table to be migrated into the service data historical library in the source data table can be grouped according to a target index field in a source data table of the service data production library, and then the records in each group are fragmented from small to large according to the sequence of self-increment fields after being grouped, that is, the records in the source data table are sequentially arranged according to an ascending sequence, so that the data fragmentation of which the whole part is zero is realized through multiple times of fragmentation, the data fragmentation efficiency can be improved, and the breakpoint continuous processing in the data migration process is facilitated; furthermore, a plurality of data fragments obtained through multiple segmentation can be efficiently migrated into the business data historical library according to a preset migration strategy, dynamic division of data records to be migrated is not needed in the migration implementation process, and records corresponding to the data fragments successfully migrated into the target data table of the business data historical library are deleted from the source data table, so that sufficient storage space is vacated for storing business data subsequently generated by a business data production library of the system, and the system can be ensured to operate normally and have stable performance.
It should be noted that the data processing method according to the embodiment of the present invention implements data migration processing at an application level without depending on a log file or the like at a bottom layer, thereby saving system resources and improving system performance.
Optionally, the target index field for grouping records to be migrated to the target data table in the source data table may be determined by using a preconfigured basic configuration table tabconfig corresponding to the source data table, and the self-increment field may also be determined based on the basic configuration table, specifically referring to table 1 below, where the target index field corresponds to a slicing field in the configuration table, and the self-increment field corresponds to a table id, that is, a tabid, in the configuration table.
TABLE 1
Optionally, in the data processing method according to the embodiment of the present invention, in order to ensure efficiency of data migration, it is necessary to keep a table structure of a source data table of a service data production library and a table structure of a target data table of a service data history library synchronized, so as to avoid that a change of the table structures affects a progress of data migration, before step 101, the method may further include the following contents:
generating a first MD5 value according to the first table structure information of the source data table;
acquiring a second MD5 value of the target data table;
determining that the table structure of the source data table is synchronous with the table structure of the target data table under the condition that the first MD5 value is equal to the second MD5 value;
and reconstructing the table structure of the target data table according to the first table structure information under the condition that the first MD5 value is not equal to the second MD5 value.
It can be understood that before the data of the business data production library is migrated to the business data historical library, whether the table structures of the two data tables are synchronous or not can be accurately judged through whether the MD5 values capable of representing the table structures of the data tables are the same or not, so that the target data table of the business data historical library can be reconstructed in time when the data are judged to be asynchronous. Wherein, the first table structure information at least includes field composition, index, primary key, etc.; further, a corresponding table building statement Structured Query Language (SQL) may be generated according to the first table structure information, and a hash function is used to generate the SQL to generate the first MD5 value, so as to perform size comparison with a second MD5 value corresponding to a table structure of a target data table of a pre-stored business data historian, that is, only a first MD5 value corresponding to a table structure of a source data table of a business data production library needs to be generated for each comparison.
In addition, in addition to the above-mentioned method of synchronizing the table structure of the target data table of the business data historian in real time when the table structure of the source data table of the business data production library is changed during the data migration, a method of periodically reconstructing the table structure of the target data table of the business data historian may be adopted, and specifically, the following may be performed:
and under the condition that the interval duration between the current time before the record to be migrated to the target data table in the source data table is migrated to the target data table in the service data history base and the latest table building time of the target data table is longer than a preset table structure reconstruction period, reconstructing the table structure of the target data table. That is, the backup table of the business data historian is reconstructed when the table structure reconstruction period condition is satisfied, regardless of whether the table structure of the source data table of the business database is changed or not.
Optionally, for the source data table, the basic configuration table tabconfig may be cycled to obtain numbers, and the table tabname to be migrated, i.e. the structure information of the source data table, is read from the syscolumns table (for looking up all field names of the table), the sysconstraints table (for restricting the type of data added to the table), and the sysendexes table (each index in the database and each table occupies one row in the table).
Optionally, the basic configuration table tabconfig may further store an MD5 value corresponding to the latest table structure of the target data table of the service data history library, that is, the MD5 of the target data table with the changed structure is synchronously maintained in the basic configuration table in real time, and the latest table creation date and the table structure reconstruction period (i.e., the table creation time interval) may also be stored in the basic configuration table, so as to be conveniently obtained in real time when needed.
Based on the above, in other words, the above step 101 in the data processing method of the embodiment of the present invention may also be executed as:
under the condition that the table structure of a target data table of a business data history library is synchronous with the table structure of a source data table of a business data production library, records to be migrated to the target data table in the source data table are grouped based on a target index field in the source data table, and the target data table is a backup table corresponding to the source data table in the business data history library.
Optionally, in the data processing method according to the embodiment of the present invention, the step 101 may be specifically implemented as follows:
and grouping records to be migrated to the target data table corresponding to the preset data migration range in the source data table based on the target index field.
It can be understood that, for further data migration efficiency, records in the source data table may be migrated to the target data table of the data history database in batches step by step, and specifically, a range of records to be migrated in the source data table may be determined in the source data table according to a preset data migration range. Optionally, the preset data migration range may also be pre-stored in the basic configuration table shown in table 1, and corresponds to the segmentation field preselected value in the table, and in order to improve the data segmentation efficiency, the value range of the segmentation field preselected value should not be too large.
Optionally, records to be migrated to the target data table corresponding to the preset data migration range in the source data table may be stored in the temporary table for further processing.
Optionally, in the data processing method according to the embodiment of the present invention, step 103 may be specifically implemented as follows:
based on the threshold of the number of migration pieces, the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments, wherein the difference between the data sequence number of the ending position and the data sequence number of the starting position of each data segment is the threshold of the number of migration pieces.
It can be understood that after the records to be migrated in the source data table are grouped based on the target index field, for the data records in each group, the data records may be partitioned at equal intervals according to a preset threshold of the number of migrated pieces to obtain a plurality of data fragments corresponding to the records to be migrated, that is, the threshold of the number of migrated pieces is cycled, the data records are sequentially counted from small to large according to the increasing field sequence, and the data records corresponding to the threshold of the number of migrated pieces form one data fragment each time. The data sequence number corresponding to each data record is identified by the self-increment field tabid.
Alternatively, the threshold of the number of transitions may be stored in advance in the basic configuration table shown in table 1, and may correspond to the upper limit of the number of transitions in the table.
Further optionally, step 105 in the data processing method according to the embodiment of the present invention may be implemented as the following specifically:
recording the fragment information of each data fragment in the plurality of data fragments into a migration configuration table according to a preset migration strategy, wherein the fragment information of each data fragment comprises a processing sequence number used for representing a fragment migration sequence, and a starting position data sequence number and an ending position data sequence number of the data fragment;
extracting target data fragments corresponding to the initial position data sequence number and the end position data sequence number in the target fragment information from the source data table according to the processing sequence number in the target fragment information, wherein the target fragment information has an unsuccessful migration identifier;
and after the target data fragments are successfully migrated to the target data table, marking the unsuccessful migration identifier of the target fragment information as a successful migration identifier.
It can be understood that, in order to ensure that the data record to be migrated is no longer dynamically divided in the data migration implementation process, a migration configuration table is generated according to the fragment information of the multiple data fragments corresponding to the data record to be migrated in the source data table, specifically, the processing serial number, the starting position data serial number, and the ending position data serial number of each data fragment may be sequentially stored in the migration configuration table, that is, an adaptive migration configuration table is dynamically generated in the data migration processing process according to the specific situation of the source data table to be migrated, wherein the storage order of the fragment information of the multiple data fragments in the migration configuration table may refer to the order of the processing serial numbers and the order of the data serial numbers of the data fragments for limiting the migration operation order of the data fragments in the data migration implementation process, and ensuring that the data migration process runs in order, the migration sequence may be specifically set to migrate the processing sequence number smaller and migrate the data sequence number smaller, that is, preferentially migrate the data record arranged in the front of the source data table.
Optionally, the process of obtaining the migration configuration table from the data in the source data table and the concrete expression form of the migration configuration table may refer to fig. 2, where a starting position data sequence number (i.e., the minimum tab) of a next data fragment is (an ending position data sequence number +1) (i.e., the maximum tab +1) of a previous data fragment adjacent to the next data fragment, until all data records in each group are obtained through grouping corresponding to a preset data migration range.
Further, when entering a data migration implementation process, the fragment information of the first data fragment which is not successfully migrated into the target data table, that is, the target fragment information marked with the unsuccessful migration identifier, may be read from the migration configuration table in sequence all the time, and the number of the target fragment information is one or more, and is used to refer to the unprocessed fragment information in the migration configuration table, further directly and sequentially extract the corresponding data records from the source data table according to the start position information and the end position information, and then migrate the data records extracted to the data fragment into the target data table of the service data history for storage, and do not perform dynamic partitioning by real-time calculation any more.
Alternatively, the processing sequence number may be stored in the basic configuration table shown in table 1.
As can be seen from the above, a migration configuration table is dynamically generated according to a plurality of data fragments obtained by grouping first and then grouping second and sequential migration is realized according to the migration configuration table, and an unsuccessful migration identifier and a successful migration identifier can be respectively marked on a successfully migrated data fragment and a data fragment to be migrated in the migration configuration table, so that accurate and non-missing breakpoint processing can be realized, the situation that data to be migrated is missing or migration confusion occurs and data that has been successfully migrated and has not been migrated can not be distinguished is avoided, and it is ensured that data records in a source data table can be orderly migrated to a service data history library as required.
Optionally, in the data processing method of the embodiment of the present invention, the data migration progress may also be controlled without stopping the system, and the data migration progress is efficiently controlled without affecting the production efficiency of the system, which specifically includes the following contents:
if the data migration progress control strategy is detected to start running, interrupting the process of migrating the plurality of data slices to the target data table according to the preset migration strategy; and
and if the data migration progress control strategy is detected to stop running, forbidding restarting the process of migrating data from the source data table to the target data table within the preset dormancy duration.
It can be understood that the data migration function is enabled or disabled at any time according to the migration progress control function provided by the system setting control system, specifically, when the data migration progress control policy is enabled, the system can kill the thread, and the soft interrupt exits, and when the data migration implementation process is re-entered next time, the migration is started from the next interrupted data fragment. And if the data migration function is stopped by stopping the data migration progress control strategy, the data migration function needs to be dormant for a period of time, and the data migration process can be restarted after the RSS synchronization of the database is completed, so that the high consistency of the data in the service data production library and the service data historical library is ensured.
Optionally, in the data processing method according to the embodiment of the present invention, the function of lossless migration may also be performed to ensure that data in the source data table of the service data production library is completely migrated to the target data table of the service data history library, which may specifically include the following contents:
counting the total migration number of the data fragments migrated from the source data table to the target data table according to the migration configuration table;
judging whether the total migration number is equal to the total fragment number of the plurality of data fragments;
if so, determining that the data migration from the source data table to the target data table is successful;
and if not, determining that the data migration from the source data table to the target data table fails.
That is, whether the current data migration is successful or not can be determined according to whether the total numbers of the data fragments before and after the data migration are equal or not by respectively counting the total numbers of the data fragments before and after the data migration.
The processes of breakpoint processing, lossless migration and controllable data migration progress based on data fragmentation realized by the data processing method of the embodiment of the present invention can be specifically referred to fig. 3. This embodiment may include 3 parts: table structure synchronization, data slicing, data migration. The data migration speed can be controlled at any time through table structure dynamic synchronization, data to be migrated slice division, one-by-one migration according to the slice sequence, breakpoint continuous moving support, migration progress control and the like, and the goal that the business system and other systems are not affected is ensured.
The table structure synchronization comprises the table structure synchronization between the production database and the backup database and the regular reconstruction of the table structure, so that the consistency of the table structure between the two databases and the data migration efficiency are ensured; data statistics is carried out on data slices, a migration operation sequence is generated, the data to be migrated are guaranteed not to be dynamically divided any more in the data migration implementation process, and the data migration operation sequence is sequentially executed according to the operation sequence; the data migration implementation comprises lossless migration, breakpoint processing, migration progress control and the like, so that the orderly and stable operation of the migration process is ensured, and the production efficiency is not influenced.
An embodiment of the present invention further provides a data processing apparatus, and as shown in fig. 4, the apparatus 400 may specifically include:
a grouping module 401, configured to group records to be migrated to a target data table in a source data table of a service data production library based on a target index field in the source data table, where the target data table is a backup table corresponding to the source data table in a service data history library;
the fragmentation module 403 is configured to fragment records in each group from small to large according to the increasing field sequence, to obtain a plurality of data fragments;
a migration module 405, configured to migrate the multiple data segments to the target data table according to a preset migration policy;
a deleting module 407, configured to delete a record in the source data table corresponding to the data fragment that has been successfully migrated into the target data table.
Optionally, in the data processing apparatus 400 according to the embodiment of the present invention, the fragmentation module 403 may be specifically configured to:
and based on the threshold of the number of migration pieces, the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain the plurality of data segments, wherein the difference between the data sequence number of the ending position and the data sequence number of the starting position of each data segment is the threshold of the number of migration pieces.
Optionally, in the data processing apparatus 400 according to the embodiment of the present invention, the migration module 405 may be specifically configured to:
recording the fragment information of each data fragment in the plurality of data fragments into a migration configuration table according to the preset migration strategy, wherein the fragment information of each data fragment comprises a processing sequence number used for representing a fragment migration sequence, and a start position data sequence number and an end position data sequence number of the data fragment;
extracting target data fragments corresponding to the initial position data sequence number and the end position data sequence number in the target fragment information from the source data table according to the processing sequence number in the target fragment information, wherein the target fragment information has an unsuccessful migration identifier;
and after the target data fragments are successfully migrated into the target data table, marking the unsuccessful migration identifier of the target fragment information as a successful migration identifier.
Optionally, the data processing apparatus 400 according to the embodiment of the present invention may further include a control module, where the control module may be specifically configured to:
if the data migration progress control strategy is detected to start running, interrupting the process of migrating the plurality of data slices to the target data table according to a preset migration strategy; and
and if the data migration progress control strategy is detected to stop running, forbidding restarting the process of migrating data from the source data table to the target data table within a preset dormancy duration.
Optionally, the data processing apparatus 400 according to the embodiment of the present invention may further include:
the counting module is used for counting the total migration number of the data fragments migrated from the source data table to the target data table according to the migration configuration table;
the judging module is used for judging whether the total migration number is equal to the total fragment number of the plurality of data fragments;
the first determining module is used for determining that the data migration from the source data table to the target data table is successful under the condition that the total migration number is equal to the total fragmentation number;
and the second determining module is used for determining that the data migration from the source data table to the target data table fails under the condition that the total migration number is not equal to the total fragmentation number.
Optionally, in the data processing apparatus 400 according to the embodiment of the present invention, the grouping module 401 may be specifically configured to:
and grouping the records to be migrated to the target data table corresponding to the preset data migration range in the source data table based on the target index field.
Optionally, the data processing apparatus 400 according to the embodiment of the present invention may further include:
a generating module, configured to generate a first MD5 value according to first table structure information of a source data table before a target index field in the source data table based on the service data production library groups records to be migrated to a target data table in the source data table;
the acquisition module is used for acquiring a second MD5 value of the target data table;
a third determination module to determine that the table structure of the source data table is synchronized with the table structure of the target data table if the first MD5 value and the second MD5 value are equal;
a reconstruction module for reconstructing a table structure of the target data table according to the first table structure information if the first MD5 value is not equal to the second MD5 value.
It can be understood that the data processing apparatus provided in the embodiment of the present invention can implement the data processing method provided in the foregoing embodiment, and the related explanations about the data processing method are applicable to the data processing apparatus, and are not described herein again.
In the embodiment of the invention, when data generated in a service data production library of a system needs to be migrated into a service data historical library for storage so as to reduce the data storage pressure of the service data production library, records in a data table to be migrated into the service data historical library in the source data table can be grouped according to a target index field in a source data table of the service data production library, and then the records in each group are fragmented from small to large according to the sequence of self-increment fields after being grouped, that is, the records in the source data table are sequentially arranged according to an ascending sequence, so that the data fragmentation of which the whole part is zero is realized through multiple times of fragmentation, the data fragmentation efficiency can be improved, and the breakpoint continuous processing in the data migration process is facilitated; furthermore, a plurality of data fragments obtained through multiple segmentation can be efficiently migrated into the business data historical library according to a preset migration strategy, dynamic division of data records to be migrated is not needed in the migration implementation process, and records corresponding to the data fragments successfully migrated into the target data table of the business data historical library are deleted from the source data table, so that sufficient storage space is vacated for storing business data subsequently generated by a business data production library of the system, and the system can be ensured to operate normally and have stable performance.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. Referring to fig. 5, at a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (peripheral component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 5, but this does not indicate only one bus or one type of bus.
And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.
The processor reads a corresponding computer program from the non-volatile memory into the memory and then runs the computer program, thereby forming the data processing device on a logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:
based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library;
the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments;
migrating the plurality of data fragments to a target data table according to a preset migration strategy;
and deleting the record corresponding to the data fragment successfully migrated to the target data table in the source data table.
In the embodiment of the invention, when data generated in a service data production library of a system needs to be migrated into a service data historical library for storage so as to reduce the data storage pressure of the service data production library, records in a data table to be migrated into the service data historical library in the source data table can be grouped according to a target index field in a source data table of the service data production library, and then the records in each group are fragmented from small to large according to the sequence of self-increment fields after being grouped, that is, the records in the source data table are sequentially arranged according to an ascending sequence, so that the data fragmentation of which the whole part is zero is realized through multiple times of fragmentation, the data fragmentation efficiency can be improved, and the breakpoint continuous processing in the data migration process is facilitated; furthermore, a plurality of data fragments obtained through multiple segmentation can be efficiently migrated into the business data historical library according to a preset migration strategy, dynamic division of data records to be migrated is not needed in the migration implementation process, and records corresponding to the data fragments successfully migrated into the target data table of the business data historical library are deleted from the source data table, so that sufficient storage space is vacated for storing business data subsequently generated by a business data production library of the system, and the system can be ensured to operate normally and have stable performance.
The method performed by the data processing apparatus according to the embodiment of the invention shown in fig. 1 may be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
The electronic device may further execute the method executed by the data processing apparatus in fig. 1, and implement the functions of the data processing apparatus in the embodiment shown in fig. 1, which are not described herein again in this embodiment of the present invention.
An embodiment of the present invention further provides a computer-readable storage medium, which stores one or more programs, where the one or more programs include instructions, which, when executed by an electronic device including a plurality of application programs, enable the electronic device to perform the method performed by the data processing apparatus in the embodiment shown in fig. 1, and are specifically configured to perform:
based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library;
the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments;
migrating the plurality of data fragments to a target data table according to a preset migration strategy;
and deleting the record corresponding to the data fragment successfully migrated to the target data table in the source data table.
In the embodiment of the invention, when data generated in a service data production library of a system needs to be migrated into a service data historical library for storage so as to reduce the data storage pressure of the service data production library, records in a data table to be migrated into the service data historical library in the source data table can be grouped according to a target index field in a source data table of the service data production library, and then the records in each group are fragmented from small to large according to the sequence of self-increment fields after being grouped, that is, the records in the source data table are sequentially arranged according to an ascending sequence, so that the data fragmentation of which the whole part is zero is realized through multiple times of fragmentation, the data fragmentation efficiency can be improved, and the breakpoint continuous processing in the data migration process is facilitated; furthermore, a plurality of data fragments obtained through multiple segmentation can be efficiently migrated into the business data historical library according to a preset migration strategy, dynamic division of data records to be migrated is not needed in the migration implementation process, and records corresponding to the data fragments successfully migrated into the target data table of the business data historical library are deleted from the source data table, so that sufficient storage space is vacated for storing business data subsequently generated by a business data production library of the system, and the system can be ensured to operate normally and have stable performance.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above description is only an example of the present invention, and is not intended to limit the present invention. Various modifications and alterations to the embodiments of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present invention should be included in the scope of claims of the embodiments of the present invention.
Claims (10)
1. A method of data processing, the method comprising:
based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library;
the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments;
migrating the plurality of data fragments to the target data table according to a preset migration strategy;
and deleting the record corresponding to the data fragment which is successfully migrated to the target data table in the source data table.
2. The method of claim 1, wherein the fragmenting the records in each group from smaller to larger in ascending field order to obtain a plurality of data fragments comprises:
and based on the threshold of the number of migration pieces, the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain the plurality of data segments, wherein the difference between the data sequence number of the ending position and the data sequence number of the starting position of each data segment is the threshold of the number of migration pieces.
3. The method according to claim 2, wherein migrating the plurality of data slices to the target data table according to a preset migration policy includes:
recording the fragment information of each data fragment in the plurality of data fragments into a migration configuration table according to the preset migration strategy, wherein the fragment information of each data fragment comprises a processing sequence number used for representing a fragment migration sequence, and a start position data sequence number and an end position data sequence number of the data fragment;
extracting target data fragments corresponding to the initial position data sequence number and the end position data sequence number in the target fragment information from the source data table according to the processing sequence number in the target fragment information, wherein the target fragment information has an unsuccessful migration identifier;
and after the target data fragments are successfully migrated into the target data table, marking the unsuccessful migration identifier of the target fragment information as a successful migration identifier.
4. The method of claim 3, further comprising:
if the data migration progress control strategy is detected to start running, interrupting the process of migrating the plurality of data slices to the target data table according to a preset migration strategy; and
and if the data migration progress control strategy is detected to stop running, forbidding restarting the process of migrating data from the source data table to the target data table within a preset dormancy duration.
5. The method of claim 3, further comprising:
counting the total migration number of the data fragments migrated from the source data table to the target data table according to the migration configuration table;
judging whether the total migration number is equal to the total fragment number of the plurality of data fragments;
if so, determining that the data migration from the source data table to the target data table is successful;
and if not, determining that the data migration from the source data table to the target data table fails.
6. The method according to claim 1, wherein the grouping records in the source data table to be migrated into the target data table based on the target index field in the source data table of the service data production library comprises:
and grouping the records to be migrated to the target data table corresponding to the preset data migration range in the source data table based on the target index field.
7. The method of claim 1, wherein before grouping the records in the source data table to be migrated into the target data table in the target index field in the source data table based on the business data production library, the method further comprises:
generating a first MD5 value according to the first table structure information of the source data table;
acquiring a second MD5 value of the target data table;
determining that a table structure of the source data table is synchronized with a table structure of the target data table if the first MD5 value and the second MD5 value are equal;
reconstructing a table structure of the target data table from the first table structure information if the first MD5 value is not equal to the second MD5 value.
8. A data processing apparatus, characterized in that the apparatus comprises:
the system comprises a grouping module, a data processing module and a data processing module, wherein the grouping module is used for grouping records to be migrated to a target data table in a source data table based on a target index field in the source data table of a service data production library, and the target data table is a backup table corresponding to the source data table in a service data historical library;
the fragmentation module is used for fragmenting the records in each group from small to large according to the increasing field sequence to obtain a plurality of data fragments;
the migration module is used for migrating the plurality of data fragments to the target data table according to a preset migration strategy;
and the deleting module is used for deleting the record corresponding to the data fragment which is successfully migrated to the target data table in the source data table.
9. An electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions that, when executed, cause the processor to:
based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library;
the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments;
migrating the plurality of data fragments to the target data table according to a preset migration strategy;
and deleting the record corresponding to the data fragment which is successfully migrated to the target data table in the source data table.
10. A computer-readable storage medium storing one or more programs that, when executed by an electronic device including a plurality of application programs, cause the electronic device to:
based on a target index field in a source data table of a service data production library, grouping records to be migrated to a target data table in the source data table, wherein the target data table is a backup table corresponding to the source data table in a service data history library;
the records in each group are respectively segmented from small to large according to the increasing field sequence to obtain a plurality of data segments;
migrating the plurality of data fragments to the target data table according to a preset migration strategy;
and deleting the record corresponding to the data fragment which is successfully migrated to the target data table in the source data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910818776.7A CN110674108A (en) | 2019-08-30 | 2019-08-30 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910818776.7A CN110674108A (en) | 2019-08-30 | 2019-08-30 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110674108A true CN110674108A (en) | 2020-01-10 |
Family
ID=69075822
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910818776.7A Pending CN110674108A (en) | 2019-08-30 | 2019-08-30 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110674108A (en) |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111831752A (en) * | 2020-07-20 | 2020-10-27 | 北京百度网讯科技有限公司 | Distributed database space arrangement method, device, equipment and storage medium |
| CN112181995A (en) * | 2020-11-06 | 2021-01-05 | 深圳视界信息技术有限公司 | Data processing method, device and equipment of data table and storage medium |
| CN112199355A (en) * | 2020-12-09 | 2021-01-08 | 北京易真学思教育科技有限公司 | Data migration method and device, electronic equipment and storage medium |
| CN112256666A (en) * | 2020-09-08 | 2021-01-22 | 山东黄金矿业(莱州)有限公司三山岛金矿 | Logic increment migration method |
| CN112286905A (en) * | 2020-10-15 | 2021-01-29 | 北京沃东天骏信息技术有限公司 | Data migration method and device, storage medium and electronic equipment |
| CN112434013A (en) * | 2020-11-30 | 2021-03-02 | 中国建设银行股份有限公司 | Data table migration method and device, electronic equipment and storage medium |
| CN112783900A (en) * | 2021-02-25 | 2021-05-11 | 百果园技术(新加坡)有限公司 | Database management method, device, equipment and storage medium |
| CN112905386A (en) * | 2021-02-08 | 2021-06-04 | 中国工商银行股份有限公司 | Table data backup cleaning method and device based on life cycle |
| CN112925743A (en) * | 2021-02-07 | 2021-06-08 | 中国工商银行股份有限公司 | File generation method and device and storage medium |
| CN113157670A (en) * | 2021-05-10 | 2021-07-23 | 中国工商银行股份有限公司 | Data migration method and system of database |
| CN113190534A (en) * | 2021-05-26 | 2021-07-30 | 中国工商银行股份有限公司 | Database data migration method and device |
| CN113220783A (en) * | 2021-05-07 | 2021-08-06 | 深圳市粤睦信息科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113220660A (en) * | 2021-04-15 | 2021-08-06 | 远景智能国际私人投资有限公司 | Data migration method, device and equipment and readable storage medium |
| CN113297182A (en) * | 2021-06-16 | 2021-08-24 | 中国农业银行股份有限公司 | Data migration method, device, storage medium and program product |
| CN113570392A (en) * | 2020-04-29 | 2021-10-29 | 中移动信息技术有限公司 | User grouping method and device, electronic equipment and computer storage medium |
| CN113656406A (en) * | 2021-08-17 | 2021-11-16 | 北京金山云网络技术有限公司 | Method, device, device and medium for index page merging |
| CN113704228A (en) * | 2021-08-31 | 2021-11-26 | 北京达佳互联信息技术有限公司 | Method and device for field self-increment, electronic equipment and storage medium |
| CN113760858A (en) * | 2020-06-05 | 2021-12-07 | 中国移动通信集团湖北有限公司 | Memory library data dynamic migration method, device, computing device and storage device |
| CN113791736A (en) * | 2021-09-15 | 2021-12-14 | 京东科技信息技术有限公司 | Data migration method, network card device, server and data migration system |
| CN113849476A (en) * | 2020-06-28 | 2021-12-28 | 中兴通讯股份有限公司 | Data migration method and device, network equipment and storage medium |
| CN114064693A (en) * | 2021-11-17 | 2022-02-18 | 中国建设银行股份有限公司 | Method, apparatus, electronic device, and computer-readable medium for processing account data |
| CN114185891A (en) * | 2021-12-10 | 2022-03-15 | 建信金融科技有限责任公司 | A data table fragmentation method, device, equipment and medium |
| CN114490583A (en) * | 2022-01-29 | 2022-05-13 | 中国建设银行股份有限公司 | Data migration method, apparatus, electronic device and storage medium |
| CN114676141A (en) * | 2022-03-31 | 2022-06-28 | 北京泰迪熊移动科技有限公司 | Data processing method and device and electronic equipment |
| CN115827788A (en) * | 2023-02-16 | 2023-03-21 | 天翼云科技有限公司 | Data migration method and device, electronic equipment and readable storage medium |
| CN116069765A (en) * | 2023-03-06 | 2023-05-05 | 天翼云科技有限公司 | Data migration method, device, electronic equipment and storage medium |
| CN116795664A (en) * | 2023-08-25 | 2023-09-22 | 四川省农村信用社联合社 | Automatic processing full-increment historical data storage method |
| CN117009147A (en) * | 2023-09-28 | 2023-11-07 | 新华三技术有限公司 | Data backup method and device of cloud platform virtual machine and electronic equipment |
| CN117931767A (en) * | 2023-12-15 | 2024-04-26 | 北京宇信科技集团股份有限公司 | Database migration platform and database migration method |
| CN118585513A (en) * | 2024-08-07 | 2024-09-03 | 天翼物联科技有限公司 | Data migration method, device, computer equipment and storage medium |
| CN118677950A (en) * | 2024-08-22 | 2024-09-20 | 浙江大华技术股份有限公司 | Migration method, device and equipment of data transmission task and storage medium |
| WO2024244422A1 (en) * | 2023-05-31 | 2024-12-05 | 中国银行股份有限公司 | Data migration method and apparatus based on unitized architecture, and electronic device |
| CN119767007A (en) * | 2025-03-05 | 2025-04-04 | 山东云海国创云计算装备产业创新中心有限公司 | Video compression method, device, medium and product |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103488687A (en) * | 2013-09-02 | 2014-01-01 | 用友软件股份有限公司 | Searching system and searching method of big data |
| CN105183371A (en) * | 2015-08-14 | 2015-12-23 | 山东大学 | Migration balancing policy based electricity-consuming information distributed file storage method and apparatus |
| CN105574217A (en) * | 2016-03-16 | 2016-05-11 | 中国联合网络通信集团有限公司 | Data synchronization method and device of distributed relational database |
| CN107515874A (en) * | 2016-06-16 | 2017-12-26 | 阿里巴巴集团控股有限公司 | The method and apparatus of synchronous incremental data in a kind of distributed non-relational database |
| CN108304553A (en) * | 2018-02-01 | 2018-07-20 | 平安普惠企业管理有限公司 | Data migration method, device, computer equipment and storage medium |
| CN109683826A (en) * | 2018-12-26 | 2019-04-26 | 北京百度网讯科技有限公司 | Expansion method and device for distributed memory system |
-
2019
- 2019-08-30 CN CN201910818776.7A patent/CN110674108A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103488687A (en) * | 2013-09-02 | 2014-01-01 | 用友软件股份有限公司 | Searching system and searching method of big data |
| CN105183371A (en) * | 2015-08-14 | 2015-12-23 | 山东大学 | Migration balancing policy based electricity-consuming information distributed file storage method and apparatus |
| CN105574217A (en) * | 2016-03-16 | 2016-05-11 | 中国联合网络通信集团有限公司 | Data synchronization method and device of distributed relational database |
| CN107515874A (en) * | 2016-06-16 | 2017-12-26 | 阿里巴巴集团控股有限公司 | The method and apparatus of synchronous incremental data in a kind of distributed non-relational database |
| CN108304553A (en) * | 2018-02-01 | 2018-07-20 | 平安普惠企业管理有限公司 | Data migration method, device, computer equipment and storage medium |
| CN109683826A (en) * | 2018-12-26 | 2019-04-26 | 北京百度网讯科技有限公司 | Expansion method and device for distributed memory system |
Non-Patent Citations (2)
| Title |
|---|
| 曹建军 等: "《数据质量导论》", 31 October 2017, 国防工业出版社 * |
| 曾喆: "PDM数据向非关系数据库迁移技术探究", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 * |
Cited By (44)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113570392B (en) * | 2020-04-29 | 2024-04-09 | 中移动信息技术有限公司 | User grouping method, device, electronic equipment and computer storage medium |
| CN113570392A (en) * | 2020-04-29 | 2021-10-29 | 中移动信息技术有限公司 | User grouping method and device, electronic equipment and computer storage medium |
| CN113760858B (en) * | 2020-06-05 | 2024-03-19 | 中国移动通信集团湖北有限公司 | Dynamic migration method and device for memory database data, computing equipment and storage equipment |
| CN113760858A (en) * | 2020-06-05 | 2021-12-07 | 中国移动通信集团湖北有限公司 | Memory library data dynamic migration method, device, computing device and storage device |
| CN113849476A (en) * | 2020-06-28 | 2021-12-28 | 中兴通讯股份有限公司 | Data migration method and device, network equipment and storage medium |
| CN111831752B (en) * | 2020-07-20 | 2024-06-04 | 北京百度网讯科技有限公司 | Spatial arrangement method, device and equipment of distributed database and storage medium |
| CN111831752A (en) * | 2020-07-20 | 2020-10-27 | 北京百度网讯科技有限公司 | Distributed database space arrangement method, device, equipment and storage medium |
| CN112256666B (en) * | 2020-09-08 | 2023-06-16 | 山东黄金矿业(莱州)有限公司三山岛金矿 | Logic increment migration method |
| CN112256666A (en) * | 2020-09-08 | 2021-01-22 | 山东黄金矿业(莱州)有限公司三山岛金矿 | Logic increment migration method |
| CN112286905A (en) * | 2020-10-15 | 2021-01-29 | 北京沃东天骏信息技术有限公司 | Data migration method and device, storage medium and electronic equipment |
| CN112181995B (en) * | 2020-11-06 | 2023-08-29 | 深圳数阔信息技术有限公司 | Data processing method, device, equipment and storage medium of data table |
| CN112181995A (en) * | 2020-11-06 | 2021-01-05 | 深圳视界信息技术有限公司 | Data processing method, device and equipment of data table and storage medium |
| CN112434013A (en) * | 2020-11-30 | 2021-03-02 | 中国建设银行股份有限公司 | Data table migration method and device, electronic equipment and storage medium |
| CN112199355A (en) * | 2020-12-09 | 2021-01-08 | 北京易真学思教育科技有限公司 | Data migration method and device, electronic equipment and storage medium |
| CN112925743A (en) * | 2021-02-07 | 2021-06-08 | 中国工商银行股份有限公司 | File generation method and device and storage medium |
| CN112905386A (en) * | 2021-02-08 | 2021-06-04 | 中国工商银行股份有限公司 | Table data backup cleaning method and device based on life cycle |
| CN112783900A (en) * | 2021-02-25 | 2021-05-11 | 百果园技术(新加坡)有限公司 | Database management method, device, equipment and storage medium |
| CN113220660A (en) * | 2021-04-15 | 2021-08-06 | 远景智能国际私人投资有限公司 | Data migration method, device and equipment and readable storage medium |
| CN113220783B (en) * | 2021-05-07 | 2024-03-26 | 深圳市粤睦信息科技有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN113220783A (en) * | 2021-05-07 | 2021-08-06 | 深圳市粤睦信息科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113157670B (en) * | 2021-05-10 | 2024-02-09 | 中国工商银行股份有限公司 | Data migration method and system of database |
| CN113157670A (en) * | 2021-05-10 | 2021-07-23 | 中国工商银行股份有限公司 | Data migration method and system of database |
| CN113190534A (en) * | 2021-05-26 | 2021-07-30 | 中国工商银行股份有限公司 | Database data migration method and device |
| CN113297182B (en) * | 2021-06-16 | 2024-01-30 | 中国农业银行股份有限公司 | Data migration method, device, storage medium and program product |
| CN113297182A (en) * | 2021-06-16 | 2021-08-24 | 中国农业银行股份有限公司 | Data migration method, device, storage medium and program product |
| CN113656406A (en) * | 2021-08-17 | 2021-11-16 | 北京金山云网络技术有限公司 | Method, device, device and medium for index page merging |
| CN113704228B (en) * | 2021-08-31 | 2024-02-06 | 北京达佳互联信息技术有限公司 | Method, device, electronic equipment and storage medium for field self-addition |
| CN113704228A (en) * | 2021-08-31 | 2021-11-26 | 北京达佳互联信息技术有限公司 | Method and device for field self-increment, electronic equipment and storage medium |
| CN113791736A (en) * | 2021-09-15 | 2021-12-14 | 京东科技信息技术有限公司 | Data migration method, network card device, server and data migration system |
| CN114064693A (en) * | 2021-11-17 | 2022-02-18 | 中国建设银行股份有限公司 | Method, apparatus, electronic device, and computer-readable medium for processing account data |
| CN114185891A (en) * | 2021-12-10 | 2022-03-15 | 建信金融科技有限责任公司 | A data table fragmentation method, device, equipment and medium |
| CN114185891B (en) * | 2021-12-10 | 2024-12-20 | 建信金融科技有限责任公司 | A data table sharding method, device, equipment and medium |
| CN114490583A (en) * | 2022-01-29 | 2022-05-13 | 中国建设银行股份有限公司 | Data migration method, apparatus, electronic device and storage medium |
| CN114676141A (en) * | 2022-03-31 | 2022-06-28 | 北京泰迪熊移动科技有限公司 | Data processing method and device and electronic equipment |
| CN115827788A (en) * | 2023-02-16 | 2023-03-21 | 天翼云科技有限公司 | Data migration method and device, electronic equipment and readable storage medium |
| CN116069765A (en) * | 2023-03-06 | 2023-05-05 | 天翼云科技有限公司 | Data migration method, device, electronic equipment and storage medium |
| WO2024244422A1 (en) * | 2023-05-31 | 2024-12-05 | 中国银行股份有限公司 | Data migration method and apparatus based on unitized architecture, and electronic device |
| CN116795664B (en) * | 2023-08-25 | 2023-10-31 | 四川省农村信用社联合社 | Automatic processing full-increment historical data storage method |
| CN116795664A (en) * | 2023-08-25 | 2023-09-22 | 四川省农村信用社联合社 | Automatic processing full-increment historical data storage method |
| CN117009147A (en) * | 2023-09-28 | 2023-11-07 | 新华三技术有限公司 | Data backup method and device of cloud platform virtual machine and electronic equipment |
| CN117931767A (en) * | 2023-12-15 | 2024-04-26 | 北京宇信科技集团股份有限公司 | Database migration platform and database migration method |
| CN118585513A (en) * | 2024-08-07 | 2024-09-03 | 天翼物联科技有限公司 | Data migration method, device, computer equipment and storage medium |
| CN118677950A (en) * | 2024-08-22 | 2024-09-20 | 浙江大华技术股份有限公司 | Migration method, device and equipment of data transmission task and storage medium |
| CN119767007A (en) * | 2025-03-05 | 2025-04-04 | 山东云海国创云计算装备产业创新中心有限公司 | Video compression method, device, medium and product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110674108A (en) | Data processing method and device | |
| CN106933823B (en) | Data synchronization method and device | |
| CN106970921B (en) | Data migration method and device | |
| CN110674213A (en) | Data synchronization method and device | |
| CN106897342B (en) | Data verification method and equipment | |
| CN111143331B (en) | Data migration method, device and computer storage medium | |
| CN106777126B (en) | Data online migration method supporting heterogeneous time sequence database | |
| WO2023231281A1 (en) | Data processing method and apparatus, and electronic device | |
| CN113542422A (en) | Data storage method, device, storage medium and electronic device | |
| CN110019932A (en) | The method and device of data processing | |
| CN111708763B (en) | Data migration method and device of sliced cluster and sliced cluster system | |
| CN106648839B (en) | Data processing method and device | |
| CN107153680B (en) | Method and system for on-line node expansion of distributed memory database | |
| CN110647463B (en) | Method and device for restoring test breakpoint and electronic equipment | |
| CN113360479B (en) | Data migration method, device, computer equipment and storage medium | |
| CN111459913A (en) | Capacity expansion method and device of distributed database and electronic equipment | |
| CN105975567B (en) | Method and device for processing internal file of application program | |
| CN112015798B (en) | Data processing method and device for guaranteeing data non-repetition and delay monitoring | |
| CN109446271B (en) | Data synchronization method, device, equipment and medium | |
| CN112015821A (en) | Database synchronization method, terminal and storage medium | |
| CN110019295B (en) | Database retrieval method, device, system and storage medium | |
| CN117076204A (en) | A remote copy task recovery method, device, equipment and media | |
| CN117251510A (en) | Data synchronization method and device | |
| CN111427871A (en) | Data processing method, device and equipment | |
| CN116737246A (en) | Task restarting method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200110 |
|
| RJ01 | Rejection of invention patent application after publication |