CN116955369A - Database-based data fusion method, device, electronic equipment and storage medium - Google Patents

Abstract

This application relates to the field of data processing technology and provides a database-based data fusion method, device, electronic device and storage medium. The method includes: synchronizing the data to be fused in the source database to the intermediate database based on the data fusion request; A fusion operation is performed to fuse the data to be fused in the intermediate database to the target database; the intermediate database is used to represent the attribute information of the data to be fused and the attribute information of the database table in the target database that fuses the data to be fused. This application implements related processing operations for the data to be fused by using an intermediate database as a transfer cache area, ensuring that the actual production of the source database and target database is not affected, and also supports data fusion of multiple non-homogeneous databases, maintaining Simultaneous accessibility of source and target databases enables large-scale complex data fusion to be achieved without affecting actual production.

Description

Database-based data fusion method, device, electronic equipment and storage medium

Technical field

The present application relates to the field of data processing technology, and in particular to a database-based data fusion method, device, electronic equipment and storage medium.

Background technique

At present, most databases have corresponding data backup and replication mechanisms, but they usually support data backup and replication in the form of export and import files to solve problems such as data backup, disaster recovery, and recovery. Therefore, it is increasingly necessary to transfer data between databases. more common.

Among related technologies, GoldenGate software can provide data disaster recovery and data replication functions, support zero-downtime database upgrades and migrations, and achieve sub-second real-time data synchronization. GoldenGate software is usually used to synchronize data between the same source database or the same business. The regional database performs data replication, thereby merging some database tables with similar names and contents in the source and target ends.

However, when the names and structures of the database tables in the source and target ends are different, the database tables need to be processed at the source end, which affects actual production and is not suitable for large-scale complex data fusion.

Contents of the invention

This application provides a database-based data fusion method, device, electronic equipment and storage medium to solve the problem in the existing technology of needing to process database tables at the source end, which affects actual production and is not suitable for large-scale complex data fusion. Defects, to achieve the purpose of being suitable for large-scale complex data fusion without affecting actual production.

This application provides a database-based data fusion method, including:

Based on the data fusion request, synchronize the data to be fused in the source database to the intermediate database;

Execute a fusion operation for merging the data to be fused in the intermediate database to the target database;

Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused.

According to a database-based data fusion method provided by this application, the execution of a fusion operation for merging the data to be fused in the intermediate database to the target database includes:

When it is determined that the target database does not have the data to be fused, the data to be fused in the intermediate database is synchronized to the target database.

According to a database-based data fusion method provided by this application, when the number of the target databases is multiple, the fusion operation is performed to fuse the data to be fused in the intermediate database to the target database, include:

When it is determined that the target database does not have the data to be fused, synchronize the data to be fused to the temporary source database of the intermediate database and perform a split operation to obtain multiple data to be fused;

Transfer the plurality of data to be fused to the temporary target database of the intermediate database;

Perform a fusion operation for each data to be fused in the temporary target database into the corresponding target database.

According to a database-based data fusion method provided by this application, the execution of a fusion operation for merging the data to be fused in the intermediate database to the target database includes:

When it is determined that there is a first database table corresponding to the data to be fused in the target database, based on the difference between the data to be fused and the first database table, perform data conversion for the data to be fused in the intermediate database operate;

Synchronize the target data to be fused obtained by the data conversion operation to the first database table of the target database.

According to a database-based data fusion method provided by this application, when the number of data to be fused is multiple and there is a preset relationship between the multiple data to be fused, the execution is directed to the data in the intermediate database. The fusion operation of merging the data to be fused to the target database includes:

When it is determined that there is a second database table corresponding to multiple data to be fused in the target database, perform a merge operation on multiple data to be fused in the intermediate database to obtain an intermediate database table;

Based on the difference between the intermediate database table and the second database table, perform a data conversion operation on the intermediate database table in the intermediate database;

Synchronize the target data to be fused obtained by the data conversion operation to the second database table of the target database.

According to a database-based data fusion method provided by this application, the execution of a fusion operation for merging the data to be fused in the intermediate database to the target database includes:

When the determination trigger collects change data that represents changes in the database table data in the source database, it triggers the generation of an incremental data fusion request;

Acquire the incremental data to be fused corresponding to the changed data, and perform an incremental fusion operation for merging the incremental data to be fused to the target database based on the incremental data fusion request.

According to a database-based data fusion method provided by this application, performing an incremental fusion operation for merging the incremental data to be fused to the target database includes:

Obtain the real-time requirements of the incremental data to be fused, and select a target incremental synchronization interface that matches the real-time requirements from the preset incremental synchronization interfaces;

Based on the target incremental synchronization interface, an incremental fusion operation is performed for merging the incremental data to be fused to the target database.

According to a database-based data fusion method provided by this application, after performing the fusion operation of merging the data to be fused in the intermediate database to the target database, the method further includes:

Obtain the current fusion task recorded at the end of the fusion operation and the task detail data of the current fusion task;

Based on the current fusion task and the task detail data, the preset fusion task history table and task data detail history table are updated.

According to a database-based data fusion method provided by this application, after performing the fusion operation of merging the data to be fused in the intermediate database to the target database, the method further includes:

When it is determined that the first abnormal information indicating the abnormality of the trigger is monitored, the first abnormal information is recorded and an alarm is provided.

According to a database-based data fusion method provided by this application, after performing the fusion operation of merging the data to be fused in the intermediate database to the target database, the method further includes:

When it is determined that the second abnormal information indicating the abnormality of the fusion operation is monitored, alarm indication information is output, and the alarm indication information is used to indicate the manual intervention processing abnormality and supports the re-fusion operation for the data to be fused corresponding to the second abnormal information.

According to a database-based data fusion method provided by this application, the method further includes:

When it is determined that the source database and the target database have the same library table and the same primary key, perform an operation of adding a new preset field for the same library table;

A joint primary key is determined based on the same primary key and the preset field obtained by the adding operation.

This application also provides a database-based data fusion device, including:

The synchronization module is used to synchronize the data to be fused in the source database to the intermediate database based on the data fusion request;

A fusion module, configured to perform a fusion operation for merging the data to be fused in the intermediate database to the target database;

Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused.

This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any one of the above database-based operations. Data fusion methods.

The present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the database-based data fusion method as described above is implemented.

The present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements any of the above-mentioned database-based data fusion methods.

This application provides a database-based data fusion method, device, electronic equipment and storage medium. The database-based data fusion method synchronizes the data to be fused in the source database to the intermediate database based on the data fusion request, and then executes the intermediate database. The fusion operation of merging the data to be fused in the database to the target database uses the intermediate database as a transfer buffer to implement the relevant processing operations for the data to be fused, ensuring that the actual production of the source database and the target database is not affected; further Since the intermediate database is used to represent the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused, the use of the intermediate database can handle the different names, structures, and fields of the library tables in the source database and the target database. Data fusion operations under various circumstances such as different databases and different database tables support data fusion of multiple non-homogeneous databases. It can also maintain the simultaneous accessibility of the source database and the target database to achieve online data fusion, so that it can also be used in The purpose of achieving large-scale complex data fusion without affecting actual production.

Description of the drawings

In order to explain the technical solutions in this application or the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are of the present invention. For some embodiments of the application, those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is one of the flow diagrams of the database-based data fusion method provided by this application;

Figure 2 is a data flow logic diagram of the database-based data fusion method provided by this application;

Figure 3 is a schematic structural diagram of the database-based data fusion device provided by this application;

Figure 4 is a schematic structural diagram of an electronic device provided by this application.

Detailed ways

In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application. , not all examples. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

At present, most database products have corresponding backup and replication mechanisms, but they provide standardized products and provide universal solutions for data replication, data backup and other scenarios. However, in the actual production process, in addition to data replication, In addition to data backup scenarios, it is also necessary to deal with the progressive fusion of multiple databases.

The most typical and mature product on the market is Oracle's GoldenGate software, which can provide data disaster recovery and data replication functions, support zero-downtime database upgrades and migrations, and achieve sub-second real-time data synchronization. , but GoldenGate software is mainly used for data replication operations between homologous databases or databases in different regions with the same business. For two originally different business databases, when the names and contents of most of the database tables are similar, the following problems cannot be solved: (1) When there are completely different database tables on the source end and the target end, they need to be deleted or added Some library tables cannot be processed automatically; (2) When the fields of the source library table and the target library table are different, some fields need to be added or deleted and cannot be processed automatically; (3) When the source library table and the target library table When there is a primary key conflict in the database table and the data is different, you can only choose to overwrite or discard it. The two corresponding data cannot be completely saved, and the data source cannot be recorded; (4) Backtracking of historical logs for data synchronization is not supported; (5) There is a lack of support for data synchronization. Supports complex scenarios of merging multiple non-homogeneous databases.

Based on the above problems, this application provides a database-based data fusion method, device, electronic device and storage medium. The execution subject of the database-based data fusion method can be a terminal device containing a source database, and the terminal device can be a personal computer ( Personal Computer (PC), portable devices, laptops, smartphones, tablets, portable wearable devices and other electronic devices, and the client has at least the data packet reception and playback function, data packet capture function, storage function, and analysis function and statistical analysis functions. This application does not limit the specific form of the client.

It should be noted that the following method embodiments are explained by taking the execution subject as a terminal device as an example, and the following method embodiments are integrated from the source database of the Group Customer Business Integrated Operation Platform (Enterprise Service Operation Platform, ESOP) library. The fusion method is described in detail using the target database of Customer Relationship Management (CRM) library as an example. The database-based data fusion method, device, electronic device and storage medium of the present application will be described below with reference to Figures 1-4.

Referring to Figure 1, a schematic flow chart of the database-based data fusion method provided by this application is shown. As shown in Figure 1, the database-based data fusion method includes the following steps:

Step 110: Based on the data fusion request, synchronize the data to be fused in the source database to the intermediate database.

Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused.

Specifically, when the data to be fused is data from at least one database table in the source database, the attribute information of the data to be fused may include the database table structure and data in the table of the at least one database table. At this time, the data fusion request is for the at least one database table. A full data fusion request for a database table; and, the attribute information of the database table that fuses the data to be fused in the target database may include the database table structure of the database table, that is, when the intermediate database includes a temporary source database and a temporary target database At this time, the temporary source database remains consistent with the structure of the database table where the data to be fused is located and the data in the table, and the temporary target database remains consistent with the database table structure of the corresponding database table in the target database. Based on this, based on the data fusion request, the data to be fused in the source database is synchronized to the intermediate database. The reliable and efficient GoldenGate technology can be used to synchronize the data to be fused to the temporary source database of the intermediate database.

Step 120: Perform a fusion operation for merging the data to be fused in the intermediate database to the target database.

Specifically, when the data to be fused in the source database is synchronously transferred to the temporary source database of the intermediate database, the fusion operation of the data to be fused to the target database can also be performed based on the reliable and efficient GoldenGate technology. As shown in Figure 2, the database tables TABLE1, TABLE2, TABLE3 and TABLE4 in the source database can be synchronously transferred to the temporary source database first, and the database tables TABLE1, TABLE2, TABLE3 and TABLE4 in the temporary source database can then be synchronously transferred to the temporary source database. In the target database, GoldenGate technology is used for both synchronization transfers. If data splitting, data conversion and other operations need to be performed on the fused data, it can also be completed in the intermediate database to ensure that the impact on the source database and target database is minimized. , so it can also ensure smooth docking and synchronization of data.

The database-based data fusion method provided by this application synchronizes the data to be fused in the source database to the intermediate database based on the data fusion request, and then performs the fusion operation of merging the data to be fused in the intermediate database to the target database. By using the intermediate database as a transfer cache area, the relevant processing operations for the data to be fused are implemented to ensure that the actual production of the source database and the target database is not affected; further, because the intermediate database is used to represent the attribute information and targets of the data to be fused The database combines the attribute information of the database tables of the data to be fused. Therefore, the use of the intermediate database can handle data fusion operations in various situations such as different database table names, different structures, different fields, and different database tables in the source database and the target database. Support The data fusion of multiple non-homogeneous databases can also maintain the simultaneous accessibility of the source database and the target database, and achieve online data fusion, thus achieving the purpose of large-scale complex data fusion without affecting actual production.

Optionally, the implementation process of step 120 may include:

When it is determined that the target database does not have the data to be fused, the data to be fused in the intermediate database is synchronized to the target database.

Specifically, when it is determined that the target database does not have a database table containing the data to be fused and the target database does not need to split the database tables according to the preset identifier, there is no need to convert and migrate the data to be fused. Instead, a new database table is created in the target database to Import the data to be fused, that is, you can use GoldenGate technology to directly transfer the data to be fused in the temporary source database of the intermediate database to the target database synchronously. At this time, the database tables where the data to be fused stored in the source database, intermediate database, and target database are located The names, structures, and field meanings are all consistent. Among them, the preset identification includes but is not limited to the regional identification.

The database-based data fusion method provided by this application improves the operability of data fusion by directly synchronously transferring the data to be fused in the intermediate database to the target database when there is no database table containing the data to be fused in the target database. The implementation utilizes an intermediate database to ensure that actual production of the source and target databases is not affected.

Optionally, when there are multiple target databases, the implementation process of step 120 may include:

First, when it is determined that the target database does not have the data to be fused, the data to be fused is synchronized to the temporary source database of the intermediate database for splitting operations to obtain multiple data to be fused; then, the multiple data to be fused are The fused data is transferred to the temporary target database of the intermediate database; finally, a fusion operation is performed for each data to be fused in the temporary target database to be merged into the corresponding target database.

Specifically, when it is determined that the target database does not have the database table where the data to be fused is located and the target database needs to split the database tables according to the preset identifier, then the data to be fused needs to be split and migrated, that is, the temporary source of the intermediate database must be first The data to be fused in the database is split into multiple database tables, and then the temporary source database is split using DUMP and INSERT. Export multiple database tables to the temporary target database, and finally use GoldenGate technology to synchronously transfer multiple database tables in the temporary target database to the corresponding target database.

The database-based data fusion method provided by this application, when there is no database table containing the data to be fused in the target database and the number of the database tables is multiple, the database table containing the data to be fused is first split in the intermediate database and then corresponding to the target database. Perform synchronous transfer to improve the flexibility and reliability of data fusion by presetting various data conversion operations and mapping relationships between the temporary source database and the temporary target database of the intermediate database, so there is no need to modify the existing business logic. Achieving the purpose of online data fusion overcomes the difficulty of traditional data synchronization and fusion scenarios that cannot meet personalized needs. It can also use intermediate databases to ensure that the actual production of the source database and target database is not affected.

Optionally, the implementation process of step 120 may also include

First, when it is determined that there is a first database table corresponding to the data to be fused in the target database, based on the difference between the data to be fused and the first database table, perform a search for the data to be fused in the intermediate database. Data conversion operation; then, synchronize the target to-be-fused data obtained by the data conversion operation to the first database table of the target database.

Specifically, when it is determined that the first database table corresponding to the database table where the data to be fused exists in the target database, the database table where the data to be fused and the first database table have at least differences such as field inconsistency and/or database table structure inconsistency. You can first synchronously transfer the database table where the data to be fused in the temporary source database of the intermediate database is located to the temporary target database, and then instruct the temporary target database to uniformly modify the fields, add fields, and / Or data conversion operations such as database table structure conversion, so that the target obtained by the data conversion operation is to be integrated with the database table structure of the database table where the data is located, such as changing the field "name" to "name". Then GoldenGate technology is used to synchronously transfer the target data to be fused obtained from the data conversion operation to the first database table of the target database.

The database-based data fusion operation provided by this application, when there is a first database table corresponding to the database table where the data to be fused exists in the target database, the data to be fused is first synchronously transferred to the temporary target database, and then the temporary target database is The method of data conversion operation on the data to be fused is to achieve the purpose of synchronously transferring the target fusion data obtained by the data conversion operation to the target database, so as to process the different table names, structures and fields in the source database and the target database through the intermediate database. Data fusion operations in various scenarios such as databases and different database tables can also achieve the purpose of large-scale complex data fusion.

Optionally, when the number of data to be fused is multiple and there is a preset relationship between the multiple data to be fused, the implementation process of step 120 may also include:

First, when it is determined that there is a second database table corresponding to multiple data to be fused in the target database, a merge operation is performed on multiple data to be fused in the intermediate database to obtain an intermediate database table; secondly, based on the The difference between the intermediate database table and the second database table is determined, and a data conversion operation is performed on the intermediate database table in the intermediate database; finally, the target to-be-fused data obtained by the data conversion operation is synchronized to the target in the second database table of the database.

Among them, the preset relationship can represent that the database tables where multiple data to be fused are essentially the same but have subtle differences. For example, when there are two database tables to be fused, they can be "girls' detailed information table" and "boys' detailed information table".

Specifically, when there are database tables in the source database that represent multiple different user data to be fused, that is, when the number of database tables where the data to be fused is multiple, if there are multiple database tables in the target database that are located in the data to be fused When the database table corresponds to a second database table, for example, the second database table can be a "personnel details information table", in this case, GoldenGate technology can be used to synchronously transfer the database tables where the multiple data to be integrated are located to the temporary source database, and instruct the temporary The source database merges the database tables containing multiple data to be merged. When the number of target databases is one, it can instruct the temporary target database to perform data conversion operations on the merged intermediate database tables; when the number of target databases is multiple , you can instruct the temporary source database to first perform a split operation on the intermediate database table obtained by merging, and then instruct the temporary target database to perform a data conversion operation, and finally synchronously transfer the target fusion data obtained from the data conversion operation to the second database table of the target database. . For the splitting operation and data conversion operation, reference may be made to the foregoing method embodiments and will not be described again here.

The database-based data fusion method provided by this application, when there is a second database table corresponding to the database table where multiple data to be fused is located in the target database, the multiple database tables where the data to be fused are located are first merged in the intermediate database By operating and then performing data conversion operations, the intermediate database is used to implement operations such as splitting operations of database tables, data conversion, adding fields, and modifying fields involved in the data fusion process, thereby solving the difficult problem of complex data fusion. question.

In the actual processing process, in addition to full data fusion, there is also incremental data fusion. That is, after the full data fusion is completed, the incremental data to be fused can be obtained based on the changed data in the source database and incremental data fusion can be performed. , that is, this application supports incremental data fusion during the full data fusion process. Based on this, the implementation process of step 120 may include:

First, it is determined that when the trigger collects change data that represents changes in the database table data in the source database, it triggers the generation of an incremental data fusion request; then, obtains the incremental data to be fused corresponding to the change data, and based on the increment Data fusion request: perform an incremental fusion operation for merging the incremental data to be fused to the target database.

Specifically, since the source database and the target database can run at the same time and run devices at the same time, the source database will continue to generate new business data, thereby causing the data in the corresponding database tables in the source database to change. Therefore, in order to ensure that The changed data can also be accurately and timely integrated into the corresponding database table of the target database. Triggers can be set for each database table in the source database. Each trigger is used to collect change data of the corresponding database table in the source database. Change data represents changes in the corresponding database table data in the source database due to changes in business processing or manual modification of data. When the trigger is determined to collect change data representing changes in the database table data in the source database, it can automatically trigger the generation of incremental data. Fusion request. In addition, the fusion task table, task data detail table, fusion task history table, and task data detail history table are added in the source database in advance, and the fusion task table and task data detail table are correspondingly modified based on the changed data to make them reach the latest version. .

Further, the incremental data to be fused is read from the latest version of the fusion task table and the task data detail table. The incremental data to be fused includes the tasks to be fused corresponding to the changed data and the task details of the tasks to be fused, and based on Incremental data fusion request: perform an incremental fusion operation on the incremental data to be fused to the target database.

It should be noted that the triggers set for each database table in the source database can all be new/updated/delete (INSERT/UPDATE/DELETE, DML) triggers. By using DML triggers to trigger the data fusion operation, And establish an audit and comparison mechanism that integrates task tables and task data details to ensure data consistency. If data anomalies are found, retrospective verification can be performed by fusing the task history table and the task data detail history table.

The database-based data fusion method provided by this application performs an incremental fusion operation on the incremental data to be fused to the target database based on the incremental data fusion request generated after the full data fusion is completed and the acquired incremental data to be fused. This achieves the purpose of real-time saving of incremental data fusion tasks and task details for changed data after full data fusion is completed, thereby ensuring that the target database integrates the changed data in the source database in a timely and effective manner, and provides synchronization logging functions and data synchronization backtracking. Function.

Optionally, when generating an incremental data fusion request, the implementation process of step 120 may also include:

First, obtain the real-time requirements of the incremental data to be fused, and select a target incremental synchronization interface that matches the real-time requirements from the preset incremental synchronization interfaces; then, based on the target incremental synchronization interface , perform an incremental fusion operation for merging the to-be-fused incremental data in the intermediate database to the target database.

Specifically, there is key information in the source database that does not allow errors, such as arrears status, balance, account number, etc. This type of key information requires high real-time performance. Therefore, when merging the incremental data to be fused, first determine the data to be fused. Whether the incremental data to be fused is data with high real-time requirements. If the incremental data to be fused does not have high real-time requirements, the file interface can be used as the target incremental synchronization interface; if the incremental data to be fused has high real-time requirements, A real-time interface can be used as the target incremental synchronization interface. Then, based on the target incremental synchronization interface, an incremental fusion operation is performed to fuse the incremental data to be fused to the target database.

It should be noted that the source database can be pre-set with incremental synchronization interfaces for incremental fusion operations, such as file interfaces and real-time interfaces. The file interface is used to synchronously transfer data with low practical requirements, and the real-time interface is used to synchronously transfer real-time data. Data with high performance requirements, that is, the original application corresponding to the database table where the incremental data to be fused has not been offline. The original application is still the source database, and the data merged into the target database is only a copy of the data of the original application, which has no impact on real-time performance. The requirements are not high. At this time, you can choose the file interface for incremental data synchronization; during the gradual online process of new applications, the original application corresponding to the database table where the incremental data to be integrated will still generate some data during the offline process. In order to ensure that the new application is The accuracy of data during the gradual online process requires incremental data fusion. For some query services with high real-time requirements, real-time interfaces need to be used for incremental data fusion. At this time, since the database table data in the source database and the target database are inconsistent, it is necessary to read the incremental data to be fused from the latest version of the fusion task table and task data detail table, and then read the database table in the source database and the target source database according to the The mapping relationship between the tables in the database can be split and/or converted to the incremental data to be fused and then synchronously transferred to the target database.

The database-based data fusion method provided by this application selects a target incremental synchronization interface that matches the real-time requirements of the incremental data to be fused, so that the incremental data to be fused can be accurately and timely synchronously transferred to the target database, thereby Improved the reliability and stability of incremental data fusion.

Optionally, for incremental data fusion, the implementation process of step 120 may include:

First, obtain the current fusion task recorded at the end of the fusion operation and the task detailed data of the current fusion task; secondly, based on the current fusion task and the task detailed data, update the preset fusion task history table and task Data detail history table.

Specifically, regardless of whether the current fusion task is for full data fusion or incremental data fusion, at the end of each fusion operation, the current fusion task and its task details will be recorded in the preset fusion task table and task data details table, and all the data will be recorded. The recorded current fusion task and its task detail data are moved into the corresponding fusion task history table and task data detail history table, thereby achieving real-time updates to the fusion task history table and task data detail history table. Among them, the specific structures of the fusion task table, task data detail table, fusion task history table and task data detail history table are as follows:

Table 1 Fusion task list

Table 2 Task data details

Table 3 Fusion task history table

Table 4 Task data details history table

The database-based data fusion method provided by this application updates the fusion task in real time by moving the recorded current fusion task and its task detail data into the corresponding fusion task history table and task data detail history table at the end of each fusion operation. History table and task data detail history table, thus providing sufficient data support for subsequent auditing and backtracking.

During the actual processing, when the data in the source database, intermediate database or target database changes, the database log will be generated and saved accordingly. Based on this, before step 110, you can set the settings in the source database, intermediate database or target database. Schedule tasks, and pre-set each database table that requires full data fusion, that is, the full table in the cutover range, so that when anomalies are found through statistical comparison of the quantities in the task detail data table and the task data detail history table, you can search through Repair each database log, and when it is determined that the data to be fused to be synchronously transferred to the target database has data missing problems, such as when the structure of the database table corresponding to the data to be fused is consistent in the source database and the target database, the data to be fused is There are 1 million entries in the source database, but more than 10,000 entries are missing when synchronized to the target database. At this time, it can be considered that there is a data missing problem; for example, for the "name" field synchronization in the database table where the data to be fused is located in the source database Check whether fields are missing or have wrong meanings after being transferred to the target database. For data missing problems, field spot checks can be performed and compared. Data missing problems can be caused by trigger exceptions or synchronization transfer exceptions, and can also be found by searching the logs of each database. manual intervention and repair; in addition, important data in the production process, such as the balance of the number, basic customer information, user attribute information and other core data that cannot make mistakes can be set as core table data in the core database, and use daily updates The generated business obtains daily incremental data for auditing and reconciliation. For non-important data in the production process, such as work orders, data in the intermediate driving process of log records and data on business processes, the data volume is usually large and can be No reconciliation is performed; for the triggers set corresponding to each database table in the source database, when a trigger fails, the change data can be obtained from each database log file for manual comparison and manual repair. Each database log file can be managed by The administrator can set the retention period and its configuration. When a trigger is abnormal, the fusion operation can also be manually triggered. The trigger can also be flexibly configured based on the database log file. For example, when trigger a is abnormal at time A, the updated data of trigger a after time A can be obtained by searching the log files of each database, so that the updated data generated from the failure of trigger a to the time when it is repaired can also be obtained. Participate in subsequent audits and reconciliations.

Optionally, after step 120, the method may also include:

When it is determined that the first abnormal information indicating the abnormality of the trigger is monitored, the first abnormal information is recorded and an alarm is provided.

Specifically, the triggers set corresponding to each database table in the source database not only have a monitoring mechanism, but also a recovery mechanism. Therefore, in order to ensure that the fusion operation is not affected when the trigger's monitoring mechanism is abnormal, real-time monitoring of triggers can be set in advance. Daemon process, in order to achieve real-time monitoring of the status of triggers. The daemon process is an independent program and can be set up on the terminal device. It can access various databases, such as source database, intermediate database and target database. When the daemon detects an exception in a trigger, it records the first exception information for the trigger and issues an alarm in a timely manner. At the same time, the abnormal trigger can start its own recovery mechanism to automatically restore the listening mechanism. , the monitoring mechanism can also be restored by manually modifying the program settings, and there are no specific limitations here.

The database-based data fusion method provided by this application can promptly detect abnormal triggers by setting up a daemon process to monitor whether the triggers are abnormal. It can not only save the log details of the fusion task for easy backtracking, but also ensure data consistency, thus When the trigger is abnormal due to failure or other reasons, it can also ensure the normal execution of data fusion operations and the effective recording of new business data, ensure the normal operation of online data fusion, and effectively improve the data consistency and the accuracy of data backtracking.

Optionally, after step 120, the method may also include:

When it is determined that the second abnormal information indicating the abnormality of the fusion operation is monitored, alarm indication information is output, and the alarm indication information is used to indicate the manual intervention processing abnormality and supports the re-fusion operation for the data to be fused corresponding to the second abnormal information.

Among them, the fusion operation anomaly can be caused by the failure of synchronously transferring the data to be fused to the target database, resulting in a backlog of queued data to be fused, and the backlog of queued data to be fused can be that the number of queued data to be fused exceeds a preset quantity threshold.

Specifically, a monitoring process can be set for the preset fusion task table or fusion task history table. The monitoring process is used to monitor whether the fusion tasks are backlogged, that is, to monitor whether the queued data to be fused is backlogged, such as the data to be fused in a short period of time. If there are too many, it may cause the fusion operation to fail until a backlog occurs. Based on this, when the monitoring process monitors the second abnormal information indicating the backlog of queued data to be fused, it can be considered that the failure of the fusion operation on a certain data to be fused causes subsequent data to be fused to be queued one by one until the backlog or a large number of fusion operations fail. , for example, when the queued data to be fused exceeds 1,000 or 10,000, the queued data to be fused is considered to be backlogged. Fusion operation failures can include abnormal failures and normal failures. Normal failures do not require alarms and manual intervention, and abnormal failures (such as system If there are too many times (timeouts), alarms and manual intervention are required. That is, for the second abnormal information, alarm indication information can be output while recording the second abnormal information. The alarm indication information is used to indicate manual intervention to handle the abnormality and supports the re-fusion operation for the data to be fused corresponding to the second abnormal information. This allows the fusion operation to be re-executed on the queued data to be fused when the manual intervention process is completed abnormally.

The database-based data fusion method provided by this application establishes a data fusion task backlog monitoring mechanism by setting up a monitoring process to monitor whether the fusion operation is backlogged, ensuring that each data fusion task can be executed normally without affecting actual production, thereby also The stability and reliability of data fusion operations are further improved.

Optionally, the method may also include:

First, when it is determined that the source database and the target database have the same library table and the same primary key, perform an adding operation of adding a preset field for the same library table; then, based on the same primary key and the result of the adding operation The preset field determines the joint primary key.

Specifically, when different source databases and target databases are merged, if the source database and the target database have the same database table and the same primary key, preset fields can be added separately for the same database table. For example, the new preset fields can include but Do not limit the "data source" field, and use the same original primary key and the preset field to re-establish the joint primary key. The joint primary key is a primary key composed of 2 or more fields. When using the fields included in the joint primary key as the primary key At that time, it is unique in both the source database and the target database, and a joint primary key index can be attached to achieve the purpose of saving the data of both the source database and the target database.

The database-based data fusion method provided by this application solves the primary key conflict problem by rebuilding the joint primary key for the same database table in the source database and the target database. At the same time, it can also completely save the data of the source database and the target database, reducing the impact on the business system. Operational impact, thereby achieving the purpose of ensuring data integrity and recording the source of data even when there is a primary key conflict between the source database and the target database and the data is different.

The database-based data fusion device provided by this application is described below. The database-based data fusion device described below and the database-based data fusion method described above can be mutually referenced.

Figure 3 is a schematic structural diagram of a database-based data fusion device provided by this application. As shown in Figure 3, the database-based data fusion device 300 includes:

The synchronization module 310 is used to synchronize the data to be fused in the source database to the intermediate database based on the data fusion request; the fusion module 320 is used to perform the fusion of the data to be fused in the intermediate database to the target database. fusion operation;

Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused.

Optionally, the fusion module 320 may be configured to synchronize the data to be fused in the intermediate database to the target database when it is determined that the target database does not have the data to be fused.

Optionally, the fusion module 320 can also be used to determine that the target database does not have the data to be fused, synchronize the data to be fused to the temporary source database of the intermediate database, and perform a split operation to obtain multiple data to be fused. Fusion data; transfer the plurality of data to be fused to a temporary target database of the intermediate database; and perform a fusion operation for each data to be fused in the temporary target database into the corresponding target database.

Optionally, the fusion module 320 may also be used to determine that there is a first library table corresponding to the data to be fused in the target database, based on the difference between the data to be fused and the first library table, execute the Data conversion operation of the data to be fused in the intermediate database; synchronizing the target data to be fused obtained by the data conversion operation to the first database table of the target database.

Optionally, the fusion module 320 may also be configured to perform a merge of multiple data to be fused in the intermediate database when it is determined that there is a second database table corresponding to multiple data to be fused in the target database. Operation to obtain an intermediate database table; based on the difference between the intermediate database table and the second database table, perform a data conversion operation for the intermediate database table in the intermediate database; and convert the target obtained by the data conversion operation to The fusion data is synchronized to the second database table of the target database.

Optionally, the fusion module 320 can also be used to trigger the generation of an incremental data fusion request when the trigger collects change data representing changes in the database table data in the source database; and obtain the incremental data to be fused corresponding to the change data. amount of data, and based on the incremental data fusion request, perform an incremental fusion operation for merging the incremental data to be fused to the target database.

Optionally, the fusion module 320 can also be used to obtain the real-time requirements of the incremental data to be fused, and select a target incremental synchronization interface that matches the real-time requirements from the preset incremental synchronization interfaces. ; Based on the target incremental synchronization interface, perform an incremental fusion operation for merging the incremental data to be fused to the target database.

Optionally, the device further includes an update module for obtaining the current fusion task recorded at the end of the fusion operation and the task detail data of the current fusion task; based on the current fusion task and the task detail data, Update the preset fusion task history table and task data detail history table.

Optionally, the device further includes an abnormality alarm module, configured to record the first abnormal information and issue an alarm when it is determined that the first abnormal information indicating an abnormality of the trigger is monitored.

Optionally, the abnormality alarm module can also be used to output alarm indication information when it is determined that the second abnormality information representing an abnormality in the fusion operation is monitored, and the alarm indication information is used to indicate manual intervention to handle the abnormality and for the second abnormality. The data to be fused corresponding to the abnormal information supports re-fusion operations.

Optionally, the device further includes a determination module configured to perform an adding operation of adding a preset field for the same library table when it is determined that the source database and the target database have the same library table and the same primary key; based on The same primary key and the preset field obtained by the adding operation determine a joint primary key.

Figure 4 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 4, the electronic device 400 may include: a processor (processor) 410, a communication interface (CommunicationsInterface) 420, a memory (memory) 430 and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 complete communication with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a database-based data fusion method, which method includes:

Based on the data fusion request, synchronize the data to be fused in the source database to the intermediate database;

Execute a fusion operation for merging the data to be fused in the intermediate database to the target database;

Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused.

In addition, the above-mentioned logical instructions in the memory 430 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code.

On the other hand, the present application also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can Execute the database-based data fusion method provided by each of the above methods, which includes:

Based on the data fusion request, synchronize the data to be fused in the source database to the intermediate database;

Execute a fusion operation for merging the data to be fused in the intermediate database to the target database;

Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused.

On the other hand, the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. The computer program is implemented when executed by the processor to execute the database-based data fusion method provided by the above methods. Methods include:

Based on the data fusion request, synchronize the data to be fused in the source database to the intermediate database;

Execute a fusion operation for merging the data to be fused in the intermediate database to the target database;

Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused.

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (15)

1. A database-based data fusion method, characterized by including: Based on the data fusion request, synchronize the data to be fused in the source database to the intermediate database; Execute a fusion operation for merging the data to be fused in the intermediate database to the target database; Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused. 2. The database-based data fusion method according to claim 1, characterized in that said performing a fusion operation for merging the data to be fused in the intermediate database to the target database includes: When it is determined that the target database does not have the data to be fused, the data to be fused in the intermediate database is synchronized to the target database. 3. The database-based data fusion method according to claim 1, characterized in that when the number of the target databases is multiple, the execution is to fuse the data to be fused in the intermediate database to the Fusion operations on the target database, including: When it is determined that the target database does not have the data to be fused, synchronize the data to be fused to the temporary source database of the intermediate database and perform a split operation to obtain multiple data to be fused; Transfer the plurality of data to be fused to the temporary target database of the intermediate database; Perform a fusion operation for each data to be fused in the temporary target database into the corresponding target database. 4. The database-based data fusion method according to claim 1, characterized in that said performing a fusion operation for merging the data to be fused in the intermediate database to the target database includes: When it is determined that there is a first database table corresponding to the data to be fused in the target database, based on the difference between the data to be fused and the first database table, perform data conversion for the data to be fused in the intermediate database operate; Synchronize the target data to be fused obtained by the data conversion operation to the first database table of the target database. 5. The database-based data fusion method according to claim 1, characterized in that when the number of the data to be fused is multiple and there is a preset relationship between the multiple data to be fused, the execution is for The fusion operation of merging the data to be fused in the intermediate database to the target database includes: When it is determined that there is a second database table corresponding to multiple data to be fused in the target database, perform a merge operation on multiple data to be fused in the intermediate database to obtain an intermediate database table; Based on the difference between the intermediate database table and the second database table, perform a data conversion operation on the intermediate database table in the intermediate database; Synchronize the target data to be fused obtained by the data conversion operation to the second database table of the target database. 6. The database-based data fusion method according to claim 1, characterized in that said performing a fusion operation for merging the data to be fused in the intermediate database to the target database includes: When the determination trigger collects change data that represents changes in the database table data in the source database, it triggers the generation of an incremental data fusion request; Acquire the incremental data to be fused corresponding to the changed data, and perform an incremental fusion operation for merging the incremental data to be fused to the target database based on the incremental data fusion request. 7. The database-based data fusion method according to claim 6, wherein said performing an incremental fusion operation for merging the incremental data to be fused to the target database includes: Obtain the real-time requirements of the incremental data to be fused, and select a target incremental synchronization interface that matches the real-time requirements from the preset incremental synchronization interfaces; Based on the target incremental synchronization interface, an incremental fusion operation is performed for merging the incremental data to be fused to the target database. 8. The database-based data fusion method according to claim 1, characterized in that, after the execution of the fusion operation for the data to be fused in the intermediate database to the target database, the method Also includes: Obtain the current fusion task recorded at the end of the fusion operation and the task detail data of the current fusion task; Based on the current fusion task and the task detail data, the preset fusion task history table and task data detail history table are updated. 9. The database-based data fusion method according to claim 1, characterized in that, after the execution of the fusion operation for the data to be fused in the intermediate database to the target database, the method Also includes: When it is determined that the first abnormal information indicating the abnormality of the trigger is monitored, the first abnormal information is recorded and an alarm is provided. 10. The database-based data fusion method according to claim 1, characterized in that, after the execution of the fusion operation for the data to be fused in the intermediate database to the target database, the method Also includes: When it is determined that the second abnormal information indicating the abnormality of the fusion operation is monitored, alarm indication information is output, and the alarm indication information is used to indicate the manual intervention processing abnormality and supports the re-fusion operation for the data to be fused corresponding to the second abnormal information. 11. The database-based data fusion method according to claim 1, characterized in that the method further includes: When it is determined that the source database and the target database have the same library table and the same primary key, perform an operation of adding a new preset field for the same library table; A joint primary key is determined based on the same primary key and the preset field obtained by the adding operation. 12. A database-based data fusion device, characterized by including: The synchronization module is used to synchronize the data to be fused in the source database to the intermediate database based on the data fusion request; A fusion module, configured to perform a fusion operation for merging the data to be fused in the intermediate database to the target database; Wherein, the intermediate database is used to characterize the attribute information of the data to be fused and the attribute information of the library table in the target database that fuses the data to be fused. 13. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that when the processor executes the program, it implements claim 1 The database-based data fusion method described in any one of to 11. 14. A non-transitory computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the database-based data fusion as claimed in any one of claims 1 to 11 is achieved. method. 15. A computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the database-based data fusion method according to any one of claims 1 to 11 is implemented.