CN107729541A - A kind of data processing method, device and computer-readable recording medium - Google Patents

Abstract

The invention discloses a kind of data processing method, including：Obtain data to be audited online from least two data sources；Rule is checked based on default, the data to be audited are checked；It will check that unsanctioned data are defined as complex data to be repaired, and, according to default reparation order, the complex data to be repaired is repaired.The present invention further simultaneously discloses a kind of data processing equipment and computer-readable recording medium.

Description

A data processing method, device and computer-readable storage medium

technical field

The present invention relates to the field of computer technology, in particular to a data processing method, device and computer-readable storage medium.

Background technique

At present, for the situation where multiple data sources are required to cooperate to realize a certain business, some data in these mutually coordinated data sources need to be consistent. In order to ensure the integrity and consistency of data, it is necessary to audit and repair the data in multiple data sources that cooperate with each other.

At present, the method for auditing and repairing data in multiple data sources includes: establishing a backup database for each data source, synchronously backing up the data in the data source to the corresponding backup database according to a preset cycle (such as one day), Then perform data audit based on the data in each backup database, and then perform data restoration on the data in the data source according to the audit results.

However, since the data audit operation is performed in the backup environment on a daily basis, that is, the data audit operation is performed offline, resulting in low real-time performance of the data audit and affecting the accuracy of the data audit. Therefore, the data audit results obtained based on the backup database are inaccurate and cannot truly reflect the data consistency in each data source. Furthermore, after the data in the data source is repaired according to the audit results obtained from the backup database, it is still impossible to ensure that all Consistency and accuracy of data in data sources.

Contents of the invention

In view of this, the embodiments of the present invention expect to provide a data processing method, device, and computer-readable storage medium, which can ensure the consistency and accuracy of data from at least two data sources.

The technical scheme of the embodiment of the present invention is realized like this:

An embodiment of the present invention provides a data processing method, the method comprising:

Obtain data to be audited online from at least two data sources;

Auditing the data to be audited based on preset audit rules;

The data that fails the audit is determined as data to be repaired, and the data to be repaired is repaired according to a preset repair sequence.

In the above solution, the online acquisition of data to be audited from at least two data sources includes:

determining pre-stored metadata corresponding to the data to be audited;

According to the metadata, the data to be audited corresponding to the metadata is acquired from the at least two data sources.

In the above scheme, the method also includes:

Acquiring the metadata from the at least two data sources according to preset acquisition rules;

And/or, receiving the metadata pushed by the at least two data sources.

In the above solution, the auditing of the data to be audited based on the preset audit rules includes:

According to the preset audit rules, it is judged whether the data to be audited is consistent in the at least two data sources;

If they are consistent, it is determined that the data to be audited has passed the audit; otherwise, it is determined that the data to be audited has not passed the audit.

In the above scheme, the method also includes:

judging whether the data to be repaired is successfully repaired;

When it is determined that the data to be repaired has not been successfully repaired, the data to be repaired is repaired again;

When the number of repairs to the data to be repaired is greater than the preset number of repairs, an alarm message is generated; the alarm message indicates that the data to be repaired cannot be repaired.

In the above solution, the judging whether the data to be repaired is successfully repaired includes:

judging whether the repaired data to be repaired is consistent in the at least two data sources;

If they are consistent, it is determined that the data to be repaired is successfully repaired; otherwise, it is determined that the data to be repaired is not successfully repaired.

An embodiment of the present invention provides a data processing device, the device comprising:

An acquisition module, configured to acquire data to be audited online from at least two data sources;

An audit module, configured to audit the data to be audited based on preset audit rules;

The repair module is configured to determine the data that fails the audit as data to be repaired, and repair the data to be repaired according to a preset repair sequence.

In the above solution, the acquiring module is specifically configured to determine pre-stored metadata corresponding to the data to be audited; according to the metadata, acquire the metadata corresponding to the metadata from the at least two data sources The data to be audited.

In the above scheme, the device also includes:

The collection module is configured to obtain the metadata from the at least two data sources according to preset collection rules; and/or receive the metadata pushed by the at least two data sources.

In the above solution, the audit module is specifically used to judge whether the data to be audited is consistent in the at least two data sources according to the preset audit rules; if they are consistent, determine that the data to be audited has passed the audit ; Otherwise, determine that the data to be audited has not passed the audit.

In the above scheme, the device also includes:

An alarm module, used to judge whether the data to be repaired is successfully repaired; when it is determined that the data to be repaired has not been successfully repaired, repair the data to be repaired again; when the number of repairs to the data to be repaired is greater than the preset When the repair times are exceeded, an alarm message is generated; the alarm message indicates that the data to be repaired cannot be repaired.

In the above solution, the alarm module is specifically used to judge whether the repaired data to be repaired is consistent in the at least two data sources; if consistent, determine that the repair of the data to be repaired is successful; otherwise, determine the The data to be repaired was not successfully repaired.

An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any data processing method described above are implemented.

An embodiment of the present invention provides a data processing device, including: a memory, a processor, and a computer program stored in the memory and operable on the processor;

Wherein, when the processor is configured to run the computer program, it executes the steps of any data processing method described above.

The data processing method, device, and computer-readable storage medium provided by the embodiments of the present invention obtain online data to be audited from at least two data sources; audit the data to be audited based on preset audit rules; The data to be repaired is determined as the data to be repaired, and the data to be repaired is repaired according to the preset repair sequence, so as to effectively realize the online audit and timely repair of the data, and improve the efficiency and accuracy of the data audit and data repair.

Description of drawings

Fig. 1 is a schematic diagram of the implementation flow of the data processing method of the embodiment of the present invention;

FIG. 2 is a schematic diagram of the composition and structure of a data processing device according to an embodiment of the present invention;

FIG. 3 is a second schematic diagram of the composition and structure of a data processing device according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a specific implementation process of online audit and repair according to an embodiment of the present invention;

FIG. 5 is a third schematic diagram of the composition and structure of the data processing device according to the embodiment of the present invention.

detailed description

In related technologies, in the case where multiple data sources need to cooperate to realize a certain service, some data in these mutually coordinated data sources need to be consistent. In order to ensure the integrity and consistency of data, it is usually necessary to perform data audit and repair on the data in multiple data sources.

For example, for the Migu reading application (APP, Application), the book data is stored in the small library, the main library and the content library respectively; among them, the small library and the main library are for book authors and editors, and the content library is user-oriented. When the editors and reviewers pass the review of book A in the small library, modify the value of the status data of the book A in the small library to 1 (that is, the status of the book is "on the shelf"), and change the value of the book A in the main library and the content library The value of the state data of A is synchronously modified to 1 to ensure that the user can read the book A; when the copyright of the book A expires, the editors and reviewers need to modify the value of the state data of the book A in the small library to 0 (ie The book status is "off the shelf"), and the value of the status data of the book A in the main library and the content library is synchronously changed to 0 to ensure that the user can no longer read the book A. In this process, it is necessary to perform data audit on the state data of the book A in the small library, the main library and the content library to determine whether the state data is consistent in the small library, the main library and the content library. Data repair, to ensure the consistency of the state data in the small library, the main library and the content library.

At present, in order to avoid affecting the normal operation of the data source, a backup database is established for each data source, and the data in the data source is synchronously backed up to the corresponding backup database according to the preset cycle (generally in days), and then Perform data audit based on the data in each backup database, and then perform data restoration on the data in the data source according to the audit results.

However, since the data audit operation is performed in the backup environment on a daily basis, that is, the data audit operation is performed offline, resulting in low real-time performance of the data audit and affecting the accuracy of the data audit. When data a is audited, the data a in the data source may have changed. Therefore, the data audit results based on the backup database are inaccurate and cannot truly reflect the data consistency in each data source. Then, according to the backup database According to the obtained audit results, after the data in the data source is repaired, the consistency of the data in each data source still cannot be ensured.

For example, still taking the aforementioned Migu Reading APP as an example, establish backup databases a, b, and c for the small library, main library, and content library respectively, and synchronize the book data in the small library to the backup databases a, b, and c on a daily basis. Synchronize the book data in the main library to the backup database b, and synchronize the book data in the content library to the backup database c. Suppose, when synchronizing the book data in the small library to the backup database a, the value of the state data of book A in the small library is "1", therefore, the value of the state data of book A in the backup database a is also "1" ". However, during the process of data audit based on the backup library, the value of the status data of book A in the small library is changed to "0". When the data audit is performed based on the backup databases a, b and c, the audit result is: the value of the state data of book A in the backup database a is also "1", and the value of the state data of book A in the backup database b is "0" , the data is inconsistent. According to the audit results, the value of the status data of book A in the main library corresponding to the backup database b needs to be synchronously modified to "1". However, during the data audit process, the value of the status data of book A in the small library has been changed to "0", therefore, after data restoration is performed on the main library according to the audit results, the status of book A in the small library and the main library is still Data has inconsistent values. At the same time, establishing a backup database for each data source also requires additional resources and cost input.

Based on this, in the embodiment of the present invention, the data to be audited is obtained online from at least two data sources; based on the preset audit rules, the data to be audited is audited; the data that fails the audit is determined as the data to be repaired, and , repairing the data to be repaired according to a preset repair sequence.

In order to understand the characteristics and technical contents of the embodiments of the present invention in more detail, the implementation of the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. The attached drawings are only for reference and description, and are not intended to limit the present invention.

As shown in Figure 1, the data processing method of the embodiment of the present invention is described in detail, including the following steps:

Step 101: Online acquisition of data to be audited from at least two data sources.

In an embodiment, the online acquisition of the data to be audited from at least two data sources includes: determining pre-stored metadata corresponding to the data to be audited; according to the metadata, from the at least two The data to be audited corresponding to the metadata is obtained from a data source.

In actual application, according to the metadata, the data to be audited corresponding to the metadata is obtained from the at least two data sources, specifically, it may be: according to the pre-stored metadata, determining the metadata Data identification information; according to the determined identification information of the metadata, obtain the data to be audited corresponding to the identification information from the at least two data sources.

For example, when the data to be audited is book data, it is assumed that the identification information of the pre-stored metadata is: "ID=123456789 (indicates unique identification), TYPE=1 (indicates books), UPLOADDATE=2017-05-07 17 : 06:00 (indicates the collection time), DOFLAG=0 (indicates that the processing status is to be audited)", extract the data to be audited corresponding to "ID=123456789" from the small library, the main library and the content library respectively according to the identification information . The data to be audited may include status data, volume data, chapter data and so on.

In an embodiment, the method further includes: acquiring the metadata from the at least two data sources according to preset collection rules; and/or receiving the metadata pushed by the at least two data sources data.

Wherein, the preset collection rule may be to check the data logs of the at least two data sources according to a preset time period, and obtain the metadata from the at least two data sources according to the data logs.

It should be noted that the preset collection rules may be set according to actual conditions, and are not specifically limited here.

In actual application, the metadata is obtained from the at least two data sources according to preset collection rules; at the same time, more data to be audited can be obtained by receiving the metadata pushed by the at least two data sources , not only expand the scope of audit, but also improve the accuracy of audit to a certain extent.

In practical applications, the collected metadata can also be deduplicated to remove duplicate data collected, which can improve the efficiency of subsequent audits to a certain extent.

In practical applications, the data format of the collected metadata can be converted into a standard format before being stored. The standard format can be composed of a series of parameters such as audit KEY value, data type, collection time, processing status, etc.; wherein, the audit KEY value can represent the unique identification of the data to be audited. For different business scenarios, the standard format of metadata can be different.

Step 102: Audit the data to be audited based on preset audit rules.

In an embodiment, the auditing of the data to be audited based on preset audit rules includes: judging whether the data to be audited is consistent in the at least two data sources according to the preset audit rules ; If consistent, it is determined that the data to be audited has passed the audit; otherwise, it is determined that the data to be audited has not passed the audit.

In actual application, different business scenarios can correspond to different preset audit rules.

For example, when the data to be audited is the book data to be audited in the main library, the content library, and the small library, and the data of the books to be audited in the main library and the content library are obtained synchronously from the data of the books to be audited in the small library, the default The audit rules can be: respectively audit the data of the books to be audited in the main library and the content library and the data of the books to be audited in the small library, that is, to determine whether the data of the books to be audited in the main library and the content library are consistent with the data of the books to be audited in the small library The audit book data is consistent.

The audit process can be specifically as follows: Assume that the value of the state data of the book data to be audited (book A) in the small library is "0" (indicating the off-shelf status and cannot be read), and the book data to be audited in the small library (chapter a) The value of the status data is also "0" (indicating off-shelf status, unreadable). By auditing the book data to be audited (book A) and the book data to be audited (chapter a) in the small library and the content library, the audit results show that the status data of the book data to be audited (book A) in the content library is "1" (indicating the status of being put on the shelf and can be read), is inconsistent with the data of the book to be audited (book A) in the small library; the value of the status data of the book data to be audited (chapter a) in the content library is also "1" (indicating Shelf status, readable), and the data of books to be audited (chapter a) in the small library are also inconsistent.

For example, in the bank transfer business, the default audit rule can be: check the charging data in the charging data source and the deduction data in the deduction data source corresponding to a transfer business, and judge the charging data in the charging data source. Whether the deduction data in the deduction data source is consistent.

Step 103: Determine the data that fails the audit as data to be repaired, and repair the data to be repaired according to a preset repair sequence.

In actual application, the preset restoration order can be obtained according to the association relationship among multiple data to be audited.

For example, the relationship among multiple data to be audited may be: the data to be audited (chapter a) is a chapter in the data to be audited (book A). According to the association relationship, the obtained preset repair order can be: first repair the data to be audited (book A), and then repair the data to be audited (chapter a).

The association relationship between multiple data to be audited may also be: the data to be audited (episode b) is an episode in the data to be audited (tv series B). According to the association relationship, the obtained preset restoration sequence may be: first restore the data to be audited (television series B), and then restore the data to be audited (episode b).

Still taking the above data to be audited as the main library, the content library and the book data to be audited in the small library as an example, the repair process can be as follows: first repair the book data to be audited (book A), and then restore the book data to be audited in the content library Change the value of the status data of the data (book A) to "0" to ensure that in the user-oriented content library, all the data of book A cannot be provided for users to read; Change the value of the state data of the book data to be audited (chapter a) to "0" to confirm the consistency of the book data to be audited in the small library and the content library.

In an embodiment, the method further includes: judging whether the data to be repaired is successfully repaired; when it is determined that the data to be repaired has not been successfully repaired, repairing the data to be repaired again; When the number of data repairs is greater than the preset number of repairs, an alarm message is generated; the alarm message indicates that the data to be repaired cannot be repaired.

In one embodiment, the judging whether the data to be repaired is successfully repaired includes: judging whether the data to be repaired after repair is consistent in the at least two data sources; if they are consistent, determining whether the data to be repaired is consistent The repair is successful; otherwise, it is determined that the data to be repaired is not successfully repaired.

In the data processing method provided by the embodiment of the present invention, the data to be audited is acquired online from at least two data sources; based on the preset audit rules, the data to be audited is audited; the data that fails the audit is determined as the data to be repaired, And, the data to be repaired is repaired according to the preset repair sequence, so as to effectively realize online audit and timely repair of data, and improve the efficiency and accuracy of data audit and data repair.

Furthermore, the embodiments of the present invention can be used for different business scenarios, and only need to determine the preset audit rules and preset repair sequences for different business scenarios, which has good versatility.

Based on the data processing method provided by each embodiment of the present application, the present application also provides a data processing device, as shown in FIG. 2 , the device includes: an acquisition module 21, an audit module 22, and a repair module 23; wherein,

An acquisition module 21, configured to acquire data to be audited online from at least two data sources;

An audit module 22, configured to audit the data to be audited based on preset audit rules;

The repair module 23 is configured to determine the data that fails the audit as data to be repaired, and repair the data to be repaired according to a preset repair sequence.

In an embodiment, the obtaining module 21 is specifically configured to determine pre-stored metadata corresponding to the data to be audited; according to the metadata, obtain the metadata related to the metadata from the at least two data sources. The data to be audited corresponding to the data.

In actual application, according to the metadata, the data to be audited corresponding to the metadata is obtained from the at least two data sources, specifically, it may be: according to the pre-stored metadata, determining the metadata Data identification information; according to the determined identification information of the metadata, obtain the data to be audited corresponding to the identification information from the at least two data sources.

In an embodiment, the device further includes: an acquisition module; wherein,

The collection module is configured to obtain the metadata from the at least two data sources according to preset collection rules; and/or receive the metadata pushed by the at least two data sources.

Wherein, the preset collection rule may be to check the data logs of the at least two data sources according to a preset time period, and obtain the metadata from the at least two data sources according to the data logs.

In one embodiment, the audit module 22 is specifically configured to determine whether the data to be audited is consistent in the at least two data sources according to the preset audit rules; if they are consistent, determine whether the data to be audited The data passes the audit; otherwise, it is determined that the data to be audited has not passed the audit.

In actual application, different business scenarios can correspond to different preset audit rules.

In one embodiment, the device further includes: an alarm module; wherein,

The alarm module is used to judge whether the data to be repaired is successfully repaired; when it is determined that the data to be repaired is not successfully repaired, the data to be repaired is repaired again; when the number of repairs to the data to be repaired is greater than When the repair times are preset, an alarm message is generated; the alarm message indicates that the data to be repaired cannot be repaired.

In one embodiment, the alarm module is specifically configured to determine whether the repaired data to be repaired is consistent in the at least two data sources; if consistent, determine that the repair of the data to be repaired is successful; otherwise, determine The data to be repaired is not successfully repaired.

It should be noted that: when the data processing device provided in the above-mentioned embodiments performs data processing, the division of the above-mentioned program modules is used as an example for illustration. In practical applications, the above-mentioned processing allocation can be completed by different program modules according to needs. That is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the data processing device and the data processing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process thereof is detailed in the method embodiments, and will not be repeated here.

In practical applications, the acquisition module 21 is realized by a network interface positioned on the data processing device; Processor (MPU, Micro Processor Unit), digital signal processor (DSP, Digital Signal Processor), or Field Programmable Gate Array (FPGA, Field Programmable GateArray) and other implementations.

FIG. 3 is a schematic structural diagram of a data processing device of the present invention. The data processing device 300 shown in FIG. Various components in the data processing device 300 are coupled together through the bus system 305 . It can be understood that the bus system 305 is used to realize connection and communication between these components. In addition to the data bus, the bus system 305 also includes a power bus, a control bus and a status signal bus. However, the various buses are labeled as bus system 305 in FIG. 3 for clarity of illustration.

Wherein, the user interface 303 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch panel or a touch screen, and the like.

It can be understood that the memory 302 may be a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Wherein, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read-Only Memory), Only Memory), Electrically Erasable Programmable Read-Only Memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), Magnetic Random Access Memory (FRAM, ferromagnetic random access memory), Flash Memory (Flash Memory), Magnetic Surface Memory , CD, or CD-ROM (CD-ROM, Compact Disc Read-Only Memory); the magnetic surface storage can be disk storage or tape storage. The volatile memory may be random access memory (RAM, Random Access Memory), which is used as an external cache. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM, Static Random Access Memory), Synchronous Static Random Access Memory (SSRAM, Synchronous Static Random Access Memory), Dynamic Random Access Memory Memory (DRAM, Dynamic Random Access Memory), Synchronous Dynamic Random Access Memory (SDRAM, Synchronous Dynamic Random Access Memory), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM, Double Data Rate Synchronous Dynamic Random Access Memory), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), Synchronous Link Dynamic Random Access Memory (SLDRAM, SyncLink Dynamic Random Access Memory), Direct Memory Bus Random Access Memory (DRRAM, Direct Rambus Random Access Memory) . The memory 302 described in the embodiments of the present invention is intended to include, but not be limited to, these and any other suitable types of memory.

The memory 302 in the embodiment of the present invention is used to store various types of data to support the operation of the data processing device 300 . Examples of these data include: any computer program for operating on the data processing device 300, such as operating system 3021 and application program 3022; wherein, the operating system 3021 includes various system programs, such as framework layer, core library layer, driver layer etc., to implement various basic services and handle hardware-based tasks. The application program 3022 may include various application programs for implementing various application services. The program for realizing the method of the embodiment of the present invention may be included in the application program 3022 .

The methods disclosed in the foregoing embodiments of the present invention may be applied to the processor 301 or implemented by the processor 301 . The processor 301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 301 or instructions in the form of software. The aforementioned processor 301 may be a general-purpose processor, a digital signal processor, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The processor 301 may implement or execute various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the methods disclosed in the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, and the storage medium is located in the memory 302. The processor 301 reads the information in the memory 302, and completes the steps of the foregoing method in combination with its hardware.

Specifically, the embodiment of the present invention also provides a data processing device, as shown in FIG. program,

Wherein, when the processor 301 is used to run the computer program, it performs the following operations:

Obtain online data to be audited from at least two data sources; audit the data to be audited based on preset audit rules; determine data that fails the audit as data to be repaired, and, according to a preset repair sequence, perform an audit on the data to be audited; Repair the data to be repaired.

In an embodiment, the processor 301 is further configured to perform the following operations when running the computer program: determine pre-stored metadata corresponding to the data to be audited; according to the metadata, from the at least The data to be audited corresponding to the metadata is acquired from two data sources.

In an embodiment, the processor 301 is further configured to perform the following operations when running the computer program: acquire the metadata from the at least two data sources according to a preset acquisition rule; and/or, The metadata pushed by the at least two data sources is received.

In an embodiment, the processor 301 is further configured to perform the following operations when running the computer program: according to the preset audit rules, determine whether the data to be audited is consistent in the at least two data sources ; If consistent, it is determined that the data to be audited has passed the audit; otherwise, it is determined that the data to be audited has not passed the audit.

In an embodiment, the processor 301 is further configured to perform the following operations when running the computer program: determine whether the data to be repaired is successfully repaired; when it is determined that the data to be repaired is not successfully repaired, The data to be repaired is repaired again; when it is determined that the number of repairs is greater than the preset number of repairs, an alarm message is generated; the alarm message indicates that the data to be repaired cannot be repaired.

In one embodiment, the processor 301 is further configured to perform the following operations when running the computer program: determine whether the repaired data to be repaired is consistent in the at least two data sources; if consistent, determine The data to be repaired is successfully repaired; otherwise, it is determined that the data to be repaired is not successfully repaired.

Based on the data processing methods provided by the embodiments of the present application, the present application also provides a computer-readable storage medium, as shown in FIG. 3 , the computer-readable storage medium may include: a memory 302 for storing computer programs, the The computer program can be executed by the processor 301 of the data processing device 300 to complete the steps described in the foregoing method. The computer-readable storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM.

Specifically, the computer-readable storage medium provided by the embodiment of the present invention has a computer program stored thereon, and when the computer program is run by a processor, the following operations are performed:

Obtain online data to be audited from at least two data sources; audit the data to be audited based on preset audit rules; determine data that fails the audit as data to be repaired, and, according to a preset repair sequence, perform an audit on the data to be audited; Repair the data to be repaired.

In an embodiment, when the computer program is executed by the processor, the following operations are further performed: determining pre-stored metadata corresponding to the data to be audited; Obtain the data to be audited corresponding to the metadata.

In an embodiment, when the computer program is executed by the processor, the following operations are further performed: according to preset collection rules, obtain the metadata from the at least two data sources; and/or receive the at least The metadata pushed by the two data sources.

In an embodiment, when the computer program is run by the processor, the following operations are further performed: according to the preset audit rules, it is judged whether the data to be audited is consistent in the at least two data sources; if they are consistent, Then it is determined that the data to be audited has passed the audit; otherwise, it is determined that the data to be audited has not passed the audit.

In an embodiment, when the computer program is run by the processor, the following operations are further performed: judging whether the data to be repaired has been successfully repaired; when it is determined that the data to be repaired has not been successfully repaired, performing Repairing again; when the number of repairs to the data to be repaired is greater than the preset number of repairs, an alarm message is generated; the alarm message indicates that the data to be repaired cannot be repaired.

In one embodiment, when the computer program is run by the processor, the following operations are further performed: judging whether the repaired data to be repaired is consistent in the at least two data sources; if they are consistent, determine the data to be repaired The data is repaired successfully; otherwise, it is determined that the data to be repaired is not successfully repaired.

The implementation process and principles of the present invention in practical applications will be described in detail below by taking the implementation of data processing as a specific example.

FIG. 4 is a schematic diagram of a specific implementation process for realizing online audit and repair according to an embodiment of the present invention. In combination with the schematic diagram of the data processing device shown in FIG. 5 , the specific implementation process includes the following steps:

Step 401: The collection module collects metadata online from the at least two data sources.

Collection methods include active collection and passive collection;

The active collection method may be: checking the data logs of the at least two data sources according to a preset time period, and obtaining the metadata from the at least two data sources according to the data logs.

The passive collection method can be as follows: the collection set module can be set with an application programming interface (API, Application Programming Interface) for each data source to call, and each data source actively pushes the metadata corresponding to itself and the data to be audited to the in the acquisition module.

The collection module can also perform deduplication operations on the collected metadata corresponding to the data to be audited, and remove the collected duplicate data, so as to improve the follow-up audit efficiency to a certain extent.

Step 402: the collection module pushes the metadata to the acquisition module.

The acquisition module may immediately push the metadata to the acquisition module for processing after storing the metadata; it may also push the metadata stored within the preset period to the acquisition module in batches every preset period of time Do preprocessing.

Step 403: The acquiring module acquires the data to be audited corresponding to the metadata from the at least two data sources according to the pre-stored metadata, and pushes the data to be audited to the auditing module.

Step 404: The audit module audits the data to be audited based on preset audit rules, and pushes the data to be repaired to the repair module.

When it is determined that the data to be audited is consistent in the at least two data sources, the audit module pushes the metadata corresponding to the audited data to the collection module, so that the collection module deletes the previously stored metadata; or , so that the acquisition module modifies the processing status in the metadata to "audit completed".

When it is determined that the data to be audited is inconsistent in the at least two data sources, the audit module determines the data that fails the audit as data to be repaired, and pushes the data to be repaired to the repair module for real-time repair.

The process of auditing can be specifically as follows: Assume that the value of the state data of the data to be audited in the small library (such as book A) is "0" (indicating the off-shelf status and cannot be read), and the data to be audited in the small library (such as chapter a) The value of the status data is also "0" (indicating off-shelf status, unreadable). By auditing the data to be audited (such as book A) and the data to be audited (such as chapter a) in the small library and content library, the audit results show that the status data of the data to be audited in the content library (such as book A) is "1" (indicating on-shelf status, readable), inconsistent with the data to be audited in the small library; the value of the status data of the data to be audited in the content library (such as chapter a) is also "1" (indicating on-shelf status, readable ), which is inconsistent with the data to be audited in the small library.

Step 405: The repair module repairs the data to be repaired according to a preset repair sequence.

The repair process can be specifically: first repair the data to be audited (such as book A), and change the value of the state data of the data to be audited in the content library (such as book A) to "0", so as to ensure that in the user-oriented content library, All the data of book A cannot be provided to users for reading; then repair the data to be audited (such as chapter a), change the value of the status data of the data to be audited in the content library (such as chapter a) to "0", and determine the small library and Data consistency to be audited in the content library.

If the restoration is not performed according to the preset restoration order, but the data to be audited (such as chapter a) is repaired first, since the restoration takes a certain amount of time, during the time period when the data to be audited (such as chapter a) is repaired, the content The value of the state data of the data to be audited in the library (such as book A) is still "1", that is, within this time period, the user-oriented content library can still provide users with all the data of book A as readable, resulting in data mistake.

Step 406: Judging whether the data to be repaired is successfully repaired; when it is determined that the data to be repaired is not successfully repaired, repairing the data to be repaired again. When the times of repairing the data to be repaired are greater than the preset times of repairing, an alarm message is generated.

Wherein, the alarm message indicates that the data to be repaired cannot be repaired. Relevant personnel can take other ways to repair in time to avoid business failures caused by inconsistencies in data in different data sources.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention.

Claims (14)

1. A method of data processing, the method comprising:

acquiring data to be audited from at least two data sources on line;

auditing the data to be audited based on a preset auditing rule;

determining data which is not passed through the audit as data to be repaired, and repairing the data to be repaired according to a preset repairing sequence.

2. The method of claim 1, wherein the online obtaining of data to be audited from at least two data sources comprises:

determining prestored metadata corresponding to the data to be audited;

and acquiring the data to be audited corresponding to the metadata from the at least two data sources according to the metadata.

3. The method of claim 2, further comprising:

acquiring the metadata from the at least two data sources according to a preset acquisition rule;

and/or receiving the metadata pushed by the at least two data sources.

4. The method of claim 1, wherein auditing the data to be audited based on preset auditing rules comprises:

judging whether the data to be audited are consistent in the at least two data sources or not according to the preset auditing rule;

if the data to be audited are consistent, determining that the data to be audited pass auditing; otherwise, determining that the data to be audited does not pass the audit.

5. The method of claim 1, further comprising:

judging whether the data to be repaired is successfully repaired;

when the data to be repaired is determined to be unsuccessfully repaired, repairing the data to be repaired again;

when the repairing times of the data to be repaired are larger than the preset repairing times, generating an alarm message; the alarm message represents that the data to be repaired cannot be repaired.

6. The method according to claim 5, wherein the determining whether the data to be repaired is successfully repaired comprises:

judging whether the repaired data to be repaired is consistent in the at least two data sources;

if the data to be repaired are consistent, determining that the data to be repaired is successfully repaired; otherwise, determining that the data to be repaired is not repaired successfully.

7. A data processing apparatus, characterized in that the apparatus comprises:

the system comprises an acquisition module, a checking module and a processing module, wherein the acquisition module is used for acquiring data to be audited from at least two data sources on line;

the auditing module is used for auditing the data to be audited based on a preset auditing rule;

and the repairing module is used for determining the data which is not passed through the audit as the data to be repaired and repairing the data to be repaired according to a preset repairing sequence.

8. The apparatus of claim 7,

the acquisition module is specifically used for determining metadata which is stored in advance and corresponds to the data to be audited; and acquiring the data to be audited corresponding to the metadata from the at least two data sources according to the metadata.

9. The apparatus of claim 8, further comprising:

the acquisition module is used for acquiring the metadata from the at least two data sources according to a preset acquisition rule; and/or receiving the metadata pushed by the at least two data sources.

10. The apparatus of claim 7,

the auditing module is specifically used for judging whether the data to be audited are consistent in the at least two data sources according to the preset auditing rule; if the data to be audited are consistent, determining that the data to be audited pass auditing; otherwise, determining that the data to be audited does not pass the audit.

11. The apparatus of claim 7, further comprising:

the alarm module is used for judging whether the data to be repaired is successfully repaired; when the data to be repaired is determined to be unsuccessfully repaired, repairing the data to be repaired again; when the repairing times of the data to be repaired are larger than the preset repairing times, generating an alarm message; the alarm message represents that the data to be repaired cannot be repaired.

12. The apparatus of claim 11,

the alarm module is specifically configured to determine whether the repaired data to be repaired is consistent in the at least two data sources; if the data to be repaired are consistent, determining that the data to be repaired is successfully repaired; otherwise, determining that the data to be repaired is not repaired successfully.

13. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.

14. A data processing apparatus, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor;

wherein the processor is adapted to perform the steps of the method of any one of claims 1 to 6 when running the computer program.