CN102073912A

CN102073912A - Data quality control method, device and system

Info

Publication number: CN102073912A
Application number: CN200910223541XA
Authority: CN
Inventors: 孟繁力; 王晓伟; 王雅文; 李宏昌
Original assignee: China Mobile Group Heilongjiang Co Ltd
Current assignee: China Mobile Group Heilongjiang Co Ltd
Priority date: 2009-11-23
Filing date: 2009-11-23
Publication date: 2011-05-25

Abstract

The invention discloses a data quality control method, device and system. The data quality control method includes: verifying the operation steps during data processing, and generating a verification file; comparing and analyzing the verification files of the operation steps, and generating an error message and sending it to the ELT control device when there is an abnormality in the data ; The ELT control interrupts the ongoing operating steps based on the error message. In the data quality control method, device and system of the present invention, by performing verification in each operation step of data processing and comparing and analyzing the verification results, when data abnormality occurs, the ELT control device is notified to interrupt the ongoing operation, so that Each operation step in the data processing process has carried out quality control, which improves the reliability of the data.

Description

Data quality control method, Apparatus and system

Technical field

The present invention relates to a kind of business support technology, relate in particular to a kind of data quality control method, Apparatus and system.

Background technology

Along with operation analysis system in the important effect of play more and more aspect business decision and the marketing management, the application that it carried is also abundant day by day, the thing followed is that data volume that system handles increases day by day, is accompanied by the continuous decline of the quality of data.

In the prior art, it mainly is guiding with the function, pressure under the pressure of system's promptness, substantially adopt program daily record monitor mode based on the control of the quality of data, situation such as just having only program run to report an error to withdraw from just have alarm produces, control based on the quality of data also is spot style, just the promptness of individual interfaces has been done verification, lacks the control device of overall data consistance and accuracy.

The shortcoming of prior art is:

1, can not wholely control BI system data quality

Existing system does not have the Data Quality Control Techniques based on data flow of the overall situation, therefore the accuracy of data is not high, the control technology of taking at present mainly is based on program and database output journal, whether determining program reports an error is withdrawed from, data problem scope that can control is narrow and small relatively, and the unusual fluctuations of data source etc. are not had calibration technology;

2, existing quality of data method of calibration influences system handles efficient

Present method checking data based on program and database output journal, checking routine is an intermediate steps of data processing link, that is to say have only checking routine success after, just can carry out subsequent treatment, such method of calibration greatly reduces system handles efficient, makes that also making the overall arrangement for data quality control becomes impossible event;

3, original data quality problem is safeguarded fully based on code level, safeguards complicated

Original data quality control does not have visual output, only deposits in the program journal file, and therefore do analysis to problem will go rule of thumb to judge which program reports an error, and watches daily record then and further analyzes, and does not have automatic analytic function.

Summary of the invention

The objective of the invention is to, a kind of data quality control method, Apparatus and system are provided, can carry out quality control, improve the reliability of data each operation steps of data processing procedure.

For achieving the above object, according to an aspect of the present invention, provide a kind of data quality control method, comprising: in data handling procedure, operation steps is carried out verification, generate verification file; The verification file of operation steps is analyzed, and when data exception occurring, generation error information also is sent to the ELT control device; Described ELT control device interrupts the ongoing operation step according to described error message.

For achieving the above object, according to another aspect of the present invention, provide a kind of quality control apparatus, comprising: the verification module, be used for operation steps being carried out verification in data handling procedure, generate verification file; Analysis module is used for the verification file of operation steps is analyzed, and when data exception occurring, generation error information also is sent to the ELT control device.

For achieving the above object, according to another aspect of the present invention, provide a kind of data quality control system, comprising: data processing equipment is used for handling receiving data from data source; Quality control apparatus carries out verification to operation steps in data handling procedure, generate verification file, and the verification file of operation steps is analyzed, and when data exception occurring, generation error information also is sent to the ELT control device; The ELT control device interrupts the ongoing operation step according to described error message.

Data quality control method of the present invention, Apparatus and system, by carrying out verification in each operation steps of data processing and check results being compared, when data exception occurring, notice ELT control device interrupts ongoing operation, make each operation steps of data handling procedure all carry out quality control, improved the reliability of data.In addition, because the process of quality control and the process of data processing are carried out synchronously, do not having under the unusual situation, the ELT control device can not interrupt carries out data processing, has farthest improved treatment effeciency.

Description of drawings

Fig. 1 is a data quality control method embodiment process flow diagram of the present invention;

Fig. 2 is an operation analysis system data handling procedure embodiment process flow diagram of the present invention;

Fig. 3 is quality control apparatus example structure figure of the present invention;

Fig. 4 is a data quality control system embodiment structural drawing of the present invention.

Embodiment

Carrying out quality control with the flow chart of data processing to operation analysis system is example, and the present invention is described in detail in conjunction with the accompanying drawings.

Method embodiment

Fig. 1 is a data quality control method embodiment process flow diagram of the present invention.As shown in Figure 1, present embodiment may further comprise the steps:

Step 102, quality control apparatus are carried out verification to each operation steps in data handling procedure, generate verification file; Quality control apparatus also can carry out verification to the operation steps in the checklist according to predefined checklist; As shown in Figure 2, the flow chart of data processing of operation analysis system comprises following operation steps: data source, extraction, conversion, loading, secondary are changed, are gathered and report form processing, also comprise between data source operation and the extraction operation: derivation, interface generate, interface transmits the corresponding generation verification file of each operation steps;

This verification file is the running log of each operation steps output of data handling procedure, comprising: the file size that data source numbering, mission number, each operation steps are handled, line number, critical field length, critical field value or the like; The verification file form is as shown in table 1 below;

Table 1 verification file form

Step 104, quality control apparatus is analyzed the verification file of each operation steps, judges whether to occur data exception, if, execution in step 106; If not, continue to carry out data handling procedure; According to actual needs, quality control apparatus is set the configuration information that the verification file to each operation steps is analyzed as required; Quality control apparatus extracts corresponding data according to this configuration information and the data of extracting is compared from verification file;

The consistance of verification msg can compare to information such as the file size in the checking file, file line number, critical field values if desired; For example, in the process that ticket is handled, require the data recording bar number must be consistent, as inconsistent then be abnormal conditions, but also some requires to get final product in the scope that error allows, so whether concrete comparison result within pre-set threshold range; Configuration file is as shown in the table,

Table 2 consistency checking configuration information

The data source numbering	The file comparison	File size	The file line number	The critical field value
					00001	A/B	1	0	0
00001	B/C	0	0	0

When a certain when being set to 1 in the configuration information, then do not compare this, think that this meets the requirements; When being set to 0, then compare this project; Rule from last table as can be seen, in the time of comparison AB verification file, only compare back two, comparison method is that the corresponding field data in the verification file are done computing, as the record line number ' 598723 ' among the verification file A is compared with the record line number among the verification file B ' 598723 ', finds unanimity as a result, then corresponding project comparison result puts 0, at last three results are got union, if there is one to be 1, then end product is 1;

The undulatory property of verification msg can compare to critical field value and the pre-set threshold in the verification file if desired; For example, for indexs such as incomes, the income of every day all is in fixing interval, too much or very fewly all assert it is system problem; Configuration file is as shown in the table,

Table 3 undulatory property checking configuration information

The data source numbering	The comparison file	Critical field value maximum	Critical field value minimum
				00001	A	190	50
00001	B	200	60

For verification file A, its critical field value is 182 to the maximum, and within 50～190 scope, critical field value minimum is 29, and not in above-mentioned scope, so verification file A can not exist unusual by this undulatory property checking;

The legitimacy of verification msg can compare to the critical field length in the verification file or the data type of critical field if desired; Configuration file is as shown in the table,

Table 4 legitimate verification configuration information

The data source numbering	The comparison file	Critical field length	The critical field type
				00001	A	11	number

For example user's phone number if its length is not 11, then is abnormal conditions; Its data type is done verification, if in the critical field data letter is arranged, then explanation is not the number type, data exception;

The rationality of verification msg can compare to the file line number in the verification file if desired; For example, for the different business data, its normal business development amount is arranged all, if the portfolio surge then is abnormal conditions, promptly the file line number in verification file surpasses specified threshold, then data exception; Configuration file is as shown in the table,

Table 5 soundness verification configuration information

The data source numbering	The comparison file	File line number maximal value	File line number minimum value
				00001	A	10000000	50000

For verification file A, its file line number 598723 is within above-mentioned file line number maximal value and minimum value scope, and is no abnormal;

Step 106, quality control apparatus generation error information is sent to the ELT control device;

For example, the form of error message is as shown in the table,

The data source numbering

Mission number

Comparison result

00001	253	0
			00001	254	1

Step 108, the ELT control device interrupts the ongoing operation step according to error message;

After the ELT control device obtains above-mentioned error message, will judge comparison result, when the result equals " 0 ", program is normally carried out, when the result equals " 1 ", the ELT engine task and the follow-up dependence task that will stop makeing mistakes so.Feedback result such as No. 254 tasks is 1, the ELT control device has obtained by reading allocation list so, 255,256, No. 257 tasks are follow-up work, and know that current moving of task is No. 257 tasks, the ELT control device then sends the order that stops No. 257 tasks so, and stops follow-up work.

The data quality control method of present embodiment, by carrying out verification in the operation steps of data processing and check results being compared, when data exception occurring, notice ELT control device interrupts ongoing operation, make the operation steps of data processing procedure has all been carried out quality control, improved the reliability of data.In addition, because the process of quality control and the process of data processing are carried out synchronously, do not having under the unusual situation, the ELT control device can not interrupt carries out data processing, has farthest improved treatment effeciency.

Device embodiment

Fig. 3 is quality control apparatus example structure figure of the present invention.As shown in Figure 3, present embodiment comprises:

Verification module 302 is used in data handling procedure operation steps being carried out verification, generates verification file; Analysis module 304 is used for the verification file of each operation steps is analyzed, and when data exception occurring, generation error information also is sent to the ELT control device.

Preferably, present embodiment also comprises: configuration module 306, be used to set the operation steps that need carry out verification, and generate checklist; Verification module 302 is carried out verification according to described checklist to the operation steps in the checklist, generates verification file.For some data handling procedures, only the operation steps of wherein easily makeing mistakes is carried out verification, improved the efficient of quality control.

In addition, configuration module 306 is also set the configuration information that the verification file to each operation steps is analyzed as required; Analysis module 304 extracts corresponding data according to configuration information from described verification file, and the data of extracting are compared.The concrete form of configuration information and the detailed process that verification file compares is described in detail in said method embodiment according to configuration information does not repeat them here.

The quality control apparatus of present embodiment, by carrying out verification in the operation steps of data processing and check results being compared, when data exception occurring, notice ELT control device interrupts ongoing operation, make the operation steps of data processing procedure has all been carried out quality control, improved the reliability of data.In addition, because the process of quality control and the process of data processing are carried out synchronously, do not having under the unusual situation, the ELT control device can not interrupt carries out data processing, has farthest improved treatment effeciency.

System embodiment

Fig. 4 is a data quality control system embodiment structural drawing of the present invention.As shown in Figure 4, present embodiment comprises:

Data processing equipment 402 is used for handling receiving data from data source;

Quality control apparatus 404 carries out verification to operation steps in data handling procedure, generate verification file, and the verification file of operation steps is analyzed, and when data exception occurring, generation error information also is sent to the ELT control device;

ELT control device 406 interrupts the ongoing operation step according to described error message.

Wherein, data processing equipment is in data handling procedure, and every complete operation steps generates and finishes sign and be sent to the ELT control device; The ELT control device can have been transferred next operation steps according to finishing sign.The ELT control device is executed the task successively by pre-set mission number, as 253 254 255... in the table 1.The ELT control device will at first transfer No. 253 tasks to carry out so, when 253 tasks are finished, can finish sign by one of anti-Kui Give ELT control device, the ELT control device obtains the information after 253 tasks are finished so, will transfer 254 tasks of waiting for, guarantee carrying out successively of task like this.

Preferably, present embodiment also comprises: exhibiting device 408 as a result, are used to show the check information and/or the error message of described each operation steps.

The data quality control system of present embodiment, by carrying out verification in the operation steps of data processing and check results being compared, when data exception occurring, notice ELT control device interrupts ongoing operation, make the operation steps of data processing procedure has been carried out quality control, improved the reliability of data.In addition, because the process of quality control and the process of data processing are carried out synchronously, do not having under the unusual situation, the ELT control device can not interrupt carries out data processing, has farthest improved treatment effeciency.In addition, the data quality control system of present embodiment can show check information and/or error message by demonstration device as a result, maintains easily personnel's inquiry and to the processing of abnormal conditions.

It should be noted that: above embodiment is only unrestricted in order to explanation the present invention, and the present invention also is not limited in above-mentioned giving an example, and all do not break away from the technical scheme and the improvement thereof of the spirit and scope of the present invention, and it all should be encompassed in the claim scope of the present invention.

Claims

1. A data quality control method, characterized in that, comprising:

Verify the operation steps during data processing and generate verification files;

Perform comparative analysis on the verification files of the operation steps, and generate an error message and send it to the ELT control device when the data is abnormal;

The ELT control interrupts an ongoing operating step based on the error message.

2. The data quality control method according to claim 1, wherein the operation of verifying the operation steps in the data processing process comprises:

According to the preset check table, check the corresponding operation steps in the check table.

3. The data quality control method according to claim 1 or 2, wherein the verification file is an operation log output by each operation step in the data processing process.

4. The data quality control method according to claim 3, wherein the verification file includes: data source number, task number, file size processed by each operation step, number of lines, length of important fields, key fields value range.

5. The data quality control method according to claim 4, wherein the operation of comparing and analyzing the verification file of the operation step comprises:

The quality control device sets the configuration information for comparative analysis of the verification files of the operation steps as required;

The quality control device extracts corresponding data from the verification file according to the configuration information and compares and analyzes the extracted data.

6 . The data quality control method according to claim 1 , wherein the error information includes: detailed information on the operation steps of abnormal data transmission and data gaps. 7 .

7. A quality control device, characterized in that, comprising:

A verification module is used to verify the operation steps during data processing and generate a verification file;

The analysis module is used to compare and analyze the verification files of the operation steps, and when the data is abnormal, generate an error message and send it to the ELT control device.

8. The quality control device according to claim 7, further comprising: a configuration module, configured to set operation steps that need to be verified, and generate a verification table;

The verification module verifies the operation steps in the verification table according to the verification table, and generates a verification file.

9. The quality control device according to claim 8, characterized in that,

The configuration module is also used to set the configuration information for comparative analysis of the verification files of the operation steps as required;

The analysis module extracts corresponding data from the verification file according to the configuration information, and compares and analyzes the extracted data.

10. A data quality control system, characterized in that, comprising:

a data processing device, configured to process data received from a data source;

The quality control device verifies the operation steps during the data processing process, generates verification files, and compares and analyzes the verification files of the operation steps. When data abnormalities occur, an error message is generated and sent to the ELT control device;

The ELT control device interrupts the ongoing operation steps according to the error information.

11. The data quality control system according to claim 10, characterized in that,

The data processing device, also during the data processing process, generates an end mark every time an operation step is completed and sends it to the ELT control device;

The ELT control device also invokes the next operation step according to the end flag.

12. The data quality control system according to claim 10, further comprising:

The result display device is used to display the verification information and/or error information of the operation steps.