CN111078738A

CN111078738A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN111078738A
Application number: CN201911165369.7A
Authority: CN
Inventors: 张方俊
Original assignee: Taikang Insurance Group Co Ltd; Taikang Online Property Insurance Co Ltd
Current assignee: Taikang Insurance Group Co Ltd; Taikang Online Property Insurance Co Ltd
Priority date: 2019-11-25
Filing date: 2019-11-25
Publication date: 2020-04-28
Anticipated expiration: 2039-11-25
Also published as: CN111078738B

Abstract

The application provides a data processing method, a data processing device, an electronic device and a storage medium, wherein the method comprises the following steps: reading data of each data source, extracting comparison data of each piece of data of each data source and detail data corresponding to the comparison data, wherein the comparison data are used for representing a key field of each piece of data, the detail data are used for representing detail information of the key field represented by the comparison data, and the detail information of the key field is used for representing the detailed field and the storage position corresponding to the key field; comparing the comparison data in each data source, and if different comparison data with different key fields exist, determining the detail information of the key fields represented by the different comparison data according to the detail data corresponding to the different comparison data; and displaying the detail information of the key field represented by the difference comparison data. The data processing method can improve the data processing efficiency.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.

Background

The internet financial insurance transaction is characterized in that the amount of each order is small, and the order quantity is large, so that the data processing system adopts a distributed architecture to improve the data processing efficiency. Compared with the traditional single architecture in the scene, the data is subjected to more circulation and interaction. For example, a data processing system for internet financial transaction includes not only business subsystems such as underwriting, wind control, and finance, but also middleware such as cache and message queue, and each node may cause data inconsistency due to problems such as program problems and hardware/network failures, thereby causing problems such as risk runaway and business and financial data inconsistency. These problems not only cause financial loss, but also introduce significant checking, analyzing, and processing efforts. Therefore, it is necessary to check the data of each node (data source) periodically and automatically.

The prior art checks the data in each data source in the following manner: all data stored in each data source are written into a common comparison base, and the common comparison base queries and derives difference data in a table connection (join) mode. This approach works substantially with small amounts of data, but as the amount of data stored by each data source increases, the data processing efficiency drops dramatically.

Disclosure of Invention

The application provides a data processing method, a data processing device, an electronic device and a storage medium, which can improve data processing efficiency.

A first aspect of the present application provides a data processing method, including:

reading data of each data source, extracting comparison data of each piece of data of each data source and detail data corresponding to the comparison data, wherein the comparison data are used for representing a key field of each piece of data, the detail data are used for representing detail information of the key field represented by the comparison data, and the detail information of the key field is used for representing the detailed field and the storage position corresponding to the key field;

comparing the comparison data in each data source, and if different comparison data with different key fields exist, determining the detail information of the key fields represented by the different comparison data according to the detail data corresponding to the different comparison data;

and displaying the detail information of the key field represented by the difference comparison data.

Optionally, after the comparison data is a comparison file, the detail data is a detail file, and the extracting of the comparison data of each data source and the detail data corresponding to the comparison data further includes:

and writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into a storage database.

Optionally, the writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into a storage database includes:

reading a comparison file of each piece of data in each data source, and caching the comparison file of the read data and a detail file corresponding to the comparison file of the read data to a cache database;

judging whether a comparison file of N pieces of data of each data source and a detail file corresponding to the comparison file of each data source are cached in the cache database;

if the comparison file of the N pieces of data of each data source and the detail file corresponding to the comparison file of each data source are cached in the cache database, writing the comparison file of the N pieces of data of each data source and the detail file corresponding to the comparison file of the N pieces of data of each data source into the storage database, and emptying the cache database;

if the comparison file of the N pieces of data of each data source and the detail file corresponding to the comparison file of each data source are not cached in the cache database, continuing to read the data in each data source until the comparison file of the N pieces of data of each data source and the detail file corresponding to the comparison file of the N pieces of data of each data source are cached in the cache database.

Optionally, after comparing the comparison data in each of the data sources, the method further includes:

reading key fields of comparison files of the N pieces of data of each data source in the storage database, and caching the key fields of the N pieces of data of each data source to the cache database;

and if the key fields of the N pieces of data of each data source recorded in the cache database are compared, determining the difference key fields with differences in each data source, and determining the comparison file to which the difference key fields in each data source belong as the difference comparison file.

Optionally, the method further includes:

and displaying a data link, wherein the data link is used for representing the identification information of the data source of the key field of the difference comparison data representation, the difference generation reason and the data processing suggestion.

Optionally, the method further includes:

and if the processing flow for processing the difference comparison data is determined to be stored according to the detail information of the key field represented by the difference comparison data, processing the difference comparison data by adopting the processing flow.

Optionally, the data in each data source is policy data, and the key field is a policy number or a serial number.

A second aspect of the present application provides a data processing apparatus comprising:

the reading module is used for reading the data of each data source;

the processing module is used for extracting comparison data of each piece of data of each data source and detail data corresponding to the comparison data, comparing the comparison data in each data source, and if different comparison data with different key fields exist, determining detail information of the key fields represented by the difference comparison data according to the detail data corresponding to the difference comparison data, wherein the comparison data is used for representing the key fields of each piece of data, the detail data is used for representing the detail information of the key fields represented by the comparison data, and the detail information of the key fields is used for representing the detail fields corresponding to the key fields and storage positions;

and the display module is used for displaying the detail information of the key field represented by the difference comparison data.

Optionally, the comparison data is a comparison file, and the detail data is a detail file.

And the writing module is used for writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into a storage database.

Optionally, the write-in module is specifically configured to read a comparison file of each piece of data in each data source, cache the comparison file of the read piece of data and a detail file corresponding to the comparison file of the read piece of data in a cache database, and determine whether the comparison file of N pieces of data of each data source and the detail file corresponding to the comparison file of each data source are cached in the cache database; if so, writing comparison files of the N pieces of data of each data source and detail files corresponding to the comparison files of the N pieces of data of each data source into the storage database, and emptying the cache database; if not, continuing to read the data in each data source until the comparison file of the N data of each data source and the detail file corresponding to the comparison file of the N data of each data source are cached in the cache database.

Optionally, the reading module is further configured to read a key field of a comparison file of the N pieces of data of each data source in the storage database, and cache the key field of the N pieces of data of each data source in the cache database;

and if the key fields of the N pieces of data of each data source recorded in the cache database are compared, and the difference key fields with differences in each data source are determined, the processing module is further configured to determine the comparison file to which the difference key fields in each data source belong as the difference comparison file.

Optionally, the display module is further configured to display a data link, where the data link is used to represent identification information of a data source of a key field of the difference comparison data representation, a difference generation reason, and a data processing suggestion.

Optionally, the processing module is further configured to, if a processing procedure for processing the difference comparison data is determined to be stored according to the detail information of the key field represented by the difference comparison data, process the difference comparison data by using the processing procedure.

Optionally, the reading module is further configured to use data in each data source as policy data, and the key field is a policy number or a serial number.

A third aspect of the present application provides an electronic device comprising: at least one processor and memory;

the memory stores computer-executable instructions;

the at least one processor executes computer-executable instructions stored by the memory to cause the electronic device to perform the data processing method of the first aspect.

A fourth aspect of the present application provides a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, implement the data processing method of the first aspect described above.

The data processing method avoids a processing mode of checking and comparing after all data in each data source are copied in the prior art, and extracts comparison data of each piece of data of each data source and detail data corresponding to the comparison data. The comparison data is used for representing the key field of each piece of data, so that key information of the data can be embodied, and the comparison data with difference is determined according to the difference of the comparison data of each piece of data of each data source. The detail information corresponding to the comparison data with the difference is obtained according to the detail data corresponding to the comparison data with the difference, so that the data in each data source with the difference can be rapidly obtained, and the data processing efficiency is improved.

Drawings

FIG. 1 is a schematic flow chart of a data processing method in the prior art;

fig. 2 is a schematic view of a scene to which the data processing method provided in the present application is applied;

FIG. 3 is a first flowchart illustrating a data processing method according to the present application;

FIG. 4 is a schematic view of the interface change provided herein;

fig. 5 is a schematic flow chart diagram of a data processing method provided in the present application;

FIG. 6 is a schematic diagram illustrating a prior art process for writing a file;

FIG. 7 is a schematic flow chart of writing a file provided herein;

FIG. 8 is a schematic diagram illustrating a process for determining details of key fields of a variance comparison document representation according to the prior art;

FIG. 9 is a schematic flow chart of details of determining the difference versus key field of a document representation provided in the present application;

FIG. 10 is a schematic flow chart of a prior art preservation method;

FIG. 11 is a schematic flow chart of a saving method provided herein;

FIG. 12 is a schematic diagram of a data processing apparatus provided in the present application;

fig. 13 is a schematic structural diagram of an electronic device provided in the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the embodiments of the present application, and it is obvious that the described embodiments are some but not all of the embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

For more clearly explaining the data processing method provided by the present application, a data processing method in the prior art is first described.

In a data processing system for internet financial transaction, assuming that N data sources exist, data in the N data sources needs to be processed. The data source may be a business subsystem such as underwriting, wind control, finance, or middleware such as a cache and a message queue. The data in the N data sources should be consistent, but the data may be inconsistent between the data sources due to problems such as program problems, hardware/network failures, etc., so that problems such as risk runaway, and inconsistent property and industry data occur. Therefore, the data in the N data sources needs to be periodically checked and compared to obtain inconsistent data, so as to further analyze the reason for the inconsistent data and solve the problem.

Fig. 1 is a schematic flow chart of a data processing method in the prior art. As shown in fig. 1, when performing the verification and comparison process on data in N data sources in the prior art, the following process needs to be executed:

s101, data checking is started.

And S102, circularly writing the data in each data source into a common comparison library.

For example, data from data Source 1, data Source 2, data Source 3 … …, data Source N, may be written cyclically into a common alignment library. The process of circular writing is adopted here, for example, taking the data source 1 as an example, m pieces of data in the data source 1 can be read, written into the common comparison library, and then m pieces of data in the data source 1 are read and written into the common comparison library until the data in the data source 1 is completely written into the common comparison library.

S103, inquiring the difference information in a public comparison library by using a table connection mode.

The detailed principle and mode for querying the difference information in the multi-table connection mode of the database are not described in detail in the present application. It should be understood that when the table connection is used to query the difference information, the data in each data source needs to be compared line by line to obtain the difference information.

And S104, storing the difference information into a storage database.

And S105, displaying the difference information and giving an alarm.

In the prior art, the difference information can be displayed after being acquired, and an alarm is given, which is not described in detail in the present application.

And S106, ending.

In the prior art, when data in each data source is checked and compared, all data in each data source needs to be written into a common comparison library, and data in each data source needs to be compared line by line, which consumes a lot of time. This approach works substantially with small amounts of data, but as the amount of data stored by each data source increases, the data processing efficiency drops dramatically.

The method in the prior art is improved as follows: and adopting multithreading, dividing the data in each data source into data sets according to a certain rule, and simultaneously comparing the data sets. This improved method can effectively improve efficiency, but can cause a large load to the database.

In order to solve the problems in the prior art, the application provides a data processing method, which extracts comparison data of each piece of data of each data source and detail data corresponding to the comparison data. The comparison data is used for representing the key field of each piece of data, so that key information of the data can be embodied, the comparison data with difference is determined according to the difference of the comparison data of each piece of data of each data source, and the comparison time can be shortened compared with the processing mode in the prior art. The detail information corresponding to the comparison data with the difference is obtained according to the detail data corresponding to the comparison data with the difference, so that the data with the difference in each data source can be rapidly obtained, and the aim of improving the data processing efficiency is fulfilled.

Fig. 2 is a schematic view of a scene to which the data processing method provided in the present application is applied. As shown in fig. 2, the applicable scenarios of the data processing method provided by the present application include: at least two data sources and a data processing device.

In the application, the data sources are different according to different scenes in which the data processing method is applied. For example, in a data processing system for internet financial transaction, the data source may be a business subsystem for underwriting, wind control, finance and the like, or middleware for caching, message queuing and the like. The data sources may be data storage servers and backup servers, etc., as in the data processing system of the educational network. Alternatively, the data source may be a server, database, or the like for storing data.

Wherein, the data processing device can process the data in each data source to obtain the data with difference. The data processing device may be an electronic device having a processing function, such as a server or a terminal device. It should be understood that each data source is illustrated in fig. 2 as a database.

The data processing method provided by the present application is described below with reference to specific embodiments. Fig. 3 is a first schematic flow chart of a data processing method provided in the present application. The execution entity of the method flow shown in fig. 2 may be a data processing device, which may be implemented by any software and/or hardware. As shown in fig. 3, the data processing method provided in this embodiment may include:

s301, reading data of each data source, extracting comparison data of each piece of data of each data source, and detail data corresponding to the comparison data, wherein the comparison data is used for representing a key field of each piece of data, the detail data is used for representing detail information of the key field represented by the comparison data, and the detail information of the key field is used for representing the detail field corresponding to the key field and a storage position.

In this embodiment, the data processing apparatus may extract the comparison data of each piece of data in each data source and the detail data corresponding to the comparison data when reading the data of each data source. It should be understood that each data source has multiple pieces of data stored therein, where each piece of data may be data stored to the data source in chronological order. Illustratively, if a piece of data is stored at time a in the data source 1, then the piece of data is a piece of data.

Wherein the alignment data is used to characterize the key fields of each piece of data. Optionally, the alignment data may be a key field of each piece of data.

One possible implementation manner of the data processing apparatus in this embodiment to extract the comparison data of each piece of data in each data source may be as follows: and extracting comparison data in the data according to a preset extraction template. Optionally, the data in each data source is policy data, and the key field may be a policy number or a serial number. In this embodiment, a row of a key field (e.g., a row to which a policy number or serial number belongs) that needs to be extracted may be preset, and data corresponding to the preset row is extracted from the data according to the preset extraction template, that is, comparison data.

Another possible implementation manner of the data processing apparatus extracting the comparison data of each piece of data in each data source may be: the semantics of the extracted key fields can be preset, and data such as names, insurance amounts, policy numbers and the like in the extracted data are taken as the key fields. Then, for each piece of data, the comparison data can be extracted according to the semantics in the piece of data.

The detail data is used for representing the detail information of the key fields represented by the comparison data, and the detail information of the key fields is used for representing the detailed fields and the storage positions corresponding to the key fields. In this embodiment, when the comparison data of each piece of data is extracted, the detail data corresponding to the comparison data may be extracted at the same time. The detailed information of the key field may include a detailed field and a storage location corresponding to the key field. The detailed field corresponding to the key field can be used to characterize other fields related to the key field that are not embodied in the key field. The storage location of the key field may be a storage space or buffer, etc. in the data source that stores the key field.

For example, as an insurance order, the key fields may be: insurance policy number and amount of the application. The detailed fields corresponding to the key fields may be the insurance year, the beneficiary, and the like, and the storage location may be a C folder in a B data disk in the data source 1. It should be understood that the detailed field corresponding to the key field may be preset.

Optionally, the executing step in S301 in this embodiment may be triggered by a user, may also be executed periodically, or may also be executed when the data amount in each data source is greater than the data amount threshold.

When the execution step in S301 is user triggering, a triggering control, such as a "comparison" control, may be displayed on the display interface of the data processing apparatus. When the user clicks or otherwise selects the "compare" control, the execution step in S301 may be triggered. Fig. 4 is a schematic view of the interface change provided in the present application. As shown in the interface 401 in fig. 4, a comparison control is displayed on the display interface of the data processing apparatus. Alternatively, in the present embodiment, a comparison period may be preset, so that the data processing apparatus periodically executes the execution step in S301. Alternatively, when the data processing apparatus detects that the data amount in each data source is larger than the data amount threshold, the execution step in S301 is executed.

Optionally, in this embodiment, if the data amount in each data source is large, the data in each data source may be partitioned according to a time sequence, so as to obtain a plurality of data areas in each data source, extract comparison data and detail data of the data in each data area for comparison,

for example, the following table one shows data after partitioning the data in the data source 1 and the data source 2:

watch 1

	Data source 1	Data source 2
			Data area 1	Data A and data B	Data A and data B'
Data area 2	Data C and data D	Data C ', data D'
			Data area 3	Data E and data F	Data E' and data F

As shown in the table i, in this embodiment, the comparison data and the detail data of the data in the data area 1 of the data source 1 and the data area 2 may be extracted.

Optionally, in this embodiment, each piece of comparison data and each piece of detail data correspond to an identifier of the data, such as "policy number" or "line number to which the data belongs", and then the data processing device may compare the comparison data of the data having the same identifier of the data according to the identifier of the data, so as to improve the comparison efficiency.

S302, comparing the comparison data in each data source, and if different comparison data with different key fields exist, determining the detail information of the key fields represented by the different comparison data according to the detail data corresponding to the different comparison data.

In this embodiment, comparison is performed according to the extracted comparison data in each data source, and if there is difference comparison data with different key fields, it is determined that the data in each data source corresponding to the comparison data is inconsistent, and in this embodiment, the detail information of the key field represented by the difference comparison data is determined according to the detail data corresponding to the difference comparison data. It is understood that if the difference alignment data is the alignment data having a difference in each data source.

For example, as shown in the above table one, when the comparison data of the data area 1 of the data source 1 and the data area 1 of the data source 2 is compared, it is found that the data B in the data source 1 is inconsistent with the data B ' in the data source 2, and then according to the detail data corresponding to the difference comparison data (data B and data B '), it may be determined that the detail information of the difference comparison data B is "the guarantee period is 5 years and the beneficiary is X, and the storage location is M folder in the data source 1", and the detail information of the difference comparison data B ' is "the guarantee period is 6 years and the beneficiary is X, and the storage location is N folder in the data source 2".

And S303, displaying the detailed information of the key field represented by the difference comparison data.

In this embodiment, after the data processing device obtains the detail information of the key field represented by the difference comparison data, the detail information may be displayed to remind the user that the data in the data sources are inconsistent.

Illustratively, as shown in the interface 402 in fig. 4, the interface displays the detailed information of "the year of application is 5 years and the beneficiary is X, and the storage location is M folder in the data source 1", "the year of application is 6 years and the beneficiary is X, and the storage location is N folder in the data source 2".

Correspondingly, based on the detail information, the user can determine inconsistent data in each data source and the storage location, that is, an effective solution can be determined according to the detail information.

Optionally, the data link may also be displayed in this embodiment. The data link is used for representing identification information of a data source of a key field of the difference comparison data representation, a difference generation reason and a data processing suggestion. The identification information of the data source of the key field is used to uniquely identify the data source to which the key field belongs, such as data source 1 or data source 2. The cause of the difference may be "packet loss or network anomaly in transmission", for example. The processing proposal may be, for example, "repair a certain module in the X data source to prevent packet loss". It is to be understood that the difference generation cause and the processing advice in the present embodiment may be set corresponding to the line.

In one possible implementation, if the processing flow storing the processing difference comparison data is determined according to the detail information of the key field represented by the difference comparison data, the processing flow is adopted to process the difference comparison data.

For example, if the data processing apparatus compares the detail information of the key field represented by the data according to the difference, it may be determined that the reason of the difference is "repairing a certain module in the data source 1 to generate a packet loss", and if a processing flow for repairing a certain module in the data source 1 is stored in the data processing apparatus, the processing flow is adopted to repair a certain module in the data source 1, so as to prevent the certain module in the data source 1 from packet loss again. Alternatively, the process flow is "copy data B in data source 1 to data source 2 and delete data B' in data source 2" to keep data in data source 1 and data source 2 consistent.

The data processing method provided by the embodiment comprises the following steps: reading data of each data source, and extracting comparison data of each piece of data of each data source and detail data corresponding to the comparison data; comparing the comparison data in each data source, and if different comparison data with different key fields exist, determining the detail information of the key fields represented by the different comparison data according to the detail data corresponding to the different comparison data; and displaying the detail information of the key fields of the difference comparison data representation. The method avoids the processing mode of checking and comparing after all data in each data source are copied in the prior art, and the comparison data of each piece of data of each data source and the detail data corresponding to the comparison data are extracted. The comparison data is used for representing the key field of each piece of data, so that the key information of the data can be embodied, and the comparison data with the difference is determined according to the difference of the comparison data of each piece of data of each data source. The detail information corresponding to the comparison data with the difference is obtained according to the detail data corresponding to the comparison data with the difference, so that the data in each data source with the difference can be rapidly obtained, and the data processing efficiency is improved.

On the basis of the above embodiments, the data processing method provided by the present application is further described below with reference to fig. 5. Fig. 5 is a schematic flow chart of a data processing method provided in the present application. As shown in fig. 5, the data processing method provided in this embodiment may include:

s501, reading data of each data source, and extracting a comparison file of each piece of data of each data source and a detail file corresponding to the comparison file.

In this embodiment, the comparison data and the detail data are stored in the form of a file. Specifically, the comparison data is a comparison file, and the detail data is a detail file.

S502, writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into a storage database.

For a clearer description of the file writing mode in the data processing method in this embodiment, first, a file writing mode in the prior art is applied to a specific mode in a scene of a write comparison file and a detail file in this application. Fig. 6 is a schematic flow chart of writing a file in the prior art. As shown in fig. 6, the manner of the file in the prior art may include:

s601, reading data in each data source.

S602, writing the data in each data source into a storage database in a write file form.

In the prior art, a stream reading mode is used to cyclically read data of a database and then write a file. After reading several pieces of data and writing a file, S603 may be executed:

and judging whether data exist in each data source. If there is no data in each data source, S604 is executed, and if there is data in each data source, S601 is executed again.

604, the write file ends.

In the prior art, in order to reduce the times of opening and closing a file, a file handle object is maintained in the whole process. This test takes 125 seconds to derive 250 thousand data from the data source. Further, the process is analyzed to find that a block is generated each time a file is written, and a user mode-core mode switching is triggered, so that the switching is triggered for each time the file is written once when data is read. This cost can be particularly expensive when the amount of data is large.

Therefore, an improved method is provided in this embodiment, that is, data in each read data source is cached first, and a file is written once after a certain number of times (for example, 100) is read, so that switching between a user mode and a core mode is greatly reduced, and file writing efficiency is improved. Specifically, fig. 7 is a schematic flow chart of writing a file provided in this application. As shown in fig. 7, the S502 may include:

s5021, reading the comparison file of each piece of data in each data source, and caching the read comparison file of the data and the detail file corresponding to the read comparison file of the data in a cache database.

In this embodiment, the comparison file of the read data in each data source and the detail file corresponding to the comparison file of the read data may be cached in the cache database, and then the comparison file and the detail file cached in the cache database may be written in the storage database, which may reduce "user mode-core mode" switching.

S5022, judging whether comparison files of the N data of each data source and detail files corresponding to the comparison files of each data source are cached in a cache database or not; if so, executing S5023, otherwise, returning to executing S5021 until the comparison files of the N data of each data source and the detail files corresponding to the comparison files of the N data of each data source are cached in the cache database.

In order to achieve the purpose of comparing data in batches to further reduce the data processing time, in this embodiment, after caching the comparison file and the detail file, it may be determined whether the comparison file of N pieces of data of each data source and the detail file corresponding to the comparison file of each data source have been cached in the cache database.

And S5023, writing the comparison files of the N pieces of data of each data source and the detail files corresponding to the comparison files of the N pieces of data of each data source into a storage database, and emptying the cache database.

In this embodiment, after the comparison file of the N pieces of data in each data source and the detail file corresponding to the comparison file have been stored in the cache database, the comparison file of the N pieces of data of each data source and the detail file corresponding to the comparison file of the N pieces of data of each data source may be written into the storage database, and the cache database is emptied, so as to cache the comparison file of the next batch of the N pieces of data and the corresponding detail file.

S503, comparing the comparison files in the data sources, and determining a difference comparison file.

Corresponding to the foregoing S5021-S5023, in this embodiment, specifically, the key fields of the comparison files storing the N pieces of data of each data source in the database may be read, and the key fields of the N pieces of data of each data source may be cached in the cache database, and if the key fields of the N pieces of data of each data source recorded in the cache database are compared, and the difference key fields of the differences of each data source are determined, the comparison file to which the difference key fields of each data source belong is determined as the difference comparison file.

S504, according to the detail file corresponding to the difference comparison file, determining the detail information of the key field represented by the difference comparison file.

In order to more clearly describe how to determine the detailed information of the key field represented by the difference comparison file in the data processing method in this embodiment, a specific manner applied to the scenes of the comparison file and the detailed file in the present application is described first according to the manner in the prior art. FIG. 8 is a schematic diagram illustrating a process for determining details of key fields of a variance comparison document representation in the prior art. As shown in fig. 8, the method may include:

s801, reading key fields of the difference comparison files in the data sources.

S802, a difference detail file corresponding to the key field of the difference comparison file is searched from the detail file.

And S803, writing the difference detail file into a storage database.

In the prior art, the key fields of the difference comparison file are read in a circulating manner, and the difference comparison file is read in a circulating manner. After reading a plurality of key fields of the difference comparison file and writing the difference detail file, S804: and judging whether the unread difference comparison file exists or not.

If there is no unread difference comparison file, step S805 is executed, and if there is no unread difference comparison file, step S801 is returned to.

And S805, ending.

This approach requires multiple reads of the difference detail file, which is inefficient. In this embodiment, the method is improved, in which the detail file is mainly read in a stream form, and if the comparison file includes a difference key field, the detail file corresponding to the comparison file is stored. This only requires one reading pass of the detail file.

Specifically, fig. 9 is a schematic flow chart of the detailed information of the key field for determining the difference comparison file representation provided in the present application. As shown in fig. 9, the S504 may include:

s5041, read key fields of the difference alignment file in each data source.

S5042, cache the key fields of the read difference comparison file into a cache database.

S5043, whether there are key fields of the unread difference comparison file is judged. If not, go to step 5044, if yes, go back to step 5041 until all the unread delta match files have been completely cached in the cache database.

S5044, reading the detail file, and judging whether the detail file is a difference detail file corresponding to the key field of the difference comparison file; if so, S5045 is performed, and if not, S5046 is performed.

S5045, writing the difference detail file to a storage database.

S5046, judging whether an unread detail file exists; if so, the process returns to step S5044, and if not, step S5047 is performed.

And S5047, finishing.

And S505, saving the detailed information of the key fields represented by the difference comparison file.

After determining the detail information of the key fields of the difference comparison file representations, the detail information of the key fields of the difference comparison file representations is usually saved in a storage database, and then the total amount and the display details are counted. In general, when the amount of inconsistent data is small, the detail information and the saving mode of the key fields of the difference comparison file representation in the prior art work well. Fig. 10 is a schematic flow chart of a storage method in the prior art. As shown in fig. 10, the saving method may include:

s1001, reading the detail information of the key fields represented by the difference comparison file.

S1002, storing the read detailed information of the key fields represented by the difference comparison file into a storage database.

S1003, judging whether the unread difference is compared with the detail information of the key field represented by the file; if yes, returning to execute the step S1001 until the detail information of the key field represented by all the difference comparison files is read, and if not, executing the step S1004.

And S1004, ending.

However, when the data size of the difference is larger than the data size of the detail information of the key fields represented by the file, the storage method causes a very large load on the storage database. In consideration of the fact that when a large amount of detail information of the key fields represented by the difference comparison file exists, only sample data is needed to perform problem analysis, and not all data is needed, therefore, the embodiment improves the mode in the prior art, and mainly only stores a preset amount of data before the detail information of the key fields represented by the difference comparison file into a storage database, and other pieces of detail information of the key fields represented by the difference comparison file form a file to be stored. When displaying, the detail information of the preset number of data can be displayed, and other detail information provides file downloading.

Specifically, fig. 11 is a schematic flow chart of the saving method provided in the present application. As shown in fig. 11, the step S505 may include:

s5051, reading the detail information of the key fields represented by the preset number of difference comparison files in the cache database.

S5052, writing the detail information of the key fields represented by the difference comparison files with a preset number into a storage database, and storing the detail information of the key fields represented by the remaining difference comparison files into the storage database in a file writing mode.

S506, displaying the detailed information of the key fields represented by the difference comparison file.

It should be understood that, in this embodiment, the implementation manners in S501 and S506 may refer to the relevant descriptions in S301 and S303 in the foregoing embodiment, and are not described herein again.

The data processing method provided by the embodiment can be applied to the scenes of settlement comparison and cumulative risk guarantee comparison:

exemplary, settlement alignment: and comparing whether the invoice data, the underwriting data and the financial data are consistent or not. If the difference is not consistent, the calculation is influenced, on one hand, the timeliness of the settlement is influenced, and huge reconciliation work, time consumption and labor consumption are caused; on the other hand, the image of the company is affected. By using the data processing mode in the application, besides daily account checking, checking of all data of the last month can be supported at the beginning of the month, inconsistent data are found, timely processing is performed, account checking work is greatly reduced, and the timeliness of settlement is improved.

Cumulative risk guarantee comparison: and comparing whether the underwriting data is consistent with the data of the accumulated risk library, if not, the underwriting data is not synchronized to the accumulated risk library, certain malicious insurance application behaviors can be missed, and further, the major financial risk is generated. And the requirement on the timeliness is very high. By using the data processing mode, the unsynchronized data can be found at the second level, the alarm is given, and the wind control is supported powerfully.

In this embodiment, in order to achieve the purpose of comparing data in batches to further reduce the data processing time, after caching the comparison file and the detail file, it may be determined whether the comparison file of N pieces of data of each data source and the detail file corresponding to the comparison file of each data source have been cached in the cache database in this embodiment; and moreover, when the comparison files of the N pieces of data of each data source and the detail files corresponding to the comparison files of each data source are written, a cache mode is adopted, so that the switching from a user mode to a core mode is reduced. In addition, in this embodiment, the detail file is also read in a stream form, and if the comparison file includes the difference key field, the detail file corresponding to the comparison file is stored. Thus, only one-time reading of the detail file is needed; in addition, in the embodiment, when a large amount of difference is considered to compare detail information of the key fields represented by the file, only sample data is needed to perform problem analysis, and not all data is needed, so that the data storage efficiency is improved.

Fig. 12 is a schematic structural diagram of a data processing apparatus provided in the present application. As shown in fig. 12, the data processing apparatus 1200 includes: a reading module 1201, a processing module 1202, a display module 1203, and a writing module 1204.

A reading module 1201, configured to read data of each data source.

The processing module 1202 is configured to extract comparison data of each piece of data of each data source and detail data corresponding to the comparison data, compare the comparison data in each data source, and if there is difference comparison data with different key fields, determine detail information of a key field represented by the difference comparison data according to the detail data corresponding to the difference comparison data, where the comparison data is used to represent the key field of each piece of data, the detail data is used to represent the detail information of the key field represented by the comparison data, and the detail information of the key field is used to represent the detail field and a storage location corresponding to the key field;

a display module 1203, configured to display detail information of the key field represented by the difference comparison data.

The writing module 1204 is configured to write the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into the storage database.

Optionally, the write-in module 1204 is specifically configured to read a comparison file of each piece of data in each data source, cache the comparison file of the read piece of data and a detail file corresponding to the comparison file of the read piece of data in a cache database, and determine whether the comparison file of N pieces of data of each data source and the detail file corresponding to the comparison file of each data source are cached in the cache database; if so, writing comparison files of the N pieces of data of each data source and detail files corresponding to the comparison files of the N pieces of data of each data source into a storage database, and emptying the cache database; if not, continuing to read the data in each data source until the comparison file of the N data of each data source and the detail file corresponding to the comparison file of the N data of each data source are cached in the cache database.

Optionally, the reading module 1201 is further configured to read key fields of a comparison file storing N pieces of data of each data source in the database, and cache the key fields of the N pieces of data of each data source in the cache database;

if the key fields of the N pieces of data of each data source recorded in the cache database are compared, and the difference key fields having differences in each data source are determined, the processing module 1202 is further configured to determine the comparison file to which the difference key fields in each data source belong as the difference comparison file.

Optionally, the display module 1203 is further configured to display a data link, where the data link is used to represent identification information of a data source of a key field of the difference comparison data representation, a difference generation reason, and a data processing suggestion.

Optionally, the processing module 1202 is further configured to process the difference comparison data by using the processing flow if the processing flow storing the processing difference comparison data is determined according to the detail information of the key field represented by the difference comparison data.

Optionally, the reading module 1201 is further configured to use data in each data source as policy data, and the key field is a policy number or a serial number.

The principle and technical effect of the data processing apparatus provided in this embodiment are similar to those of the data processing method, and are not described herein again.

Fig. 13 is a schematic structural diagram of an electronic device provided in the present application. As shown in fig. 13, the electronic device 1300 includes: memory 1301 and at least one processor 1302.

A memory 1301 for storing program instructions.

The processor 1302 is configured to implement the data processing method in this embodiment when the program instructions are executed, and specific implementation principles may be referred to in the foregoing embodiments, which are not described herein again.

The electronic device 1300 may also include an input/output interface 1303.

Input/output interface 1303 may include separate output and input interfaces, or may be an integrated interface that integrates input and output. The output interface is used for outputting data, the input interface is used for acquiring input data, the output data is a general name output in the method embodiment, and the input data is a general name input in the method embodiment.

The present application further provides a readable storage medium, in which an execution instruction is stored, and when the execution instruction is executed by at least one processor of the electronic device, the data processing method in the above embodiments is implemented when the computer execution instruction is executed by the processor.

The present application also provides a program product comprising execution instructions stored in a readable storage medium. The at least one processor of the electronic device may read the execution instruction from the readable storage medium, and the execution of the execution instruction by the at least one processor causes the electronic device to implement the data processing method provided by the various embodiments described above.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware form, and can also be realized in a form of hardware and a software functional module.

The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present application. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In the above embodiments of the data Processing apparatus or the electronic device, it should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present application may be embodied directly in a hardware processor, or in a combination of the hardware and software modules in the processor.

Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, comprising:

2. The method according to claim 1, wherein the comparison data is a comparison file, the detail data is a detail file, and after the extraction of the comparison data of each piece of data of each of the data sources and the detail data corresponding to the comparison data, the method further comprises:

3. The method according to claim 2, wherein writing the comparison file of each piece of data of each data source and the detail file corresponding to the comparison file into a storage database comprises:

4. The method of claim 3, wherein after comparing the comparison data in each of the data sources, further comprising:

5. The method according to any one of claims 1-4, further comprising:

6. The method of claim 5, further comprising:

7. The method of any one of claims 1-4, wherein the data in each of the data sources is policy data and the key field is a policy number or a serial number.

8. A data processing apparatus, comprising:

the reading module is used for reading the data of each data source;

9. An electronic device, comprising: at least one processor and memory;

the memory stores computer-executable instructions;

the at least one processor executing the computer-executable instructions stored by the memory causes the electronic device to perform the method of any of claims 1-7.

10. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a processor, implement the method of any one of claims 1-7.