CN115357625B - Structured data comparison method, device, electronic device and storage medium - Google Patents
Structured data comparison method, device, electronic device and storage medium Download PDFInfo
- Publication number
- CN115357625B CN115357625B CN202211085688.9A CN202211085688A CN115357625B CN 115357625 B CN115357625 B CN 115357625B CN 202211085688 A CN202211085688 A CN 202211085688A CN 115357625 B CN115357625 B CN 115357625B
- Authority
- CN
- China
- Prior art keywords
- comparison
- data
- field
- structured data
- structured
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012545 processing Methods 0.000 claims abstract description 34
- 238000012163 sequencing technique Methods 0.000 claims description 43
- 230000006870 function Effects 0.000 claims description 37
- 238000007781 pre-processing Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 230000008859 change Effects 0.000 abstract description 15
- 238000005516 engineering process Methods 0.000 abstract description 5
- 238000013508 migration Methods 0.000 description 10
- 230000005012 migration Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 238000012423 maintenance Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012217 deletion Methods 0.000 description 3
- 230000037430 deletion Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Artificial Intelligence (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请涉及数据处理技术领域，提供一种结构化数据比对方法、装置、电子设备及存储介质，该方法包括：获取结构化数据集合和结构化数据集合对应的应用场景需求，结构化数据集合包括第一结构化数据和第二结构化数据，第一结构化数据为基准源数据，第二结构化数据为经过变动后的数据；基于应用场景需求获取字段比对规则，并基于字段比对规则对结构化数据集合进行解析，得到比对项编号和排序字段；基于比对项编号将结构化数据集合进行分组，得到至少一组比对数据；针对每一组比对数据，基于排序字段对比对数据进行排序，并依次进行比对，得到比对结果。这样，可以支持符合各种应用场景需求的字段比对规则，并同时支持批量数据的比对，提高比对效率。
The present application relates to the field of data processing technology, and provides a structured data comparison method, device, electronic device and storage medium, the method comprising: obtaining a structured data set and an application scenario requirement corresponding to the structured data set, the structured data set comprising a first structured data and a second structured data, the first structured data being the reference source data, and the second structured data being the data after the change; obtaining a field comparison rule based on the application scenario requirement, and parsing the structured data set based on the field comparison rule to obtain a comparison item number and a sorting field; grouping the structured data set based on the comparison item number to obtain at least one group of comparison data; for each group of comparison data, sorting the comparison data based on the sorting field, and comparing them in sequence to obtain a comparison result. In this way, field comparison rules that meet the requirements of various application scenarios can be supported, and at the same time, batch data comparison can be supported to improve comparison efficiency.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a structured data comparison method, a structured data comparison device, an electronic device, and a storage medium.
Background
The structured data is also called row data, has specific fields, can be represented and stored by using a relational database, and can logically express the realized data by using a two-dimensional table structure, and when the structured data is subjected to data migration, the data before and after the migration need to be compared so as to ensure the correctness of the data migration work.
In the prior art, when structured data are compared, structured data files can be loaded into relational databases such as Oracle/Mysql, and two tables are associated in a Full Join (SQL Full Join) mode, so that character strings in the two tables are compared.
However, the above data comparison method relies on the performance of a single database, and when there is batch data for comparison, it takes a lot of time, and the comparison efficiency is poor.
Disclosure of Invention
The application provides a structured data comparison method, a structured data comparison device, electronic equipment and a storage medium, which can support field comparison rules meeting the requirements of various application scenes, and simultaneously support the comparison of batch data, thereby improving the comparison efficiency.
In a first aspect, the present application provides a structured data comparison method, the method comprising:
Acquiring a structured data set and application scene requirements corresponding to the structured data set, wherein the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data;
acquiring a field comparison rule based on the application scene requirement, and analyzing the structured data set based on the field comparison rule to obtain comparison item numbers and sequencing fields;
grouping the structured data sets based on the comparison item numbers to obtain at least one group of comparison data;
And sequencing the comparison data based on the sequencing field for each group of comparison data, and sequentially comparing to obtain a comparison result.
Optionally, analyzing the structured data set based on the field comparison rule to obtain a comparison item number and a ranking field, including:
Obtaining a corresponding comparison field in the field comparison rule, wherein the comparison field comprises a type field, a function field and a number field;
and acquiring corresponding data in the structured data set based on the comparison field, and preprocessing the data based on the field comparison rule to obtain a comparison item number and a sequencing field corresponding to the data.
Optionally, preprocessing the data based on the field comparison rule to obtain a comparison item number and a sequencing field corresponding to the data, including:
Identifying a character string to be compared in the data based on the type field, and splicing the character string to be compared, wherein the character string to be compared comprises a primary key character string and a service character string;
processing the spliced character strings to be compared based on the function field to obtain a comparison character string;
and assembling the comparison character strings by using the number field, and identifying the comparison item numbers and the sequencing fields corresponding to the assembled comparison character strings.
Optionally, grouping the structured data sets based on the comparison item numbers includes:
And acquiring a primary key character string corresponding to each datum in the structured data set, and grouping the structured data set based on the primary key character string and the comparison item number.
Optionally, the data are ordered based on the ordering field, and are sequentially compared to obtain a comparison result, which includes:
acquiring the number of data in each group of comparison data, and judging whether the number of the data is larger than 1;
If yes, sequencing the comparison data based on the sequencing field, and sequentially comparing to obtain a comparison result;
if not, obtaining a comparison result based on the source of the comparison data.
Optionally, obtaining the comparison result based on the source of the comparison data includes:
If the comparison data is from the first structured data, determining that the comparison data is deleted data;
and if the comparison data is from the second structured data, determining that the comparison data is newly added data.
Optionally, obtaining the structured data set and an application scenario requirement corresponding to the structured data set includes:
acquiring real-time data in different databases by using a standardized interface, and performing format conversion on the real-time data to obtain a structured data set;
And aiming at the structured data sets corresponding to different databases, acquiring the application scene requirement corresponding to each structured data set.
Optionally, the method further comprises:
After the comparison result is obtained, judging whether the comparison result is consistent;
If yes, comparing the structured data sets again at intervals of preset time to obtain a comparison result which is covered last time after the comparison result so as to verify the accuracy of the comparison;
if not, generating an alarm prompt to remind the user to check the comparison result.
In a second aspect, the present application provides a structured data alignment apparatus, the apparatus comprising:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring a structured data set and application scene requirements corresponding to the structured data set, the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data;
the analysis module is used for acquiring a field comparison rule based on the application scene requirement, and analyzing the structured data set based on the field comparison rule to obtain comparison item numbers and sequencing fields;
The grouping module is used for grouping the structured data sets based on the comparison item numbers to obtain at least one group of comparison data;
And the comparison module is used for sequencing the comparison data based on the sequencing field aiming at each group of comparison data, and sequentially comparing the comparison data to obtain a comparison result.
In a third aspect, the present application provides an electronic device comprising a processor, and a memory communicatively coupled to the processor;
The memory stores computer-executable instructions;
the processor executes computer-executable instructions stored by the memory to implement the method of any one of the first aspects.
In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the method of any one of the first aspects when executed by a processor.
In a fifth aspect, the present application provides a computer program product comprising program code for performing the method according to any of the first aspects when the computer program runs on a computer.
In summary, the application provides a structured data comparison method, a structured data comparison device, an electronic device and a storage medium, which can obtain a structured data set and application scene requirements corresponding to the structured data set, further analyze the structured data set based on the application scene requirements to obtain comparison item numbers and sequencing fields, further group the structured data set based on the comparison item numbers to obtain at least one group of comparison data, further, sequence the comparison data based on the sequencing fields for each group of comparison data, and sequentially compare the comparison data to obtain the comparison result, wherein the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data, so that the field comparison rule meeting various application scene requirements can be supported, the comparison of batch data is supported, the correctness of the data comparison is ensured, and the comparison efficiency is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic view of an application scenario provided in an embodiment of the present application;
FIG. 2 is a flow chart of a structured data comparison method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a standardized interface according to an embodiment of the present application;
FIG. 4 is a flow chart of a method for comparing batch data according to an embodiment of the present application;
FIG. 5 is a flow chart of a real-time data comparison method according to an embodiment of the present application;
FIG. 6 is a schematic structural diagram of a structured data alignment apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.
In order to clearly describe the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same item or similar items having substantially the same function and effect. For example, the first device and the second device are merely for distinguishing between different devices, and are not limited in their order of precedence. It will be appreciated by those of skill in the art that the words "first," "second," and the like do not limit the amount and order of execution, and that the words "first," "second," and the like do not necessarily differ.
In the present application, the words "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.
In the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or" describes an association of associated objects, meaning that there may be three relationships, e.g., A and/or B, and that there may be A alone, while A and B are present, and B alone, where A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (a, b, or c) of a, b, c, a-b, a-c, b-c, or a-b-c may be represented, wherein a, b, c may be single or plural.
The present application will be described with reference to the accompanying drawings, and fig. 1 is a schematic view of an application scenario provided by an embodiment of the present application, where a structured data comparison method provided by the present application may be applied to the application scenario shown in fig. 1. The application scene comprises a first database 1011, a second database 1012, a data processing platform 102 and a display terminal 103, wherein when data in the first database 1011 is migrated to the second database 1012, the data before and after migration need to be compared.
Specifically, the data processing platform 102 may obtain the reference source data from the first database 1011, obtain the migrated data from the second database 1012, and correspondingly obtain a corresponding configuration file in the application scenario, where the configuration file is a field comparison rule configured manually in advance and applicable to the application scenario, and the configuration file includes a comparison field and a comparison rule that need to be compared with data, further, the data processing platform 102 analyzes the configuration file to obtain the field comparison rule, and uses the field comparison rule to compare the reference source data obtained from the first database 1011 and the migrated data obtained from the second database 1012, obtain a comparison result, record the data inconsistent with the comparison, generate an alarm prompt, and send the alarm prompt to the display terminal 103, so that the user operates the display terminal 103 to search for the data inconsistent with the comparison for verification.
It should be noted that, the display terminal 103 may be a display device corresponding to the data processing platform 102 or may be a terminal device of a user, which is not limited in particular, and in the present application, the data processing platform 102 may compare data from multiple databases or compare changed data in one database, which is not limited to the above-mentioned data comparison in the two databases in the data migration scenario, and the number of databases and the application scenario in the embodiment of the present application are not limited in particular.
The terminal device may be a wireless terminal or a wired terminal. A wireless terminal may be a device that provides voice and/or other traffic data connectivity to a user, a handheld device with wireless connectivity, or other processing device connected to a wireless modem. The wireless terminal may communicate with one or more core network devices via a radio access network (Radio Access Network, RAN for short), which may be mobile terminals such as mobile phones (or "cellular" phones) and computers with mobile terminals, for example, portable, pocket, hand-held, computer-built-in or vehicle-mounted mobile devices that exchange voice and/or data with the radio access network. For another example, the wireless terminal may be a Personal communication service (Personal Communication Service, PCS) phone, a cordless phone, a session initiation protocol (Session Initiation Protocol, SIP) phone, a wireless local loop (Wireless Local Loop, WLL) station, a Personal digital assistant (Personal DIGITAL ASSISTANT, PDA) or the like. A wireless Terminal may also be referred to as a system, subscriber Unit (Subscriber Unit), subscriber Station (Subscriber Station), mobile Station (Mobile Station), remote Terminal (Remote Terminal), access Terminal (ACCESS TERMINAL), user Terminal (User Terminal), user Agent (User Agent), user equipment (User Device or User Equipment), without limitation. Optionally, the terminal device may also be a smart phone, a tablet computer, or other devices.
Therefore, the comparison of the structured data can be applied to the scenes such as data migration or online maintenance transaction, and particularly, when the structured data is subjected to data migration, the data before and after migration needs to be compared to ensure the correctness of data migration, and when the online maintenance transaction modifies the data, the difference before and after the data modification needs to be seen at the front end in real time, namely, the difference before and after the data modification during the online maintenance transaction needs to be recorded.
In one possible implementation manner, when the structured data are compared, the structured data file may be loaded into a relational database such as Oracle/Mysql, and the two tables are associated in a fully connected manner, so as to compare the character strings in the two tables.
However, the above data comparison method requires loading and storing files, and relies on the performance of a single database, and when there is batch data for comparison, a lot of time is required, and the comparison efficiency is poor.
It should be noted that when there is batch data for comparison, multiple comparison rules are also needed to adapt to different comparison requirements, but implementation of complex comparison rules has a certain threshold for developers, and for real-time comparison requirements, if implemented based on online transaction, it needs to be logically bound with transaction services, which makes online transaction very complex.
According to the structural data comparison method, a Spark program is used for developing, namely a single database is replaced by a distributed technology, field comparison rules under different application scenes are loaded and analyzed through the Spark program, further, a structural data set is analyzed based on the field comparison rules to obtain comparison item numbers and sequencing fields, the structural data set comprises reference source data and changed data, further, the structural data set is grouped by using the comparison item numbers to obtain at least one group of comparison data, further, the comparison data in the same group are sequenced based on sequencing fields, further, data comparison is carried out, namely the comparison rules meeting the requirements of various application scenes can be supported after the comparison of the same record and the comparison data of the same comparison item number are placed together in sequence, further, flexible comparison of batch data is supported, and the comparison efficiency is improved.
The technical scheme of the application is described in detail below by specific examples. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 2 is a schematic flow chart of a structured data comparison method according to an embodiment of the present application, as shown in fig. 2, where the structured data comparison method includes the following steps:
S201, acquiring a structured data set and application scene requirements corresponding to the structured data set, wherein the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data.
In the embodiment of the present application, the first structured data and the second structured data may be from the same database or different databases, and the structured data may be a database table or a data file, where the first structured data is reference source data, that is, original data stored in the database, and the second structured data is changed data, such as data after addition, deletion, update, and migration, which is not specifically limited in the embodiment of the present application.
Optionally, the second structured data may be change data captured in real time by the real-time data change collection component from the redolog log, where the change data includes all fields recorded after the new addition, deletion and update, and the first structured data includes all fields recorded before the new addition, deletion and update, so that the present application can collect real-time data and compare the real-time data.
S202, acquiring a field comparison rule based on the application scene requirement, and analyzing the structured data set based on the field comparison rule to obtain comparison item numbers and sequencing fields.
In the embodiment of the application, the field comparison rule is a configuration file which is defined in advance, different application scenes correspond to different configuration files, the configuration files comprise comparison fields and comparison rules, the comparison fields are field types corresponding to data in the structured data set, such as field names, type fields, function fields, number fields and the like, the comparison rules can be that the obtained function fields are used for processing and then sequentially comparing the data, and/or the number fields are used for combining and then sequentially comparing the data, and/or the comparison is sequentially carried out in the data corresponding to a certain type of fields.
In this step, the structured data set is parsed based on the field comparison rule, that is, the comparison field corresponding to the field comparison rule is parsed, the comparison item number and the ordering field corresponding to the structured data set are obtained by using the comparison field, the comparison item number is used for marking the field in which the comparison data needs to be compared, for example 200010 indicates that the 9 th to 10 th fields in a certain piece of data need to be compared, the ordering field may refer to the field in the data which can be used for ordering, for example, the operator number, the organization number, the update timestamp, etc., and the application is not limited in particular.
S203, grouping the structured data sets based on the comparison item numbers to obtain at least one group of comparison data.
In this step, the structured data sets are grouped based on the comparison item numbers, that is, based on the comparison item numbers, character strings to be compared are assembled, and further, the comparison data of the same record and having the same comparison item number are grouped into one group, so as to obtain at least one group of comparison data.
S204, sequencing the comparison data based on the sequencing field for each group of comparison data, and sequentially comparing to obtain a comparison result.
In the step, the data are ordered based on the ordering field, such as a data update time stamp corresponding to each group of structured data set, the data in each group are ordered according to the sequence based on the data update time stamp, and further, the ordered data are sequentially compared to obtain a comparison result, wherein the comparison result comprises a data consistent result and a data inconsistent result, and the application can store the inconsistent data according to the data inconsistent result so as to facilitate subsequent inquiry.
The application provides a structured data comparison method, which can obtain structured data sets and application scene requirements corresponding to the structured data sets, further analyze the structured data sets based on field comparison rules to obtain comparison item numbers and sequencing fields, further group the structured data sets based on the comparison item numbers to obtain at least one group of comparison data, further, compare the comparison data based on the sequencing fields for each group of comparison data, and sequentially compare the comparison data to obtain comparison result, wherein the structured data sets comprise standard source data and changed data.
Optionally, analyzing the structured data set based on the field comparison rule to obtain a comparison item number and a ranking field, including:
Obtaining a corresponding comparison field in the field comparison rule, wherein the comparison field comprises a type field, a function field and a number field;
and acquiring corresponding data in the structured data set based on the comparison field, and preprocessing the data based on the field comparison rule to obtain a comparison item number and a sequencing field corresponding to the data.
In the embodiment of the application, the comparison field can comprise a type field, a function field and a number field, wherein the type field is used for marking the type of field comparison, such as Y, N, K, S, C, HB, D, Y is used for comparing, N is used for not comparing, K is used as a main key, C is used as a fixed field, the field can be designed in a self-defined manner, S is used for corresponding to a comparison item number, HB is used as a part of a joint field and is used for participating in comparison, D is used for ordering the field, the function field can be a function expression for processing the field, such as trim, to_char, lower and the like, the to_char is used for converting the data type of the field into a character type, the trim is used for deleting spaces in the data of the field, the data of the field is converted into a character string with lower case that the function field does not need to be processed, the number field is used for marking the comparison result of each service field, and the number can be used for marking the comparison result of the service field to be a certain number from the number to 200010, such as a certain number 10.
For example, in the embodiment of the present application, the corresponding comparison fields in the field comparison rules may be as shown in table 1, and the field comparison rules meeting the requirements of different application scenarios may be formed based on the comparison fields:
TABLE 1
The first column represents the english names of the fields corresponding to the different fields in the data, the foregoing is only illustrative, the english names of the fields corresponding to the fields in the data may be represented in other forms, the embodiment of the application is not particularly limited to this, the second column represents the type field corresponding to the data, the third column represents the function field corresponding to the data, and the fourth column represents the number field corresponding to the data, where the type field, the function field, and the number field may be custom designed, and the embodiment of the application is not particularly limited to this, but merely illustrates the foregoing.
It can be understood that if the data in the structured data set corresponds to a function field, the function corresponding to the function field may be used to perform function processing on the field, and then the data after the function processing is used for data comparison, where a common string processing function is optionally preset in a field comparison rule, and the field comparison rule may also support a user-defined function to expand a function type corresponding to the function field.
Therefore, the embodiment of the application supports complicated and various field comparison rules, and can improve the processing flexibility by acquiring the comparison fields in the field comparison rules to preprocess the data.
Optionally, preprocessing the data based on the field comparison rule to obtain a comparison item number and a sequencing field corresponding to the data, including:
Identifying a character string to be compared in the data based on the type field, and splicing the character string to be compared, wherein the character string to be compared comprises a primary key character string and a service character string;
processing the spliced character strings to be compared based on the function field to obtain a comparison character string;
and assembling the comparison character strings by using the number field, and identifying the comparison item numbers and the sequencing fields corresponding to the assembled comparison character strings.
In the embodiment of the present application, the primary key character string may refer to character string data corresponding to a primary key, such as cst_id, and the service character string refers to character string data corresponding to a plurality of service fields forming a certain piece of data, for example, the contact address includes a plurality of service fields of a country region, a province, a city, a county, a detailed address, etc., that is, a country B, a city C, a county E village.
Optionally, if the data is composed of multiple service fields, each field is separated by @ | such as data corresponding to a contact address, and thus, the application supports that the multiple service fields are combined together to be used as a field for comparison, but if any one of the combined fields changes, the combined fields are required to be used as an output result for data comparison, and the separator for combining each service field together is not particularly limited, which is only an example.
It is understood that a piece of data may be split into multiple service fields for comparison in terms of bytes, so that a certain service field in a piece of data may represent one service meaning, and other adjacent service fields may represent another service meaning.
In the step, the character strings to be compared in the data are identified based on the type field, unnecessary comparison information such as the identification of the primary key field in the data and the splicing of the primary key character strings based on the identified primary key field can be removed, further, the number field is utilized to assemble the comparison character strings, such as the number of each comparison item, the character strings to be compared are assembled, and further, the comparison item numbers and the sequencing fields corresponding to the assembled comparison character strings are identified, wherein the fixed output fields corresponding to the comparison character strings can be identified for data comparison, and the accuracy of comparison is improved.
Therefore, the embodiment of the application can support complex comparison logic, such as the data can be split into a plurality of service fields for comparison, and the support function processes the data, so that unnecessary comparison information can be removed, and the application scene is wide.
Optionally, grouping the structured data sets based on the comparison item numbers includes:
And acquiring a primary key character string corresponding to each datum in the structured data set, and grouping the structured data set based on the primary key character string and the comparison item number.
It is to be appreciated that the present application can group the structured data sets based on the primary key strings.
Therefore, the embodiment of the application can group the structured data sets by combining the primary key character strings and the comparison item numbers, and can group the structured data sets only based on the primary key character strings or the comparison item numbers, thereby improving the grouping accuracy.
Optionally, the data are ordered based on the ordering field, and are sequentially compared to obtain a comparison result, which includes:
acquiring the number of data in each group of comparison data, and judging whether the number of the data is larger than 1;
If yes, sequencing the comparison data based on the sequencing field, and sequentially comparing to obtain a comparison result;
if not, obtaining a comparison result based on the source of the comparison data.
In the application scenario of fig. 1, the data processing platform 102 groups the reference source data obtained from the first database 1011 and the migrated data obtained from the second database 1012 based on the comparison item numbers to obtain at least one group of comparison data, and then the data processing platform 102 may obtain the number N of data in each group of comparison data and determine whether the number N of data is greater than 1, if so, order the comparison data based on the ordering field, and sequentially perform comparison to obtain a comparison result, and if not, obtain a comparison result based on whether the comparison data is from the first database 1011 or the second database 1012, where N is a positive integer greater than or equal to 1.
It can be understood that the embodiment of the application can also set a preset requirement, determine whether the preset requirement is met based on the number of data bars, determine whether the comparison data are required to be sequenced and then compared, and obtain a comparison result.
Therefore, the embodiment of the application can sort and compare the data by using the sorting field based on whether the number of the data in each group of comparison data meets the preset requirement, so as to obtain a comparison result, and can directly determine the comparison result according to the source of the comparison data when the number of the data does not meet the preset requirement, thereby improving the comparison efficiency.
Optionally, obtaining the comparison result based on the source of the comparison data includes:
If the comparison data is from the first structured data, determining that the comparison data is deleted data;
and if the comparison data is from the second structured data, determining that the comparison data is newly added data.
In this step, if it is determined that the comparison data is from the first structured data and is not present in the second structured data, it is indicated that the comparison data is deleted data, and if it is determined that the comparison data is from the second structured data and is not present in the first structured data, it is indicated that the comparison data is newly added data.
Therefore, the embodiment of the application can directly obtain the comparison result based on the source of the comparison data, and improve the timeliness of the comparison.
Optionally, obtaining the structured data set and an application scenario requirement corresponding to the structured data set includes:
acquiring real-time data in different databases by using a standardized interface, and performing format conversion on the real-time data to obtain a structured data set;
And aiming at the structured data sets corresponding to different databases, acquiring the application scene requirement corresponding to each structured data set.
In the embodiment of the application, the standardized interface can refer to a designed unified interface for calling data in different databases, and is a code frame defined in advance, and when the data in different databases is called, only the corresponding parameter values in the code frame are required to be modified.
In the step, the real-time change data of the database can be read by using the Kafka message middleware, but the results of analyzing the change data by different database products are quite different, so that the change data needs to be standardized, namely, the acquired real-time data is subjected to format conversion to obtain a structured data set for data comparison, and the standardized data format is unified, so that the unification of the follow-up processing flow is facilitated.
The data in batch data comparison is in a text format, and the standardized output of the real-time data comparison process is also in a text format, so that the text file formats of the data and the text file are consistent.
An exemplary embodiment of the present application provides a standardized interface, as shown in fig. 3, where the standardized interface is provided in fig. 3, and each type of change data of each type of database is used to call a corresponding implementation class, and call of each implementation class needs to be implemented based on the standardized interface, and implementation classes corresponding to the change data of the databases, such as Mysql, oracle, DB, oceanBase, include databaseType (data type), processinsertRec (add), processDeleteRec (delete), processUpdateNotKeyRec (update non-primary key field), processUpdateKeyRec (update primary key field), and are implemented based on the standardized interface, so that for various types of databases, a user only needs to implement the standardized interface.
It should be noted that, the comparison result of each dml operation includes transaction start and end marks, only one result statement is newly added/deleted, the comparison result corresponding to the update non-primary key common field is one record before update or one record after update, the comparison result corresponding to the update primary key field is V, D, I three statements, the corresponding comparison result is one delete operation, one insert operation, and other dml operations for the update primary key field of the OceanBase database, and the corresponding comparison result is one record containing information before and after change.
It will be appreciated that real-time alignment typically requires on-line maintenance transactions or back-end specific alignment procedures to customize the alignment of each change of data, is costly to develop and maintain, and requires significant repetitive effort, the embodiment of the application can realize real-time data comparison based on the standardized interface and also support the analysis of data from different database products, wherein the real-time comparison logic is consistent with the batch comparison logic, the core code is shared, the processing caliber is consistent, and the later operation and maintenance cost can be reduced.
Optionally, the method further comprises:
After the comparison result is obtained, judging whether the comparison result is consistent;
If yes, comparing the structured data sets again at intervals of preset time to obtain a comparison result which is covered last time after the comparison result so as to verify the accuracy of the comparison;
if not, generating an alarm prompt to remind the user to check the comparison result.
In the embodiment of the present application, the preset time may refer to a time set for performing secondary comparison, a specific value corresponding to the preset time is not limited, and may be one day, and the embodiment of the present application also does not specifically limit the content and the transmission form of the generated alarm prompt, and may generate the alarm prompt based on inconsistent data, for example, the alarm prompt is "data 1 in table a and data 2 in table B are inconsistent", and the transmission form may be a short message, a display frame, and the like.
It should be noted that, although the real-time data comparison makes up the defect of poor timeliness of batch processing, the situation that real-time data is possibly lost exists at the same time, so after the preset time interval is supported, the data comparison is executed again, the comparison result of the last time is covered after the comparison result is obtained, namely, the repeated comparison is not felt by the data reading application through the database idempotent method, and the data timeliness is ensured while the data integrity is also ensured.
Therefore, the embodiment of the application reduces the possible loss of the real-time data in a re-execution mode, improves the data integrity while improving the timeliness of the data, and can generate an alarm prompt to prompt a user to compare the abnormal condition of the result so as to facilitate the verification of the subsequent call data and improve the convenience.
In combination with the above embodiment, the structured data comparison method provided by the present application supports two data comparison methods, real-time and batch, the real-time comparison is implemented based on FLINK DATASTREAM technology, the batch comparison is implemented based on FLINK DATASET technology, the real-time comparison logic is consistent with the batch comparison logic, the comparison port is consistent with the comparison port, and the two share a core code, and specifically, fig. 4 is a schematic flow diagram of the batch data comparison method provided by the embodiment of the present application, as shown in fig. 4, the batch data comparison method includes the following steps:
Step 1, starting a spark program, loading and analyzing a configuration file (application scene requirement), reading data in a file A and a file B from a data file, or reading data in a table a and a table B from a database table, analyzing the data according to the configuration file, and executing step 2.
It should be noted that, the data in the file a and the table a are the reference source data, and the data in the file B and the table B are the changed data, and the embodiment of the application may perform the comparison processing on the data in the file a and the file B alone, may perform the comparison processing on the data in the table a and the table B alone, may perform the comparison processing on the data in the file a and the file B alone, and may perform the comparison processing on the data in the table a and the table B alone, and the embodiment of the application is not limited thereto specifically.
And 2, merging two parts of data in the file A and the file B, and/or merging two parts of data in the table a and the table B, grouping input data (structured data set) according to a main key and a comparison item number to obtain at least one group of comparison data, and sequencing the comparison data in the same group according to a data update timestamp, wherein the number of data strips in the same group is recorded as N, further, determining a comparison result according to the number of the data strips N, wherein N is a positive integer greater than or equal to 1, and further executing the step 3.
And 3, if the number of the data pieces is 1 and the data is determined to be from the file A and/or the table a, the file B and/or the table B, the record corresponding to the data is deleted, if the data is determined to be from the file B and/or the table B, the record corresponding to the data is newly added, and if the number of the data pieces is greater than 1, the ordered results can be sequentially compared to obtain a comparison result, wherein the data corresponding to the comparison inconsistent item can be recorded in the comparison result for the user to check.
Fig. 5 is a flow chart of a real-time data comparison method according to an embodiment of the present application, as shown in fig. 5, the real-time data comparison method includes the following steps:
The method comprises the steps of starting a SPARK STREAMING program, loading and analyzing a configuration file (application scene requirement), calling a unified standardized interface by using a Kafka message middleware to collect change data and original data in an Oracle database in real time, standardizing change records into json character strings in a unified format for adapting various database products, correspondingly, further requiring the value of unified data update types, and adapting the records of an update main key, namely converting the change data and the original data into the unified format, wherein the data in the Oracle database is modified to be called change data, and further executing subsequent steps.
It should be noted that the subsequent steps are similar to step 2 and step 3 in the embodiment shown in fig. 4, and the details thereof are described with reference to the embodiment shown in fig. 4 and will not be described herein.
The application supports two modes of real-time and batch comparison of structured data, supports complex and various field comparison rules in the form of configuration files, has lower learning and use thresholds, ensures that a user does not need repeated development, presets common character string processing functions, supports custom functions to expand the function processing functions, performs data comparison in a stream batch combination mode, reduces the later operation and maintenance cost, and improves timeliness.
In the foregoing embodiment, the structured data comparison method provided in the embodiment of the present application is described, and in order to implement each function in the method provided in the embodiment of the present application, the electronic device as the execution body may include a hardware structure and/or a software module, and each function may be implemented in the form of a hardware structure, a software module, or a hardware structure plus a software module. Some of the functions described above are performed in a hardware configuration, a software module, or a combination of hardware and software modules, depending on the specific application of the solution and design constraints.
For example, fig. 6 is a schematic structural diagram of a structured data comparison device provided by the embodiment of the present application, as shown in fig. 6, where the device includes an obtaining module 610, an analyzing module 620, a grouping module 630, and a comparison module 640, where the obtaining module 610 is configured to obtain a structured data set and an application scenario requirement corresponding to the structured data set, where the structured data set includes first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data;
The parsing module 620 is configured to obtain a field comparison rule based on the application scenario requirement, and parse the structured data set based on the field comparison rule to obtain a comparison item number and a ranking field;
the grouping module 630 is configured to group the structured data sets based on the comparison item numbers, to obtain at least one group of comparison data;
The comparison module 640 is configured to sequence the comparison data based on the sequence field for each set of comparison data, and perform comparison in sequence to obtain a comparison result.
Optionally, the parsing module 620 includes an obtaining unit and a preprocessing unit;
specifically, the acquiring unit is configured to acquire a comparison field corresponding to the field comparison rule, where the comparison field includes a type field, a function field, and a number field;
The preprocessing unit is used for acquiring corresponding data in the structured data set based on the comparison field, and preprocessing the data based on the field comparison rule to obtain a comparison item number and a sequencing field corresponding to the data.
Optionally, the preprocessing unit is specifically configured to:
Identifying a character string to be compared in the data based on the type field, and splicing the character string to be compared, wherein the character string to be compared comprises a primary key character string and a service character string;
processing the spliced character strings to be compared based on the function field to obtain a comparison character string;
and assembling the comparison character strings by using the number field, and identifying the comparison item numbers and the sequencing fields corresponding to the assembled comparison character strings.
Optionally, the grouping module 630 is specifically configured to:
And acquiring a primary key character string corresponding to each datum in the structured data set, and grouping the structured data set based on the primary key character string and the comparison item number.
Optionally, the comparison module 640 includes a judging unit, a comparison unit, and a determining unit;
specifically, the judging unit is configured to obtain the number of data in each set of comparison data, and judge whether the number of data is greater than 1;
the comparison unit is used for sequencing the comparison data based on the sequencing field when the number of the data strips is greater than 1, and sequentially comparing the data strips to obtain a comparison result;
And the determining unit is used for obtaining a comparison result based on the source of the comparison data when the number of the data bars is equal to 1.
Optionally, the determining unit is specifically configured to:
If the comparison data is from the first structured data, determining that the comparison data is deleted data;
and if the comparison data is from the second structured data, determining that the comparison data is newly added data.
Optionally, the acquiring module 610 is specifically configured to:
acquiring real-time data in different databases by using a standardized interface, and performing format conversion on the real-time data to obtain a structured data set;
And aiming at the structured data sets corresponding to different databases, acquiring the application scene requirement corresponding to each structured data set.
Optionally, the device further includes a judging module, where the judging module is configured to:
After the comparison result is obtained, judging whether the comparison result is consistent;
If yes, comparing the structured data sets again at intervals of preset time to obtain a comparison result which is covered last time after the comparison result so as to verify the accuracy of the comparison;
if not, generating an alarm prompt to remind the user to check the comparison result.
The specific implementation principle and effect of the structured data comparison device provided by the embodiment of the application can be referred to the related description and effect corresponding to the above embodiment, and will not be repeated here.
The embodiment of the application further provides a schematic structural diagram of an electronic device, and fig. 7 is a schematic structural diagram of an electronic device provided by the embodiment of the application, as shown in fig. 7, the electronic device may include a processor 701 and a memory 702 communicatively connected to the processor, where the memory 702 stores a computer program, and the processor 701 executes the computer program stored in the memory 702, so that the processor 701 executes the method described in any of the embodiments.
Wherein the memory 702 and the processor 701 may be connected by a bus 703.
Embodiments of the present application also provide a computer-readable storage medium storing computer program-executable instructions that, when executed by a processor, are configured to implement a method as described in any of the foregoing embodiments of the present application.
The embodiment of the application also provides a chip for running instructions, and the chip is used for executing the method in any of the previous embodiments executed by the electronic equipment in any of the previous embodiments.
Embodiments of the present application also provide a computer program product comprising a program code for performing a method as in any of the preceding embodiments of the present application as performed by an electronic device when the computer program runs on the computer program.
In the technical scheme of the application, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the information such as financial data or user data are in accordance with the regulations of related laws and regulations, and the public welfare is not violated.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The modules illustrated as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.
In addition, each functional module in the embodiments of the present application may be integrated in one processing unit, or each module may exist alone physically, or two or more modules may be integrated in one unit. The units formed by the modules can be realized in a form of hardware or a form of hardware and software functional units.
The integrated modules, which are implemented in the form of software functional modules, may be stored in a computer readable storage medium. The software functional modules described above are stored in a storage medium and include instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or processor to perform some of the steps of the methods described in the various embodiments of the application.
It should be appreciated that the Processor may be a central processing unit (Central Processing Unit, abbreviated as CPU), or may be other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, abbreviated as DSP), application SPECIFIC INTEGRATED Circuit (ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present application may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.
The Memory may include a high-speed random access Memory (Random Access Memory, abbreviated as RAM), and may further include a Non-volatile Memory (NVM), such as at least one magnetic disk Memory, and may also be a U-disk, a removable hard disk, a read-only Memory, a magnetic disk, or an optical disk.
The bus may be an industry standard architecture (Industry Standard Architecture, ISA) bus, an external device interconnect (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, the buses in the drawings of the present application are not limited to only one bus or to one type of bus.
The storage medium may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random-Access Memory (SRAM), electrically erasable programmable Read-Only Memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ ONLY MEMORY EEPROM), erasable programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an Application SPECIFIC INTEGRATED Circuits (ASIC). It is also possible that the processor and the storage medium reside as discrete components in an electronic device or a master device.
The foregoing is merely a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this, and any changes or substitutions within the technical scope disclosed in the embodiment of the present application should be covered in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.
Claims (8)
1. A structured data alignment method, the method comprising:
Acquiring a structured data set and application scene requirements corresponding to the structured data set, wherein the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data;
acquiring a field comparison rule based on the application scene requirement, and analyzing the structured data set based on the field comparison rule to obtain comparison item numbers and sequencing fields;
grouping the structured data sets based on the comparison item numbers to obtain at least one group of comparison data;
Sequencing the comparison data based on the sequencing field for each group of comparison data, and sequentially comparing to obtain comparison results, wherein the comparison results comprise a result with consistent data and a result with inconsistent data;
Analyzing the structured data set based on the field comparison rule to obtain comparison item numbers and ordering fields, wherein the method comprises the following steps:
Obtaining a corresponding comparison field in the field comparison rule, wherein the comparison field comprises a type field, a function field and a number field, the type field is used for marking the type of field comparison, and the type comprises a type used for representing whether the fields are compared or not;
Acquiring corresponding data in the structured data set based on the comparison field, and
Identifying a character string to be compared in the data based on the type field, and splicing the character string to be compared, wherein the character string to be compared comprises a primary key character string and a service character string;
processing the spliced character strings to be compared based on the function field to obtain a comparison character string;
Assembling the comparison character strings by using the number fields, and identifying comparison item numbers and sequencing fields corresponding to the assembled comparison character strings;
the step of sorting the comparison data based on the sorting field and comparing the comparison data in sequence to obtain a comparison result comprises the following steps:
acquiring the number of data in each group of comparison data, and judging whether the number of the data is larger than 1;
When the number of the data strips is greater than 1, sequencing the comparison data based on the sequencing field, and sequentially comparing to obtain a comparison result;
And when the number of the data strips is equal to 1, determining that the comparison data is deleted data if the comparison data is from the first structured data, and determining that the comparison data is newly added data if the comparison data is from the second structured data.
2. The method of claim 1, wherein grouping the structured data sets based on the alignment item numbers comprises:
And acquiring a primary key character string corresponding to each datum in the structured data set, and grouping the structured data set based on the primary key character string and the comparison item number.
3. The method of claim 1, wherein obtaining a structured data set and application scenario requirements corresponding to the structured data set comprises:
acquiring real-time data in different databases by using a standardized interface, and performing format conversion on the real-time data to obtain a structured data set;
And aiming at the structured data sets corresponding to different databases, acquiring the application scene requirement corresponding to each structured data set.
4. A method according to any one of claims 1-3, wherein the method further comprises:
After the comparison result is obtained, judging whether the comparison result is consistent;
If yes, comparing the structured data sets again at intervals of preset time to obtain a comparison result which is covered last time after the comparison result so as to verify the accuracy of the comparison;
if not, generating an alarm prompt to remind the user to check the comparison result.
5. A structured data alignment apparatus, the apparatus comprising:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring a structured data set and application scene requirements corresponding to the structured data set, the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data;
the analysis module is used for acquiring a field comparison rule based on the application scene requirement, and analyzing the structured data set based on the field comparison rule to obtain comparison item numbers and sequencing fields;
The grouping module is used for grouping the structured data sets based on the comparison item numbers to obtain at least one group of comparison data;
the comparison module is used for sequencing the comparison data based on the sequencing field for each group of comparison data, and sequentially comparing the comparison data to obtain comparison results, wherein the comparison results comprise a result with consistent data and a result with inconsistent data;
the analysis module comprises an acquisition unit and a preprocessing unit;
The obtaining unit is configured to obtain a comparison field corresponding to the field comparison rule, where the comparison field includes a type field, a function field, and a number field, where the type field is used to mark a type of field comparison, and the type includes a type used to characterize whether the field is compared;
the preprocessing unit is used for acquiring corresponding data in the structured data set based on the comparison field,
Identifying a character string to be compared in the data based on the type field, and splicing the character string to be compared, wherein the character string to be compared comprises a primary key character string and a service character string;
processing the spliced character strings to be compared based on the function field to obtain a comparison character string;
Assembling the comparison character strings by using the number fields, and identifying comparison item numbers and sequencing fields corresponding to the assembled comparison character strings;
the comparison module comprises a judging unit, a comparison unit and a determining unit;
the judging unit is used for acquiring the number of data in each group of comparison data and judging whether the number of data is larger than 1;
the comparison unit is used for sequencing the comparison data based on the sequencing field when the number of the data strips is greater than 1, and sequentially comparing the data strips to obtain a comparison result;
The determining unit is configured to determine that the comparison data is deleted data if the comparison data is from the first structured data and determine that the comparison data is newly added data if the comparison data is from the second structured data when the number of the data pieces is equal to 1.
6. An electronic device comprising a processor and a memory communicatively coupled to the processor;
The memory stores computer-executable instructions;
The processor executes computer-executable instructions stored in the memory to implement the method of any one of claims 1-4.
7. A computer readable storage medium storing computer executable instructions which when executed by a processor are adapted to implement the method of any one of claims 1-4.
8. A computer program product comprising program code for performing the method of any of claims 1-4 when the computer program is run by a computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211085688.9A CN115357625B (en) | 2022-09-06 | 2022-09-06 | Structured data comparison method, device, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211085688.9A CN115357625B (en) | 2022-09-06 | 2022-09-06 | Structured data comparison method, device, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115357625A CN115357625A (en) | 2022-11-18 |
| CN115357625B true CN115357625B (en) | 2024-12-20 |
Family
ID=84005920
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211085688.9A Active CN115357625B (en) | 2022-09-06 | 2022-09-06 | Structured data comparison method, device, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115357625B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116628107A (en) * | 2023-05-30 | 2023-08-22 | 曙光云计算集团有限公司 | Data comparison method, device, equipment and medium |
| CN118689912B (en) * | 2024-08-27 | 2024-12-31 | 杭州乒乓智能技术有限公司 | Data comparison method and device, business data comparison system and electronic device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111782657A (en) * | 2020-07-08 | 2020-10-16 | 上海乾臻信息科技有限公司 | Data processing method and device |
| CN112199935A (en) * | 2020-09-24 | 2021-01-08 | 建信金融科技有限责任公司 | Data comparison method and device, electronic equipment and computer readable storage medium |
| CN113642311A (en) * | 2021-08-12 | 2021-11-12 | 北京奇艺世纪科技有限公司 | Data comparison method and device, electronic equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| NL2012222C2 (en) * | 2014-02-06 | 2015-08-10 | Genalice B V | A method of storing/reconstructing a multitude of sequences in/from a data storage structure. |
| CN114461611A (en) * | 2022-01-26 | 2022-05-10 | 连连（杭州）信息技术有限公司 | A data comparison method, device, electronic device and storage medium |
-
2022
- 2022-09-06 CN CN202211085688.9A patent/CN115357625B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111782657A (en) * | 2020-07-08 | 2020-10-16 | 上海乾臻信息科技有限公司 | Data processing method and device |
| CN112199935A (en) * | 2020-09-24 | 2021-01-08 | 建信金融科技有限责任公司 | Data comparison method and device, electronic equipment and computer readable storage medium |
| CN113642311A (en) * | 2021-08-12 | 2021-11-12 | 北京奇艺世纪科技有限公司 | Data comparison method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115357625A (en) | 2022-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109522746B (en) | A data processing method, electronic device and computer storage medium | |
| CN111553137B (en) | Report generation method and device, storage medium and computer equipment | |
| CN115357625B (en) | Structured data comparison method, device, electronic device and storage medium | |
| CN111400392B (en) | Multi-source heterogeneous data processing method and device | |
| CN108334609B (en) | Method, device, equipment and storage medium for realizing JSON format data access in Oracle | |
| CN105678625A (en) | Method and equipment for determining identity information of user | |
| CN113268500B (en) | Service processing method and device and electronic equipment | |
| CN105678129A (en) | Method and device for determining user identity information | |
| CN108460149B (en) | Text data processing method, device and equipment and computer readable storage medium | |
| CN114049016B (en) | Index similarity judgment method, system, terminal device and computer storage medium | |
| CN115237805A (en) | Test case data preparation method and device | |
| CN115544132A (en) | Data import method and system and electronic equipment | |
| CN115390847A (en) | Log processing method and device, computer readable storage medium and terminal | |
| US11934396B2 (en) | Data reconciliation for big data environments | |
| CN117827902A (en) | Service data processing method, device, computer equipment and storage medium | |
| CN113190236B (en) | HQL script verification method and device | |
| CN106682107B (en) | Method and device for determining incidence relation of database table | |
| CN113419957B (en) | Rule-based big data offline batch processing performance capacity scanning method and device | |
| CN113792055A (en) | Data processing method, electronic device and storage medium | |
| CN109785099B (en) | Method and system for automatically processing service data information | |
| CN114090014A (en) | Program splitting method, device, device and computer storage medium | |
| US9251253B2 (en) | Expeditious citation indexing | |
| CN114818645B (en) | Automatic report generation method, device, equipment and medium based on data body | |
| CN114218441B (en) | Method for calling and displaying UOF document | |
| CN110287191B (en) | Data alignment method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |