CN116955377A - Data processing method and device, storage medium and computer equipment - Google Patents

Abstract

The invention discloses a data processing method and device, storage medium and computer equipment, which belongs to the field of data processing technology. It mainly aims at solving the problem of difficulty in data sharing between independent data management systems, including obtaining first database data and second database data. Database data, and perform data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data; obtain a data template, and perform data deduplication on the deduplicated third database data based on the data template. The second database data is subjected to standardized conversion to obtain standardized conversion data, so that the standardized conversion data and the first database data have the same data-related information; the first database data is updated based on the standardized conversion data.

Description

A data processing method and device, storage medium, and computer equipment

Technical field

The present invention relates to the field of data processing technology, and in particular, to a data processing method and device, storage medium, and computer equipment.

Background technique

With the continuous reform and deepening of the business of large enterprises, many subsidiaries have been hatched. Since the subsidiary and the parent company are operationally independent, independent data management systems are used for data management operations in production and operations. Independent data management systems have characteristics such as database differences and operating system heterogeneity, making it difficult to interact and share data between the parent company and its subsidiaries, forming data islands. With the continuous progress and development of enterprises, they need to rely on data for analysis operations such as trend analysis and business quality analysis. Since data cannot be shared between independent data management systems, data analysis is more difficult and cannot satisfy the needs of managers. The need for decision-making analysis of existing data resources.

Contents of the invention

In view of this, the present invention provides a data processing method and device, storage medium, and computer equipment. The main purpose is to solve the problem of difficulty in data sharing between independent data management systems.

According to one aspect of the present invention, a data processing method is provided, including:

Obtain the first database data and the second database data, and perform data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data;

Obtain a data template, and perform standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data and the first database data have the same data-related information;

Data updating is performed on the first database data based on the standardized conversion data.

Further, before obtaining the first database data and the second database data, the method further includes:

classifying the first database data and the second database data into a plurality of data categories;

The same data index is added to the first database data and the second database data having the same data category.

Further, performing data deduplication processing on the second database data based on the first database data, and obtaining the deduplicated second database data includes:

Obtain the first data index of the first database data, and obtain the second data index of the second database data;

Perform matching processing on the first data index and the second data index, and obtain target first database data and target second database data having the same data index;

Perform data deduplication processing on the target second database data based on the target first database data to obtain the deduplicated second database data.

Further, before obtaining the data template, the method further includes:

Obtain data-related information of the first database data, where the data-related information includes data categories, data types, data formats and associated data;

Construct data templates with the same number as the data categories based on the data category, the data type, the data format and the associated data;

An association relationship is established between the data template and the data category.

Further, obtaining a data template and performing standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data includes:

Based on the data category of the second database data after deduplication, query the association relationship and determine the target data template;

Obtain the target data type, target data format and target associated data of the target data template;

Standardized conversion is performed on the deduplicated second database data based on the target data type, the target data format and the target associated data to obtain standardized conversion data.

Further, updating the first database data based on the standardized conversion data includes:

Send an update request carrying the standardized conversion data to the first database; so that the first database performs an approval operation based on the update request, and after the update request is approved, update the standardized conversion data to the in the first database.

Further, the method also includes:

Receive data access to the second database from multiple clients, so that the clients obtain the standardized conversion data; the clients are connected to the second database through an enterprise service bus;

Correspondingly, data update operations on the second database from multiple clients are received, including deactivation, addition and modification.

According to another aspect of the present invention, a data processing device is provided, including:

A deduplication module, used to obtain the first database data and the second database data, and perform data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data;

A conversion module, configured to obtain a data template, and perform standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data has the same characteristics as the first database data. data related information;

An update module, configured to update the first database data based on the standardized conversion data.

Further, the device also includes an index adding module for:

classifying the first database data and the second database data into a plurality of data categories;

The same data index is added to the first database data and the second database data having the same data category.

Furthermore, the deduplication module is also used to:

Obtain the first data index of the first database data, and obtain the second data index of the second database data;

Perform matching processing on the first data index and the second data index, and obtain target first database data and target second database data having the same data index;

Perform data deduplication processing on the target second database data based on the target first database data to obtain the deduplicated second database data.

Further, the device also includes a template creation module for:

Obtain data-related information of the first database data, where the data-related information includes data categories, data types, data formats and associated data;

Construct data templates with the same number as the data categories based on the data category, the data type, the data format and the associated data;

An association relationship is established between the data template and the data category.

Further, the conversion module is also used to:

Based on the data category of the second database data after deduplication, query the association relationship and determine the target data template;

Obtain the target data type, target data format and target associated data of the target data template;

Standardized conversion is performed on the deduplicated second database data based on the target data type, the target data format and the target associated data to obtain standardized conversion data.

Further, the update module is also used to:

Send an update request carrying the standardized conversion data to the first database; so that the first database performs an approval operation based on the update request, and after the update request is approved, update the standardized conversion data to the in the first database.

Further, the device also includes a receiving and updating module for:

Receive data access to the second database from multiple clients, so that the clients obtain the standardized conversion data; the clients are connected to the second database through an enterprise service bus;

Correspondingly, data update operations on the second database from multiple clients are received, including deactivation, addition and modification.

According to another aspect of the present invention, a storage medium is provided, and at least one executable instruction is stored in the storage medium. The executable instruction causes the processor to perform operations corresponding to the above-mentioned data processing method.

According to another aspect of the present invention, a computer device is provided, including a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface complete communication with each other through the communication bus;

The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the above data processing method.

Through the above technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages:

The present invention provides a data processing method and device, storage medium, and computer equipment. Compared with the existing technology, the present invention obtains first database data and second database data, and processes the data based on the first database data. The second database data is subjected to data deduplication processing to obtain the deduplicated second database data; a data template is obtained, and the deduplicated second database data is standardized and converted based on the data template to obtain standardized conversion data, so that The standardized conversion data and the first database data have the same data-related information; data updating of the first database data based on the standardized conversion data can realize data interaction and sharing between the parent company and the subsidiary company. , effectively avoiding the formation of data islands, and further providing an important data basis for decision-making analysis of data resources.

The above description is only an overview of the technical solution of the present invention. In order to have a clearer understanding of the technical means of the present invention, it can be implemented according to the content of the description, and in order to make the above and other objects, features and advantages of the present invention more obvious and understandable. , the specific embodiments of the present invention are listed below.

Description of the drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be construed as limiting the invention. Also throughout the drawings, the same reference characters are used to designate the same components. In the attached picture:

Figure 1 shows a schematic flow chart of a data processing method provided by an embodiment of the present invention;

Figure 2 shows a schematic flow chart of another data processing method provided by an embodiment of the present invention;

Figure 3 shows a schematic flow chart of yet another data processing method provided by an embodiment of the present invention;

Figure 4 shows a schematic flow chart of yet another data processing method provided by an embodiment of the present invention;

Figure 5 shows a schematic structural diagram of a data processing device provided by an embodiment of the present invention;

FIG. 6 shows a schematic structural diagram of a computer device provided by an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a thorough understanding of the disclosure, and to fully convey the scope of the disclosure to those skilled in the art.

An embodiment of the present invention provides a data processing method, as shown in Figure 1. The method includes:

101. Obtain the first database data and the second database data, and perform data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data;

In the embodiment of the present invention, the current execution end obtains the first database data and the second database data, wherein the first database data is the parent company data, and the data standard of the parent company data can be used as the data standard of any subsidiary data in the group. Among them, the second database data is subsidiary data. At present, subsidiary data has not yet been unified data storage and management based on parent company data as the data standard. Therefore, data islands are formed between each subsidiary data and parent company data. After acquiring the first database data and the second database data, the current execution end performs data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data. Among them, the data deduplication process is used to represent the deletion of data that is duplicated with the first database data from the acquired second database data, which can effectively avoid data redundancy in the current execution end.

102. Obtain a data template, and perform standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data and the first database data have the same data correlation. information;

In the embodiment of the present invention, the current execution end obtains a data template, which is pre-created based on data categories and used to represent standardized processing rules for data storage. The current execution end performs standardized conversion on the deduplicated second database data based on the data template. The standardized conversion includes standardized conversion of data types, standardized conversion of data formats, standardized conversion of data types of associated data, standardized conversion of data formats of associated data, etc., which are not specifically limited in the embodiment of the present invention. After the standardized conversion, the standardized converted data is obtained, and the obtained standardized converted data should have the same data-related information as the first database data.

103. Update the first database data based on the standardized conversion data.

In the embodiment of the present invention, the current execution end performs data update on the first database data based on the standardized conversion data, that is, the current execution end sends an update request carrying the standardized conversion data based on the data management interface of the first database; when the first database When the data management related party receives the above update request, it performs an approval operation on the update request. When it is detected that the update request is approved, the standardized conversion data is updated into the first database. In addition, if no approval information is detected within a certain time interval, the current execution end sends data update failure information to the data management related party of the second database, so that the data management related party performs the standardized conversion data Further verification is performed to ensure that the standardized conversion data and the first database data have the same data-related information.

Further, as a refinement and expansion of the specific implementation of the above embodiment, in order to facilitate classification management of data and improve the efficiency of data query, another data processing method is provided, with the steps of obtaining the first database data and the second database data. Previously, the method also included:

classifying the first database data and the second database data into a plurality of data categories;

The same data index is added to the first database data and the second database data having the same data category.

In the embodiment of the present invention, the current execution end divides the first database data and the second database data into multiple data categories, such as material data, customer data, enterprise employee data, enterprise organization data, etc., the present invention implements The examples are not specifically limited. The current execution end adds the same data index to the first database data and the second database data of the same category, so as to facilitate querying of the first database data and the second database data of the same category through the same data index.

Further, as a refinement and expansion of the specific implementation of the above embodiment, in order to reduce redundant data and reduce the workload of later data standardization conversion, another data processing method is provided, as shown in Figure 2, the steps are based on The first database data performs data deduplication processing on the second database data, and the deduplicated second database data includes:

201. Obtain the first data index of the first database data, and obtain the second data index of the second database data;

202. Perform matching processing on the first data index and the second data index, and obtain the target first database data and the target second database data having the same data index;

203. Perform data deduplication processing on the target second database data based on the target first database data to obtain the deduplicated second database data.

In the embodiment of the present invention, the current execution end obtains the first data index of the first database data and obtains the second data index of the second database data. After the current execution end obtains the first data index and the second data index, it performs matching processing on the first data index and the second data index, and matches the data with the same data index, that is, from the first database data and the second database data Obtain data of the same category as target first database data and target second database data. The current execution end performs data deduplication processing on the target second database data based on the target first database data, and obtains the deduplicated second database data. For example, the current execution end obtains the target first database data with data index A as customer class data H, and obtains the target second database data with data index A as customer class data M. The current execution end needs to calculate customer data based on customer class data H. Class data M is deduplicated, that is, the merchant name or merchant unique identification code in M is matched with the merchant name or merchant unique identification code in H. If the match is successful, it is considered that M has the same merchant information as H. Redundant customer and merchant information needs to be deleted from M, which is not specifically limited in the embodiment of the present invention.

Further, as a refinement and expansion of the specific implementation of the above embodiment, in order to facilitate data standardization processing of the second database data, another data processing method is provided. As shown in Figure 3, before the step of obtaining the data template, The above methods also include:

301. Obtain data-related information of the first database data. The data-related information includes data category, data type, data format and associated data;

302. Construct data templates with the same number as the data categories based on the data category, the data type, the data format and the associated data;

303. Establish an association between the data template and the data category.

In the embodiment of the present invention, the current execution end obtains data-related information of the first database data. The data-related information includes data category, data type, data format and associated data. For example, the associated data obtained for customer data includes customer name, customer identification code, customer contact address, customer contact number, etc. The data type includes character, integer, floating point, etc., and the data format includes numerical value, character, binary, etc., this document The embodiments of the invention are not specifically limited. The current execution end builds data templates with the same number as the data categories based on the data categories, data types, data formats and associated data in the above data-related information, that is, a corresponding data template is created for each data category. For example, the data template corresponding to merchant data must include associated data such as merchant name, merchant identification code, merchant contact address, merchant contact number, etc. The data type and data format of each associated data are standardized and limited in the template. After the current execution end creates the data template, it also establishes an association between the data category and the corresponding data template to facilitate subsequent querying of the data template based on the data category.

Further, as a refinement and expansion of the specific implementation of the above embodiment, in order to quickly perform data standardization conversion on the second database data, another data processing method is provided. As shown in Figure 4, the steps are to obtain a data template, and Perform standardized conversion on the deduplicated second database data based on the data template, and obtain standardized conversion data including:

401. Based on the data category of the second database data after deduplication, query the association relationship and determine the target data template;

402. Obtain the target data type, target data format and target associated data of the target data template;

403. Perform standardized conversion on the deduplicated second database data based on the target data type, the target data format, and the target associated data to obtain standardized conversion data.

In the embodiment of the present invention, the current execution end performs data template query based on the data category of the second database data after deduplication. If the data currently to be converted is customer category data, the association relationship is queried and the data template corresponding to the customer category is Determine the target data template. The current execution end obtains the target associated data of the target data template, such as customer name, customer identification code, customer contact address, customer contact number, etc., and converts the above customer name, customer identification code, customer contact address, customer contact number and other data into The target data type and target data format specified by the target data template, that is, the standardized conversion data obtained after conversion have the same standards as the first database data, which facilitates later data sharing and analysis.

Further, as a refinement and expansion of the specific implementation of the above embodiment, in order to achieve access and update manipulation of the second database data within the subsidiary, another data processing method is provided, and the method also includes:

Receive data access to the second database from multiple clients, so that the clients obtain the standardized conversion data; the clients are connected to the second database through an enterprise service bus;

Correspondingly, data update operations on the second database from multiple clients are received, including deactivation, addition and modification.

In the embodiment of the present invention, the current execution end receives data access requests from multiple clients to the second database, obtains the accessed data based on the data access requests, such as customer data, equipment data, etc., and sends the accessed data to the corresponding client. end, so that the client can obtain the standardized conversion data after standardized conversion. Among them, each client is a client used by the subsidiary to perform various services of the subsidiary to carry out the company's internal business, including clients related to production materials used by the production department, and clients related to company finance used by the financial department. Clients related to production management used by the management department, clients related to logistics used by the logistics department, etc. are not specifically limited in the embodiment of the present invention. Each client is connected to the second database through the enterprise service bus ESB. Correspondingly, the current execution end also receives data update operations on the second database from multiple clients, including deactivation, addition, modification, etc., which are not specifically limited in the embodiment of the present invention.

It should be noted that after receiving the client's data update instruction for new data, the current execution end first determines the data category of the data to be added, and then queries the association between the data category and the data template based on the data category to determine the target. data template, and then add standardized data specified by the template based on the target data template, including standardized associated data, standardized data types, standardized data formats, etc., which are not specifically limited in the embodiment of the present invention.

Embodiments of the present invention provide a data processing method. Compared with the existing technology, the present invention obtains first database data and second database data, and performs data processing on the second database data based on the first database data. Deduplication processing to obtain the second database data after deduplication; obtaining a data template, and performing standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data is consistent with The first database data has the same data-related information; data updating of the first database data based on the standardized conversion data can realize data interaction and sharing between the parent company and the subsidiary company, effectively avoiding data islands. The formation further provides an important data basis for decision-making analysis of data resources.

As an implementation of the method shown in Figure 1, an embodiment of the present invention provides a data processing device, as shown in Figure 5. The device includes:

The deduplication module 51 is used to obtain the first database data and the second database data, and perform data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data;

The conversion module 52 is used to obtain a data template, and perform standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data has the same characteristics as the first database data. The same data-related information;

The update module 53 is configured to update the first database data based on the standardized conversion data.

Further, the device also includes an index adding module for:

classifying the first database data and the second database data into a plurality of data categories;

The same data index is added to the first database data and the second database data having the same data category.

Further, the deduplication module 51 is also used to:

Obtain the first data index of the first database data, and obtain the second data index of the second database data;

Perform matching processing on the first data index and the second data index, and obtain target first database data and target second database data having the same data index;

Perform data deduplication processing on the target second database data based on the target first database data to obtain the deduplicated second database data.

Further, the device also includes a template creation module for:

Obtain data-related information of the first database data, where the data-related information includes data categories, data types, data formats and associated data;

Construct data templates with the same number as the data categories based on the data category, the data type, the data format and the associated data;

An association relationship is established between the data template and the data category.

Further, the conversion module 52 is also used to:

Based on the data category of the second database data after deduplication, query the association relationship and determine the target data template;

Obtain the target data type, target data format and target associated data of the target data template;

Standardized conversion is performed on the deduplicated second database data based on the target data type, the target data format and the target associated data to obtain standardized conversion data.

Further, the update module 53 is also used to:

Send an update request carrying the standardized conversion data to the first database; so that the first database performs an approval operation based on the update request, and after the update request is approved, update the standardized conversion data to the in the first database.

Further, the device also includes a receiving and updating module for:

Receive data access to the second database from multiple clients, so that the clients obtain the standardized conversion data; the clients are connected to the second database through an enterprise service bus;

Correspondingly, data update operations on the second database from multiple clients are received, including deactivation, addition and modification.

Embodiments of the present invention provide a data processing device. Compared with the existing technology, the present invention obtains first database data and second database data, and performs data processing on the second database data based on the first database data. Deduplication processing to obtain the second database data after deduplication; obtaining a data template, and performing standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data is consistent with The first database data has the same data-related information; data updating of the first database data based on the standardized conversion data can realize data interaction and sharing between the parent company and the subsidiary company, effectively avoiding data islands. The formation further provides an important data basis for decision-making analysis of data resources.

According to an embodiment of the present invention, a storage medium is provided. The storage medium stores at least one executable instruction. The computer-executable instruction can execute the data processing method in any of the above method embodiments.

FIG. 6 shows a schematic structural diagram of a computer device according to an embodiment of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

As shown in Figure 6, the computer device may include: a processor (processor) 602, a communications interface (Communications Interface) 604, a memory (memory) 606, and a communication bus 608.

Among them: the processor 602, the communication interface 604, and the memory 606 complete communication with each other through the communication bus 608.

The communication interface 604 is used to communicate with network elements of other devices such as clients or other servers.

The processor 602 is used to execute the program 610, and specifically can execute the relevant steps of the above-mentioned data processing method.

Specifically, program 610 may include program code including computer operating instructions.

The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the computer device may be the same type of processor, such as one or more CPUs; or they may be different types of processors, such as one or more CPUs and one or more ASICs.

Memory 606 is used to store program 610. The memory 606 may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 610 may be specifically used to cause the processor 602 to perform the following operations:

Obtain the first database data and the second database data, and perform data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data;

Obtain a data template, and perform standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data and the first database data have the same data-related information;

Data updating is performed on the first database data based on the standardized conversion data.

Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented using general-purpose computing devices. They can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices. , optionally, they may be implemented in program code executable by a computing device, such that they may be stored in a storage device for execution by the computing device, and in some cases, may be in a sequence different from that herein. The steps shown or described are performed either individually as individual integrated circuit modules, or as multiple modules or steps among them as a single integrated circuit module. As such, the invention is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. A data processing method, characterized by including: Obtain the first database data and the second database data, and perform data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data; Obtain a data template, and perform standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data and the first database data have the same data-related information; Data updating is performed on the first database data based on the standardized conversion data. 2. The method according to claim 1, characterized in that before obtaining the first database data and the second database data, the method further includes: classifying the first database data and the second database data into a plurality of data categories; The same data index is added to the first database data and the second database data having the same data category. 3. The method according to claim 1, characterized in that, performing data deduplication processing on the second database data based on the first database data, and obtaining the deduplicated second database data includes: Obtain the first data index of the first database data, and obtain the second data index of the second database data; Perform matching processing on the first data index and the second data index, and obtain target first database data and target second database data having the same data index; Perform data deduplication processing on the target second database data based on the target first database data to obtain the deduplicated second database data. 4. The method according to claim 1, characterized in that before obtaining the data template, the method further includes: Obtain data-related information of the first database data, where the data-related information includes data categories, data types, data formats and associated data; Construct data templates with the same number as the data categories based on the data category, the data type, the data format and the associated data; An association relationship is established between the data template and the data category. 5. The method according to claim 4, characterized in that said obtaining a data template and performing standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data includes: Based on the data category of the second database data after deduplication, query the association relationship and determine the target data template; Obtain the target data type, target data format and target associated data of the target data template; Standardized conversion is performed on the deduplicated second database data based on the target data type, the target data format and the target associated data to obtain standardized conversion data. 6. The method of claim 5, wherein updating the first database data based on the standardized conversion data includes: Send an update request carrying the standardized conversion data to the first database; so that the first database performs an approval operation based on the update request, and after the update request is approved, update the standardized conversion data to the in the first database. 7. The method according to any one of claims 1 to 6, characterized in that the method further includes: Receive data access to the second database from multiple clients, so that the clients obtain the standardized conversion data; the clients are connected to the second database through an enterprise service bus; Correspondingly, data update operations on the second database from multiple clients are received, including deactivation, addition and modification. 8. A data processing device, characterized in that it includes: A deduplication module, used to obtain the first database data and the second database data, and perform data deduplication processing on the second database data based on the first database data to obtain the deduplicated second database data; A conversion module, configured to obtain a data template, and perform standardized conversion on the deduplicated second database data based on the data template to obtain standardized conversion data, so that the standardized conversion data has the same characteristics as the first database data. data related information; An update module, configured to update the first database data based on the standardized conversion data. 9. A storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction executes an operation corresponding to the data processing method according to any one of claims 1 to 7. 10. A computer device, including a processor, a memory, a communication interface and a communication bus. The processor, the memory and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the data processing method according to any one of claims 1 to 7.