CN118535658A - Data processing method, device, equipment, medium and program product - Google Patents

Abstract

The present disclosure provides a data processing method, device, equipment, medium and program product, which can be applied to the fields of data processing technology and financial technology technology. The method includes: in response to a scheduled task in a data management system being triggered, determining a target cluster from a data warehouse according to the scheduled task, wherein the data management system is used to manage metadata in the data warehouse, and the scheduled task is used to keep the metadata in the data management system consistent with the metadata in the target cluster; obtaining a processing instruction for processing the metadata in the target cluster; determining a target instruction for changing the metadata in the target cluster from the processing instruction; determining target metadata from the target cluster according to the target instruction, wherein the target metadata represents the metadata processed by the target instruction; and processing the metadata in the data management system according to the target metadata.

Description

Data processing method, device, equipment, medium and program product

Technical Field

The present disclosure relates to the fields of data processing technology and financial technology, and specifically to a data processing method, device, equipment, medium and program product.

Background Art

Metadata is data that describes data and can record descriptive information about the data. Metadata can be used to manage and maintain data. For example, metadata includes information such as the storage format, indexing, or partitioning strategy of a data set. When querying related information, metadata can be used for querying.

In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: the processing method of metadata in the related art is inefficient and consumes a lot of resources.

Summary of the invention

In view of the above problems, the present disclosure provides a data processing method, apparatus, device, medium and program product.

According to a first aspect of the present disclosure, there is provided a data processing method, comprising:

In response to a scheduled task in the data management system being triggered, determining a target cluster from the data warehouse according to the scheduled task, wherein the data management system is used to manage metadata in the data warehouse, and the scheduled task is used to keep the metadata in the data management system consistent with the metadata in the target cluster;

Obtaining processing instructions for processing metadata in the target cluster;

Determine a target instruction for performing a change process on the metadata in the target cluster from the processing instructions;

Determining target metadata from the target cluster according to the target instruction, wherein the target metadata represents metadata processed by the target instruction; and

The metadata in the data management system is processed according to the target metadata.

According to an embodiment of the present disclosure, determining target metadata from the target cluster according to the target instruction includes:

According to the target instruction, determining the target identifier corresponding to the target instruction;

Determine a target data table matching the target identifier from the target cluster; and

The target metadata is determined according to the update time and update content of the target data table.

According to an embodiment of the present disclosure, the metadata in the data management system is processed according to the target metadata, including:

Determine, according to the change type of the target metadata, how to process the metadata in the data management system;

According to the above processing method, the metadata in the above data management system is processed.

According to an embodiment of the present disclosure, the above-mentioned change type includes a modification type, an addition type and a deletion type, wherein the above-mentioned determination of a processing method for the metadata in the above-mentioned data management system according to the change type of the above-mentioned target metadata includes:

In the case where it is determined that the change type of the target metadata is a modification type, determining that the processing method for the metadata of the data management system is a modification processing method;

In the case where it is determined that the change type of the target metadata is an addition type, determining that the processing method for the metadata of the data management system is an addition processing method;

When it is determined that the change type of the target metadata is a deletion type, the processing method of the metadata of the data management system is determined to be a deletion processing method.

According to an embodiment of the present disclosure, the metadata in the data management system is processed according to the processing method, including:

In the case where it is determined that the processing method is the modification processing method, obtaining first metadata to be modified related to the target metadata from the data management system;

Modifying the first metadata to be modified according to the target metadata;

In the case where the above processing method is the above adding processing method, determining the second metadata to be modified according to the above target metadata;

Adding the second metadata to be modified in the data management system;

In the case where the processing method is the deletion processing method, obtaining third metadata to be modified related to the target metadata from the data management system;

The third metadata to be modified is deleted from the data management system.

According to an embodiment of the present disclosure, the above method further includes:

In response to the data management system receiving a query request about business data from a target object, performing identity authentication on the target object;

When it is determined that the identity authentication of the target object is passed, obtaining metadata related to the business data from the data management system;

Determine a target query template from a set of business data query templates according to metadata related to the business data;

In response to receiving a processing request from the target object regarding the target query template, the business data is acquired from the data warehouse.

According to an embodiment of the present disclosure, the above method further includes:

According to the cluster size of the target cluster in the data warehouse, determine the number of parallel processing and the processing frequency that match the target cluster;

The scheduled task is generated according to the parallel processing data volume and the processing frequency.

According to an embodiment of the present disclosure, the above method further includes:

When it is determined that the metadata in the target cluster is updated for the first time in the data management system, all metadata in the target cluster are updated to the data management system.

A second aspect of the present disclosure provides a data processing device, including:

A first determination module, configured to determine a target cluster from a data warehouse in response to a scheduled task in a data management system being triggered according to the scheduled task, wherein the data management system is configured to manage metadata in the data warehouse, and the scheduled task is configured to keep the metadata in the data management system consistent with the metadata in the target cluster;

A first acquisition module is used to acquire a processing instruction for processing the metadata in the target cluster;

A second determining module is used to determine a target instruction for performing a change process on the metadata in the target cluster from the processing instructions;

A third determination module is configured to determine target metadata from the target cluster according to the target instruction, wherein the target metadata represents metadata processed by the target instruction; and

The processing module is used to process the metadata in the data management system according to the target metadata.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

The fourth aspect of the present disclosure further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the computer program or instructions are executed by a processor.

According to the data processing method, device, equipment, medium and program product provided by the present disclosure, in response to the triggering of a scheduled task in a data management system, a target cluster is determined from a data warehouse according to the scheduled task, wherein the data management system is used to manage metadata in the data warehouse, and the scheduled task is used to keep the metadata in the data management system consistent with the metadata in the target cluster; obtain a processing instruction for processing the metadata in the target cluster; determine a target instruction for changing the metadata in the target cluster from the processing instruction; determine the target metadata from the target cluster according to the target instruction, wherein the target metadata represents the metadata processed by the target instruction; and process the metadata in the data management system according to the target metadata. Since the target cluster in the data warehouse can be automatically determined by setting a scheduled task, the target instruction for changing the metadata in the target cluster is obtained according to the processing instruction for processing the metadata in the target cluster, and the target metadata that has been changed can be determined from the second. The data management system is processed based on the target metadata, and there is no need to perform a full update according to all the metadata in the target cluster, thereby avoiding the full update occupying a large amount of resources and improving resource utilization efficiency. In addition, it is convenient and quick to determine the target metadata through instructions, and there is no need to compare the new and old data.

BRIEF DESCRIPTION OF THE DRAWINGS

The above contents and other purposes, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

FIG1 schematically shows an application scenario diagram of a data processing method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

FIG2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure;

FIG3 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure;

FIG4 schematically shows a block diagram of a data processing framework according to an embodiment of the present disclosure;

FIG5 schematically shows a structural block diagram of a data processing device according to an embodiment of the present disclosure; and

FIG6 schematically shows a block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and/or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and/or A, B, C, etc.).

In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

Data management systems mainly serve users such as data analysts, who have a high frequency demand for online metadata queries. For example, a data analyst may need to perform the following operations: Data lineage analysis: Data analysts need to quickly access metadata to track the source of data, understand the historical changes of data, and how data is processed and transformed, which is critical to ensuring the accuracy and reliability of data. Performance optimization query: In order to improve query efficiency, data analysts may need to view the storage format, index status, or partition strategy of a specific data set, which is contained in metadata. Security and compliance audit: Understanding who can access specific data sets, as well as the access rights and history of these data sets, is another key task for data analysts, and metadata usually contains this information. Resource management and estimation: Data analysts need to understand the size and usage of each data set in order to better allocate resources and predict future storage needs, which also involves querying metadata. These scenarios show that online metadata queries are not only frequent but also critical for users of data management systems, so improving the efficiency and performance of this link is crucial to the success of the entire data governance system.

In related technologies, a massively parallel processing database (MPPDB) distributed cluster is usually used to build a data warehouse. The data warehouse processes data through a batch scheduling system and then provides it for query by data analysts. The following are typical scenarios for data import, processing, and use: Data import: A large amount of business data generated every day is imported into the data warehouse. During the import process, the data may need to be format converted, cleaned, and verified to ensure data quality. Data processing: After the data is imported, the batch scheduling system will start processing the data. The processing process includes operations such as data merging, aggregation, and calculation to generate a data set that can be used for analysis. For example, it may be necessary to calculate the total transaction volume for a day, or generate a summary of customer behavior. Data use: After the data is processed, data analysts can query the processed data through the query interface. They may perform risk analysis, trend prediction, behavioral analysis, and other operations. At this stage, efficient metadata queries are crucial because they enable data analysts to quickly understand the structure and characteristics of the data set.

Current data management systems often require full processing when acquiring metadata. However, full processing has limitations. Since the MPPDB cluster is mainly oriented to high-throughput massive data processing, it may encounter performance bottlenecks when processing high-frequency and diverse metadata query requirements. This is because the object of MPPDB cluster optimization is mainly data, rather than frequent and flexible queries on metadata. In addition, batch processing may encounter the following problems when processing extremely large amounts of metadata: Low efficiency of batch processing: When processing a large amount of metadata, the batch processing module is prone to job interruptions due to long processing time. Resource consumption and benefits do not match: In order to achieve full cleaning and loading, the data management system needs to invest a lot of job resources, but at the same time, since metadata query is a non-core requirement, the benefits of its output are often not proportional to the resource investment. Difficulty in identifying metadata for incremental changes: Since data warehouses usually contain a huge amount of data, when processing this data, even incremental updates involve huge data sets, making it complex and time-consuming to effectively identify and process updates. Timely updates of data are crucial for decision support, risk management, compliance, and other aspects. This requires that incremental updates must be not only accurate but also fast. When dealing with incremental updates, data consistency and accuracy must be ensured. This means that any updates or deletes must be accurately reflected in the metadata to prevent data inconsistencies or information loss.

In view of this, an embodiment of the present disclosure provides a data processing method, including responding to a scheduled task in a data management system being triggered, determining a target cluster from a data warehouse according to the scheduled task, wherein the data management system is used to manage metadata in the data warehouse, and the scheduled task is used to keep the metadata in the data management system consistent with the metadata in the target cluster; obtaining processing instructions for processing the metadata in the target cluster; determining target instructions for changing the metadata in the target cluster from the processing instructions; determining target metadata from the target cluster according to the target instructions, wherein the target metadata represents the metadata processed by the target instructions; and processing the metadata in the data management system according to the target metadata.

FIG1 schematically shows an application scenario diagram of a data processing method, apparatus, device, medium, and program product according to an embodiment of the present disclosure.

As shown in Fig. 1, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and/or the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in Figure 1 is only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

FIG2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure.

As shown in FIG. 2 , the data processing method of this embodiment includes operations S210 to S250 .

In operation S210, in response to a scheduled task in the data management system being triggered, a target cluster is determined from the data warehouse according to the scheduled task, wherein the data management system is used to manage metadata in the data warehouse, and the scheduled task is used to keep the metadata in the data management system consistent with the metadata in the target cluster.

In operation S220 , a processing instruction for processing metadata in the target cluster is obtained.

In operation S230 , a target instruction for performing a change process on metadata in a target cluster is determined from the processing instructions.

In operation S240 , target metadata is determined from the target cluster according to the target instruction, wherein the target metadata represents metadata processed by the target instruction.

In operation S250, metadata in the data management system is processed according to target metadata.

According to an embodiment of the present disclosure, a data warehouse may include business data and metadata, where the business data is the actual data of the business, and the metadata is the data that describes the business data.

According to an embodiment of the present disclosure, in order to achieve better analysis of metadata, metadata may be managed through a data management system.

According to an embodiment of the present disclosure, in order to ensure that the metadata in the data management system is consistent with the metadata in the target cluster, a scheduled task is set in the data management system so that the metadata in the data warehouse can be regularly updated to the data management system.

According to an embodiment of the present disclosure, a scheduled task may include a cluster range that needs to be synchronized for this task, and a target cluster is determined from a data warehouse according to the scheduled task.

According to an embodiment of the present disclosure, processing instructions for processing metadata in the target cluster may be obtained based on the processing log of the target cluster. The processing instructions may include calling instructions, viewing instructions, and changing instructions for the metadata.

According to an embodiment of the present disclosure, a change instruction may be an instruction for performing change processing on metadata. After the metadata is changed, the metadata changes. In order to ensure the consistency between the metadata in the data management system and the metadata in the target cluster, the metadata in the data management system needs to be processed. According to the change instruction, the target metadata that has undergone the change can be better determined, thereby processing the metadata in the data management system to ensure that the metadata in the data management system is consistent with the metadata in the target cluster.

According to the embodiments of the present disclosure, the target cluster in the data warehouse can be automatically determined by setting a scheduled task, and the target instruction for changing the metadata in the target cluster can be screened according to the processing instruction for processing the metadata in the target cluster, so that the changed target metadata can be determined, and the data management system can be processed based on the target metadata, without the need to perform a full update based on all the metadata in the target cluster, thus avoiding the full update from occupying a large amount of resources and improving resource utilization efficiency. In addition, it is convenient and quick to determine the target metadata through instructions, and there is no need to compare the new and old data.

According to an embodiment of the present disclosure, target metadata is determined from a target cluster according to a target instruction, including: determining a target identifier corresponding to the target instruction according to the target instruction; determining a target data table matching the target identifier from the target cluster; and determining the target metadata according to an update time and update content of the target data table.

According to an embodiment of the present disclosure, a processing instruction can leave an identifier in the data table where the metadata is located after processing the metadata. Therefore, the target instruction will also leave an identifier in the data table after processing the metadata. The target data table can be determined from the target cluster based on the target identifier corresponding to the target instruction. The target data table can have a target identifier, that is, the target data table has been processed by the target instruction.

According to the embodiments of the present disclosure, the update time and update content of the target data table can be obtained, and when the update time meets the conditions, the target metadata can be determined according to the update content.

According to an embodiment of the present disclosure, the update time satisfies the condition that the update time is between the current scheduled task and the last scheduled task. The update content may be a change in a field in a target data table, and an update file may be generated including the target cluster, database name, table name, field change status, update time, etc. of the target data table.

According to an embodiment of the present disclosure, the target data table is identified by a target identifier corresponding to a target instruction. Compared with identifying the target data table by comparing new and old data, it is not necessary to compare all the data, which is more efficient.

According to an embodiment of the present disclosure, metadata in a data management system is processed according to target metadata, including: determining a processing method for metadata in the data management system according to a change type of the target metadata; and processing metadata in the data management system according to the processing method.

According to an embodiment of the present disclosure, the target metadata may include different change types, and different change types correspond to different processing methods. According to the change type of the target metadata, a processing method that matches the metadata of the data management system can be determined, so that the metadata in the data management system can be processed according to the processing method.

According to an embodiment of the present disclosure, the change type may include a modification type, an addition type, and a deletion type.

According to an embodiment of the present disclosure, when it is determined that the change type of the target metadata is a modification type, the processing method for the metadata of the data management system is determined to be a modification processing method; when it is determined that the processing method is a modification processing method, the first metadata to be modified related to the target metadata is obtained from the data management system; according to the target metadata, the first metadata to be modified is modified.

According to an embodiment of the present disclosure, the change type of the target metadata is a modification type, and the first metadata to be modified related to the target metadata may be metadata consistent with data before the target metadata is changed in the data management system.

According to an embodiment of the present disclosure, the first metadata to be modified may be modified to be data consistent with the target metadata, thereby ensuring that the metadata of the data management system is consistent with the metadata in the target cluster.

According to an embodiment of the present disclosure, when it is determined that the change type of the target metadata is an addition type, the processing method for the metadata of the data management system is determined to be an addition processing method; when the processing method is the addition processing method, the second metadata to be modified is determined according to the target metadata; and the second metadata to be modified is added in the data management system.

According to an embodiment of the present disclosure, since the change type of the target metadata is an addition type, that is, the metadata is not included in the data management system, the second metadata to be modified can be the same data as the target metadata. The second metadata to be modified is added to the data management system to ensure that the metadata of the data management system is consistent with the metadata in the target cluster.

According to an embodiment of the present disclosure, when it is determined that the change type of the target metadata is a deletion type, a processing method for the metadata of the data management system is determined to be a deletion processing method; when the processing method is the deletion processing method, third metadata to be modified related to the target metadata is obtained from the data management system; and the third metadata to be modified is deleted from the data management system.

According to an embodiment of the present disclosure, the change type of the target metadata is a deletion type, and the data identical to the target metadata in the data management system also needs to be deleted. The third metadata to be modified can be the same data as the target metadata. The third metadata to be modified is deleted from the data management system to ensure that the metadata of the data management system is consistent with the metadata in the target cluster.

According to an embodiment of the present disclosure, the metadata in the data management system can be processed in a corresponding manner according to the modification type, addition type or deletion type of the target metadata, ensuring that the metadata of the data management system is consistent with the metadata in the target cluster.

The following is an example to illustrate how to determine the target metadata.

According to an embodiment of the present disclosure, it is assumed that the following change information is detected in the data warehouse: cluster name: ClusterA; database name: Database1; table name: CustomerInfo; field changes: 1. Field name: Age, field type changed from Integer to Varchar; 2. New field name: Email, field type: Varchar; last updated on January 1, 2024; status: updated. The relevant information of the target metadata can be recorded in the update file, which is as shown in Table 1 below:

Table 1

In this example, the CustomerInfo table has undergone a structural change in the Database1 database, where the type of one field has been modified and a new field has been added. These changes are recorded in the update file so that the data management system can synchronize these changes and ensure that its records are consistent with the actual data warehouse state.

According to an embodiment of the present disclosure, the deleted table in the business can be identified and confirmed based on the drop table information in the log, and its status can be set to deleted. Add a scenario of deleting a table based on the above example. Assume that the data asset application system also detects the deletion of a table during the same incremental update process: Cluster name: ClusterA; Database name: Database2; Table name: OldSalesData; Last updated: January 1, 2024; Status: Deleted. The relevant information of the target metadata can be recorded in the update file, and the update file is as follows in Table 2:

Table 2

In this additional scenario, the OldSalesData table is deleted in the Database2 database. This change is also recorded in the update file. The field name and field type columns are empty because the entire table is deleted. Based on this record, the data management system will delete the metadata of the corresponding table from its library to ensure data consistency and accuracy.

According to an embodiment of the present disclosure, records in an update state are modified or added, and records in a deletion state are deleted, so as to maintain the consistency of the clusters in the data management system and the data warehouse.

FIG3 schematically shows a flow chart of a data processing method according to yet another embodiment of the present disclosure.

As shown in FIG. 3 , the data processing method of this embodiment includes operations S310 to S340 .

In operation S310 , in response to the data management system receiving a query request about business data from a target object, the target object is authenticated.

In operation S320, when it is determined that the identity authentication of the target object is passed, metadata related to the business data is obtained from the data management system.

In operation S330 , a target query template is determined from a set of business data query templates according to metadata related to the business data.

In operation S340 , in response to receiving a processing request from a target object regarding a target query template, business data is acquired from a data warehouse.

According to an embodiment of the present disclosure, the target object may be a data analyst, and the target object may query business data, and the business data is queried through metadata. The data management system may perform identity authentication on the target object when receiving a query request from the target object about business data to ensure data security. When it is determined that the identity authentication of the target object is passed, metadata related to the business data may be obtained from the data management system based on the identification information of the business data, and a target query template may be determined by matching from a set of business data query templates based on the metadata related to the business system. The target query template may be a template that matches the metadata of the business data queried by the target object, that is, the metadata may be filled in the target query template, and the business data may be accurately queried using the target query template, and the target query template may be displayed to the target object, and the business data may be obtained from the data warehouse based on the processing request of the target object about the target query template.

According to an embodiment of the present disclosure, the steps for the target object to query business data are as follows: identity authentication is performed in the data management system. After the identity authentication is passed, the target object can determine the metadata related to the queried business data based on the metadata related to the business data obtained in the data management system, thereby determining the target query template in the business data query template. The target object can fill in the target query template based on the metadata related to the queried business data, and the data management system can also automatically fill in the target query template based on the metadata related to the business data. After completion, click query to connect to the data warehouse and obtain business data.

According to the embodiments of the present disclosure, by authenticating the target object, the security of the data is improved, and by setting the query template, the convenience of the query is increased, thereby preventing the user from not knowing where to query during the query process.

According to an embodiment of the present disclosure, the data processing method may further include: determining the number of parallel processing and the processing frequency that match the target cluster according to the cluster size of the target cluster in the data warehouse; and generating a scheduled task according to the parallel processing data volume and the processing frequency.

According to an embodiment of the present disclosure, the scale of the cluster may include cluster-level, database-level, schema-level, and table-level blacklists. Among them, the cluster level may be the default option, which is suitable for smaller-scale clusters. It is not recommended to enable it for large-scale clusters. The database level may be suitable for medium-sized clusters or clusters with a single business scenario. The schema level may be suitable for ultra-large-scale clusters, in which a single database runs multiple business loads and classifies data according to the schema. The table-level blacklist can further narrow the scope that does not need to be scanned through regular matching, excluding temporary tables, external tables, etc. in the cluster. Different levels may correspond to different numbers of parallel processing and processing frequencies. For example, the number of parallel processing corresponding to the cluster-level cluster may be 5, and the processing frequency may be daily, the number of parallel processing corresponding to the database-level cluster may be 10, and the processing frequency may be weekly, and the number of parallel processing corresponding to the schema-level cluster may be the parallelism of 20 tables, and the processing frequency may be monthly.

According to the embodiments of the present disclosure, the corresponding level can be determined according to the cluster size of the target cluster, and the corresponding parallel processing data volume and processing frequency can be determined, thereby generating a scheduled task.

According to the embodiments of the present disclosure, the data management system can process and organize the metadata in the data warehouse for easy analysis. Therefore, when obtaining data in the data warehouse, the name of the data warehouse cluster, the name of the database in the cluster, the name of the table in the database, the name of the field in the table, the data type of each field, the last update time of the table or field, the status, etc. can be obtained.

According to the embodiments of the present disclosure, by setting the scheduled tasks according to the size of the target cluster, resources can be better utilized and waste of resources can be avoided.

According to an embodiment of the present disclosure, the data processing method may further include: when it is determined that the metadata in the target cluster is updated for the first time in the data management system, updating all metadata in the target cluster to the data management system.

According to an embodiment of the present disclosure, during the first update, it is necessary to obtain the full amount of data in the target cluster, and subsequent updates are all based on the full amount of data obtained for the first time.

According to an embodiment of the present disclosure, the metadata updated for the first time may include cluster name, database name, table name, field name, field type, last update time, and status (all marked as updated). The result file is fully loaded into the data management system for query by the target object.

According to the embodiments of the present disclosure, all data can be updated to the data management system through the first full update, ensuring the integrity of the data and serving as a basis for subsequent processing.

FIG4 schematically shows a block diagram of a data processing framework according to an embodiment of the present disclosure.

As shown in FIG. 4 , the data processing framework 400 includes a data warehouse 410 , a data management system 420 and a data analysis service system 430 .

The data warehouse 410 includes a massively parallel processing database (MPPDB) cluster module and a batch processing module, the data management system 420 includes a data batch update module, a data real-time update module, a data query interface module and a data online module, and the data analysis service system 430 includes a business data query template module, an identity authentication module, a business data query module and a metadata query module.

According to the embodiments of the present disclosure, the MPPDB cluster module, as the core of data processing and storage, is responsible for storing a large number of data sets including business data and metadata. For example, data such as transaction records, market data, and user information can be stored. The batch processing module can process a large amount of data from multiple sources, perform data cleaning, conversion, and aggregation operations to prepare the data for subsequent analysis. This module is also responsible for generating and updating metadata, such as the structure of the data set, change history, and access rights.

According to an embodiment of the present disclosure, the data batch update module can regularly update the processed metadata in the data warehouse 410 to the data management system 420. This does not involve the business data itself, but focuses on the descriptive part of the data assets, that is, metadata, such as data models, field definitions, data quality indicators, etc. The data real-time update module can correspond to the batch update module. The data real-time update module is responsible for processing metadata changes that need to be reflected immediately, and can provide real-time and accurate data analysis. For example, when a new data set is added or the structure of an existing data set changes, the data real-time update module will update the metadata in a timely manner. The data query interface module can provide an interface for accessing and querying metadata for the target object, and the data query interface module can support complex query operations on data assets, such as data source analysis, data set structure query, etc. The data online module can efficiently process metadata queries, and the database in the data online module can store all metadata information and support high-frequency queries so that the target object can quickly access and understand the details of the data assets.

According to an embodiment of the present disclosure, the business data query template module provides a set of predefined business data query templates for common business data queries. The business data query template set is directly connected to the MPPDB cluster module of the data warehouse 410, allowing the target object to quickly perform standardized data queries without having to deeply understand the underlying data structure. The identity authentication module verifies the identity of the target object accessing the data analysis service system 430. The identity authentication module ensures that only authorized target objects can access sensitive business data and metadata, thereby protecting the security and compliance of the data. The business data query module allows the target object to perform custom queries to access the business data in the data warehouse 410 according to the target query module. The business data query module is tightly integrated with the MPPDB cluster module to provide efficient data access and processing capabilities. The metadata query module is used to query and access metadata stored in the data management system. Through this metadata query module, the target object can obtain detailed information about the data, such as data table structure, field definition, data quality indicators, etc.

The data processing method of the embodiment of the present disclosure is described below by using an example.

According to an embodiment of the present disclosure, it is assumed that the data warehouse includes two tables in a database instance DB_Finance in a certain MPPDB cluster ClusterA: Customer_Transactions and Account_Balances. Two tables are selected in the scheduled task: Customer_Transactions and Account_Balances in the DB_Finance database.

When the data management system is run for the first time, full metadata of the two tables is extracted, including the field name and type of each table, and stored in the online library of the data asset management platform.

At a certain moment, the scheduled task is triggered, the Customer_Transactions table adds a new field Transaction_Location, the Account_Balances table adds a new record, and the field Last_Updated_Timestamp is modified, that is, both tables are changed. The data management system identifies the changes in the two tables through audit logs or last modification time. For Customer_Transactions, the data management system only extracts the information of the new field Transaction_Location for update. For Account_Balances, the system updates the latest record and the Last_Updated_Timestamp field information.

When the target object needs to query the latest structure of the two tables, it sends a query request through the data analysis service system. The data asset query module accesses the data management system and provides the latest structure information of the two tables, including the new fields of the Customer_Transactions table and the updated records of the Account_Balances table. The target object uses this latest table structure information to perform complex data analysis, which may be cross-table analysis, such as analyzing the correlation between customer transaction behavior and account balances. The analysis results can be used to generate reports or insights to support business decisions.

It can be seen that the data management system and the data analysis service system manage the metadata changes of multiple tables at the same time, ensuring that the target objects can access the latest and accurate multi-source data information, so as to conduct effective data analysis, improve the efficiency of data processing, and reduce repetitive full data extraction work. It is suitable for dynamically changing and large-scale financial data warehouse environments.

According to the embodiments of the present disclosure, the data processing method of the embodiments of the present disclosure can improve the efficiency of metadata processing in the data warehouse and reduce the time cost of metadata update and maintenance. Through precise metadata management and incremental update processing, the quality and accuracy of metadata in the data management system are guaranteed, providing a solid data foundation for data analysis and business decision-making. It can also meet the needs of complex data analysis.

Based on the above data processing method, the present disclosure further provides a data processing device, which will be described in detail below in conjunction with FIG5 .

FIG5 schematically shows a structural block diagram of a data processing device according to an embodiment of the present disclosure.

As shown in FIG. 5 , the data processing device 500 of this embodiment includes a first determining module 510 , a first acquiring module 520 , a second determining module 530 , a third determining module 540 and a processing module 550 .

The first determination module 510 is used to determine the target cluster from the data warehouse according to the scheduled task in response to the timing task in the data management system being triggered, wherein the data management system is used to manage the metadata in the data warehouse, and the scheduled task is used to keep the metadata in the data management system consistent with the metadata in the target cluster. In one embodiment, the first determination module 510 can be used to perform the operation S210 described above, which will not be repeated here.

The first acquisition module 520 is used to acquire a processing instruction for processing metadata in the target cluster. In one embodiment, the first acquisition module 520 can be used to perform the operation S220 described above, which will not be described in detail here.

The second determination module 530 is used to determine a target instruction for performing a change process on the metadata in the target cluster from the processing instruction. In one embodiment, the second determination module 530 can be used to perform the operation S230 described above, which will not be described in detail here.

The third determination module 540 is used to determine target metadata from the target cluster according to the target instruction, wherein the target metadata represents metadata processed by the target instruction. In one embodiment, the third determination module 540 can be used to perform the operation S240 described above, which will not be described in detail here.

The processing module 550 is used to process the metadata in the data management system according to the target metadata. In one embodiment, the processing module 550 can be used to perform the operation S250 described above, which will not be described in detail here.

According to an embodiment of the present disclosure, the third determination module 540 for determining target metadata from the target cluster according to the target instruction includes:

A first determining unit, configured to determine a target identifier corresponding to the target instruction according to the target instruction;

A second determining unit is used to determine a target data table matching the target identifier from the target cluster; and

The third determining unit is used to determine the target metadata according to the update time and update content of the target data table.

According to an embodiment of the present disclosure, the processing module 550 for processing metadata in the data management system according to target metadata includes:

A first processing unit, configured to determine a processing method for the metadata in the data management system according to a change type of the target metadata;

The second processing unit is used to process the metadata in the data management system according to the processing method.

According to an embodiment of the present disclosure, the change type includes a modification type, an addition type, and a deletion type, wherein the first processing unit for determining the processing method of the metadata in the data management system according to the change type of the target metadata includes:

The first processing subunit is used to determine that the processing mode of the metadata of the data management system is a modification processing mode when it is determined that the change type of the target metadata is a modification type;

The second processing subunit is used to determine that the processing mode of the metadata of the data management system is an addition processing mode when it is determined that the change type of the target metadata is an addition type;

The third processing subunit is used to determine that the processing method for the metadata of the data management system is a deletion processing method when it is determined that the change type of the target metadata is a deletion type.

According to an embodiment of the present disclosure, the second processing unit for processing metadata in the data management system according to the processing mode includes:

The fourth processing subunit is used to obtain first metadata to be modified related to the target metadata from the data management system when the processing mode is determined to be the modification processing mode;

A fifth processing subunit, configured to modify the first metadata to be modified according to the target metadata;

A sixth processing subunit, configured to determine second metadata to be modified according to the target metadata when the processing mode is an adding processing mode;

A seventh processing subunit, configured to add second metadata to be modified in the data management system;

an eighth processing subunit, configured to obtain third metadata to be modified related to the target metadata from the data management system when the processing mode is a deletion processing mode;

The ninth processing subunit is configured to delete the third metadata to be modified from the data management system.

According to an embodiment of the present disclosure, the apparatus 500 further includes:

An authentication module, configured to perform identity authentication on a target object in response to a query request about business data received from the target object by the data management system;

The second acquisition module is used to acquire metadata related to the business data from the data management system when it is determined that the identity authentication of the target object has passed;

A fourth determination module, configured to determine a target query template from a set of business data query templates according to metadata related to the business data;

The third acquisition module is used to acquire business data from the data warehouse in response to receiving a processing request from the target object regarding the target query template.

According to an embodiment of the present disclosure, the apparatus 500 further includes:

A fifth determination module, used to determine the number of parallel processing and the processing frequency matching the target cluster according to the cluster size of the target cluster in the data warehouse;

The generation module is used to generate scheduled tasks according to the parallel processing data volume and processing frequency.

According to an embodiment of the present disclosure, the apparatus 500 further includes:

The update module is used to update all metadata in the target cluster to the data management system when it is determined that the metadata in the target cluster is updated for the first time in the data management system.

According to an embodiment of the present disclosure, any multiple modules among the first determination module 510, the first acquisition module 520, the second determination module 530, the third determination module 540 and the processing module 550 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first determination module 510, the first acquisition module 520, the second determination module 530, the third determination module 540 and the processing module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the first determination module 510, the first acquisition module 520, the second determination module 530, the third determination module 540 and the processing module 550 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

FIG6 schematically shows a block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure.

As shown in FIG6 , the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and/or a related chipset and/or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other through bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and/or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

According to an embodiment of the present disclosure, the electronic device 600 may further include an input/output (I/O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

The present disclosure also provides a computer-readable storage medium, which may be included in the device/apparatus/system described in the above embodiments; or may exist independently without being assembled into the device/apparatus/system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and/or RAM 603 described above and/or one or more memories other than ROM 602 and RAM 603.

The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

The above functions defined in the system/device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and/or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and/or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

It will be appreciated by those skilled in the art that the features described in the various embodiments and/or claims of the present disclosure may be combined and/or combined in various ways, even if such combinations and/or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and/or claims of the present disclosure may be combined and/or combined in various ways without departing from the spirit and teachings of the present disclosure. All of these combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these embodiments are only for the purpose of illustration and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims (12)

1. A data processing method, characterized in that the method comprises: In response to a scheduled task in a data management system being triggered, determining a target cluster from a data warehouse according to the scheduled task, wherein the data management system is used to manage metadata in the data warehouse, and the scheduled task is used to keep the metadata in the data management system consistent with the metadata in the target cluster; Obtaining a processing instruction for processing metadata in the target cluster; Determining a target instruction for performing a change process on the metadata in the target cluster from the processing instructions; Determining target metadata from the target cluster according to the target instruction, wherein the target metadata represents metadata processed by the target instruction; and The metadata in the data management system is processed according to the target metadata. 2. The method according to claim 1, wherein determining target metadata from the target cluster according to the target instruction comprises: According to the target instruction, determining a target identifier corresponding to the target instruction; Determining a target data table matching the target identifier from the target cluster; and The target metadata is determined according to the update time and update content of the target data table. 3. The method according to claim 1, characterized in that, according to the target metadata, processing the metadata in the data management system comprises: Determining, according to the change type of the target metadata, a processing method for the metadata in the data management system; According to the processing method, the metadata in the data management system is processed. 4. The method according to claim 3, wherein the change type includes a modification type, an addition type, and a deletion type, wherein determining a processing method for the metadata in the data management system according to the change type of the target metadata comprises: In the case where it is determined that the change type of the target metadata is a modification type, determining that a processing method for the metadata of the data management system is a modification processing method; In the case where it is determined that the change type of the target metadata is an addition type, determining that a processing method for the metadata of the data management system is an addition processing method; When it is determined that the change type of the target metadata is a deletion type, the processing method for the metadata of the data management system is determined to be a deletion processing method. 5. The method according to claim 4, characterized in that the processing of metadata in the data management system according to the processing mode comprises: When it is determined that the processing mode is the modification processing mode, obtaining first metadata to be modified related to the target metadata from the data management system; Modifying the first metadata to be modified according to the target metadata; In a case where the processing mode is the adding processing mode, determining second metadata to be modified according to the target metadata; Adding the second metadata to be modified in the data management system; When the processing mode is the deletion processing mode, obtaining third metadata to be modified related to the target metadata from the data management system; The third metadata to be modified is deleted from the data management system. 6. The method according to any one of claims 1 to 5, characterized in that the method further comprises: In response to the data management system receiving a query request about business data from a target object, performing identity authentication on the target object; When it is determined that the identity authentication of the target object is passed, obtaining metadata related to the business data from the data management system; Determining a target query template from a set of business data query templates according to metadata related to the business data; In response to receiving a processing request from the target object regarding the target query template, acquiring the business data from the data warehouse. 7. The method according to claim 1, characterized in that the method further comprises: Determining the number of parallel processing and the processing frequency that match the target cluster according to the cluster size of the target cluster in the data warehouse; The scheduled task is generated according to the parallel processing data volume and the processing frequency. 8. The method according to claim 1, characterized in that the method further comprises: When it is determined that the metadata in the target cluster is updated for the first time in the data management system, all metadata in the target cluster are updated to the data management system. 9. A data processing device, characterized in that the device comprises: A first determination module, configured to determine a target cluster from a data warehouse in response to a scheduled task in a data management system being triggered according to the scheduled task, wherein the data management system is configured to manage metadata in the data warehouse, and the scheduled task is configured to keep the metadata in the data management system consistent with the metadata in the target cluster; A first acquisition module, used to acquire a processing instruction for processing metadata in the target cluster; A second determining module, configured to determine, from the processing instructions, a target instruction for performing a change process on the metadata in the target cluster; a third determination module, configured to determine target metadata from the target cluster according to the target instruction, wherein the target metadata represents metadata processed by the target instruction; and A processing module is used to process the metadata in the data management system according to the target metadata. 10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8. 11. A computer-readable storage medium having a computer program or instruction stored thereon, wherein the computer program or instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8. 12. A computer program product, comprising a computer program or instructions, characterized in that when the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.