CN114490612A - Data self-cleaning method, device, electronic device and storage medium - Google Patents

Abstract

The invention provides a data self-cleaning method, a device, electronic equipment and a storage medium, wherein for a target data table and target data to be processed, field information of the target data table can be obtained, and as the field information can represent a first field in the target data table and a field sequence between the first field and the second field, after each piece of data in the target data is analyzed to determine a second field and a field value corresponding to the second field, mapping operation can be performed on the first field and the second field according to the field sequence, so that the field value corresponding to the first field of each piece of data is determined. Based on the method and the device, the fields of the unstructured data and the fields in the data table can be automatically kept in the consistent sequence, the accuracy of the data is guaranteed, even if the fields of the subsequent data table are expanded, the expanded fields can be determined in real time through the field information and are continuously kept consistent with the fields in the data table, self-adaptive cleaning is achieved, the workload of manual checking is greatly reduced, and the efficiency is improved.

Description

Data self-cleaning method, device, electronic device and storage medium

technical field

The invention relates to the technical field of data cleaning of big data ETL (Extract-Transform-Load, extraction-transform-load), and more particularly, to a data self-cleaning method, device, electronic device and storage medium.

Background technique

For the big data hive data warehouse, data cleaning is the first step in establishing a data warehouse. Its basic function is to convert unstructured data into structured data according to the fields of the data table (ie hive table) in the hive data warehouse. Provide the most primitive data for subsequent analysis and statistics. In data cleaning, how to automatically and accurately load the fields of unstructured data into the fields corresponding to the hive table has become the most important step.

At present, data cleaning mostly uses the way of manually specified UDF (User Defined Function, user-defined function) output fields to parse unstructured data, which requires that the UDF output fields must be in the order of fields in the data table and in the data table. The fields are in one-to-one correspondence, and once the fields are misplaced, the data of the entire data table will be wrong. However, there are hundreds of data tables in a large hive data warehouse, and each data table may have hundreds of fields. Therefore, a lot of manual work is required to check the UDF output fields, which brings a great workload. , and once the fields are expanded in the subsequent data table, the workload of modifying the UDF output fields and data validation is also quite large.

SUMMARY OF THE INVENTION

In view of this, in order to solve the above problems, the present invention provides a data self-cleaning method, device, electronic equipment and storage medium, and the technical solutions are as follows:

A data self-cleaning method, comprising:

Determine the target data table and target data to be processed, and the target data is unstructured data corresponding to the target data table and contains at least one piece of data;

acquiring field information corresponding to the target data table, where the field information can represent the first field in the target data table and the field order between the first fields;

For each piece of data in the target data, a parsing operation is performed on the piece of data to determine a second field in the piece of data and a field value corresponding to the second field; A mapping operation is performed between a field and the second field to determine the field value corresponding to the first field of the piece of data.

Preferably, the acquiring the field information corresponding to the target data table includes:

obtaining a metadata information table corresponding to the target data table, where the metadata information table at least contains ordered field metadata information;

Read the ordered field metadata information in turn to determine the field matched by the currently read field metadata information;

The determined fields are sequentially written into the determined ordered field list, where the ordered field list is used to store the fields, and the order of the fields is the same as the order of the field metadata information in the metadata information table.

Preferably, the method further includes:

The ordered field list is output.

Preferably, performing the parsing operation on the piece of data includes:

Extract the key-value pair in the piece of data, the key in the key-value pair represents the field, and the value in the key-value pair represents the field value;

A corresponding relationship between fields and field values in the key-value pair is established.

Preferably, performing the mapping operation on the first field and the second field according to the field order includes:

determining a target field matching the first field in the second field according to the field order;

Determine the field value corresponding to the target field, and write the determined field value into the determined ordered data list in turn, and the ordered data list is used to store the field value, and the order of the field value is the same as the order of the field value. The fields are in the same order.

Preferably, the method further includes:

The ordered list of data is output.

A data self-cleaning device, the device includes:

a determination module, configured to determine a target data table to be processed and target data, where the target data is unstructured data corresponding to the target data table and contains at least one piece of data;

an acquisition module, configured to acquire field information corresponding to the target data table, where the field information can represent a first field in the target data table and a field order between the first fields;

a cleaning module, configured to perform a parsing operation on each piece of data in the target data to determine a second field in the piece of data and a field value corresponding to the second field; according to the field A mapping operation is performed on the first field and the second field in order to determine the field value corresponding to the first field of the piece of data.

Preferably, the acquisition module is specifically used for:

Obtain the metadata information table corresponding to the target data table, where the metadata information table at least contains ordered field metadata information; read the ordered field metadata information in turn to determine the currently read The fields matched by the field metadata information; the determined fields are sequentially written into the determined ordered field list, the ordered field list is used to store the fields, and the order of the fields is consistent with the metadata information table The order of the field metadata information is the same.

An electronic device comprising: at least one memory and at least one processor; the memory stores a program, the processor calls the program stored in the memory, and the program is used to implement the one Data self-cleaning method.

A storage medium storing computer-executable instructions, the computer-executable instructions being used to execute the data self-cleaning method.

Compared with the prior art, the beneficial effects realized by the present invention are:

The present invention provides a data self-cleaning method, device, electronic device and storage medium. For the target data table and target data to be processed, the field information of the target data table can be obtained, because the field information can represent the data in the target data table. The first field and the field order between them, so after performing a parsing operation on each piece of data in the target data to determine the second field and its corresponding field value, the first field and the second field can be sorted according to the field order. A mapping operation is performed to determine the field value corresponding to each piece of data in the first field. Based on the present invention, the fields of unstructured data can be automatically kept in the same order as the fields in the data table, so as to ensure the accuracy of the data. Even if the fields of the subsequent data table are extended, the extended fields can be determined in real time through the field information, and continue to communicate with The fields in the data table are consistent, and self-adaptive cleaning is achieved, which greatly reduces the workload of manual verification and improves efficiency.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

Fig. 1 is a method flowchart of a data self-cleaning method provided by an embodiment of the present invention;

2 is a partial method flowchart of a data self-cleaning method provided by an embodiment of the present invention;

Fig. 3 is another part of the method flowchart of the data self-cleaning method provided by the embodiment of the present invention;

Fig. 4 is another part of the method flowchart of the data self-cleaning method provided by the embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a data self-cleaning device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In order to make the above objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

An embodiment of the present invention provides a data self-cleaning method. The method flowchart of the method is shown in FIG. 1 , and includes the following steps:

S10: Determine the target data table to be processed and the target data, where the target data is unstructured data corresponding to the target data table and contains at least one piece of data.

In the embodiment of the present invention, the target data table is the data table to be processed in the hive data warehouse, that is, the hive table to be processed. The target data is unstructured data to be processed corresponding to the target data table, which may be business data or log data, which is not limited in this embodiment of the present invention.

In addition, the target data contains one or more pieces of data, and one piece of data in the target data corresponds to a data record in the target data table, that is, each piece of data in the target data after data cleaning has a corresponding piece of data in the target data table. The data record contains the field value of the corresponding piece of data under each field of the target data table, and the corresponding fields between any two data records in the target data table are the same.

It should be noted that the data self-cleaning solution in the embodiment of the present invention can be applied to hive UDF. hiveUDF, that is, the above-mentioned UDF (User Defined Function, user-defined function), which is a user-defined hive function. When the functions that come with hive cannot fully meet the business requirements, user-defined functions are required to handle the business requirements. Similar to plug-ins, some processing methods can be customized in the hive data warehouse based on the business logic that needs to be processed.

S20: Acquire field information corresponding to the target data table, where the field information can represent the first field in the target data table and the field order among the first fields.

In this embodiment of the present invention, for the target data table, the corresponding field information can be obtained, and the field information is used to represent the fields (ie, the first fields) in the target data table and the field order among the first fields.

For example, the target data table is a user data table, and the user data table contains three fields of "name", "age", and "gender", and according to the user's reading order of the user data table, such as three fields from left to right The fields are "name", "age", and "gender" in sequence, then the field order among the three fields in the user data table is "name→age→gender".

Therefore, by acquiring the field information of the user data table, on the one hand, it can be determined that the user data table contains three fields of "name", "age" and "gender", and on the other hand, the "name" and "age" can also be determined. , the order between the three fields of "gender", namely "name→age→gender".

In some embodiments, the field information of the data table can be obtained by automatically reading hive metadata information. In the specific implementation process, step S20 "obtaining the field information corresponding to the target data table" may adopt the following steps, and the method flowchart is shown in Figure 2:

S201: Obtain a metadata information table corresponding to the target data table, where the metadata information table at least contains ordered field metadata information.

In the embodiment of the present invention, when the target data table is constructed by DDL, its field information will be stored in its corresponding metadata information table. Part of the field information (ie, field metadata information) corresponding to the input field is also ordered.

It should be noted that DDL (data definition language, data definition language), that is, the definition language for creating hive tables, DDL is mainly used to define or change the structure, data types, links and constraints between tables, etc. In initialization work, it is mostly used when creating tables. Continue to use the user data table to illustrate, in which the three fields of "name", "age" and "gender" are identified by "name", "age" and "sex" respectively, and the "name" corresponding to "name" is identified. " field is a character type (ie "string"), the "age" field corresponding to the identification "age" is a numeric type (ie "number"), and the "gender" field corresponding to the identification "sex" is a character type (ie " string"), the corresponding DDL is "create tableuser(name string, age number, sex string)".

Continue to take the DDL as "create table user(name string, age number, sex string)" as an example, through which a user data table can be created, and the three fields of "name", "age" and "gender" are respectively The metadata information of the fields is sequentially stored in the metadata information table corresponding to the user data table according to the field order "name→age→gender". Specifically, as shown in the following table, first, the identifier "name" and the corresponding character type "string" are stored in the metadata information table, and then the identifier "age" and the corresponding digital type "number" are stored, and finally the identifier " sex" and the corresponding character type "string" are stored. That is to say, the metadata information table corresponding to the user data table contains the field metadata information of three fields in order, which are the identifier "name" corresponding to the "name" field, and the character type "string" and "age" fields. The corresponding identifier "age" and the numeric type "number", and the identifier "sex" corresponding to the "gender" field and the character type "string".

It should be noted that, in the following table, "table" indicates a data table, "field" indicates a field, and "type" indicates a data type.

It should also be noted that the field metadata information corresponding to each field in the data table in the above example only includes the data type. Correspondingly, the metadata information table also only includes the field identifier and the field data type. It is understandable that , this is just an example. In practical applications, the metadata information table stores the data of the attributes of the data table, including but not limited to the identifier of the field, the data type of the field, and the storage location corresponding to the data table.

S202: Read the ordered field metadata information in sequence to determine a field matched by the currently read field metadata information.

In this embodiment of the present invention, since the metadata information table corresponding to the target data table contains at least ordered field metadata information, that is, the metadata information includes field metadata information of one field, or ordered multiple fields field metadata information. Therefore, the main function of reading the metadata information table is to read the field metadata information of the target data table. Based on the relevant information (such as the identifier) of the target data table, the metadata corresponding to the target data table can be read in the UDF. The field metadata information of each field is read from the data information table.

Moreover, since the field metadata information stored in the metadata information table is in order when the target data table is constructed by DDL, the field metadata information read from the metadata information table is also in order. . In this regard, for the currently read field metadata information, the matched fields can be determined. Continue to explain with the user data table, the field metadata information of the three fields of "name", "age", and "gender" has been stored in the metadata information table in turn. Therefore, for the currently read field metadata information The corresponding field can be determined, for example, the identifier "name" and the character type "string" are currently read, and thus the matched field can be determined as "name".

S203, the determined fields are sequentially written into the determined ordered field list, where the ordered field list is used to store the fields, and the order of the fields is the same as the order of the field metadata information in the metadata information table.

In this embodiment of the present invention, for the currently read field metadata information, after the matching field is determined, the field may be stored in an ordered field list, and the order of the fields in the ordered field list is related to the metadata information table. The order of the metadata information of the fields in the Hive is the same, which can keep the unstructured data fields in the same order as the fields in the data table by automatically reading the hive metadata information. For the subsequent data table extension fields, there is no need to add fields. Check, can be adaptive and improve efficiency.

In some other embodiments, the ordered field list may be further output to inform the UDF of the output order of each field, which is a necessary step in the hive UDF to give the ordered field list to the UDF's field output method.

S30, for each piece of data in the target data, perform a parsing operation on the piece of data to determine a second field in the piece of data and a field value corresponding to the second field; analyze the first field and the second field according to the field order A mapping operation is performed to determine the field value corresponding to the first field of the piece of data.

In this embodiment of the present invention, the target data includes at least one piece of data, and one piece of data corresponds to a data record in the target data table. Take the following data as an example in the target data:

"http://user.info?name=lucky&age=30&sex=female" is a piece of data in the target data, and performing a parsing operation on the piece of data can determine the field (ie, the second field) and the second field in the piece of data. The field value corresponding to the field, that is to say, the data contains three fields "name", "age", "gender" and the field values corresponding to these three fields, that is, the "lucky" corresponding to the "name" is identified. ", "30" corresponding to the logo "age", and "female" corresponding to the logo "sex".

Therefore, the second field in the piece of data can be mapped with the first field in the target data table to determine a data record corresponding to the piece of data in the target data table, that is, the field value of the "name" field is A data record with the field value of "lucky", "age" field being "30", and the field value of "gender" field being "female".

It should be noted that when performing the parsing operation on each piece of data in the target data, the storage form of the piece of data may be considered to select a corresponding parsing method. For example, log data is mostly stored in text format, which mainly adopts the form of k1=v1&k2=v2 or the storage form of json, and the log data is parsed through the parsing methods corresponding to different storage forms.

In the specific implementation process, in step S30, the following steps may be adopted for "performing the analysis operation on this piece of data", and the method flowchart is shown in Figure 3:

S301 , extract the key-value pair in the piece of data, where the key in the key-value pair represents the field, and the value in the key-value pair represents the field value.

In this embodiment of the present invention, the unstructured data parsing is mainly to parse log data, and a piece of data can be parsed into the form of k and v, and a set of k and v is a key-value pair. For example, "http://user. info?name=lucky&age=30&sex=female" contains three key-value pairs, namely "k=name,v=lucky", "k=age,v=30", "k=sex,v=female" ".

S302, establishing a correspondence between fields and field values in the key-value pair.

In this embodiment of the present invention, each key-value pair extracted from a piece of data may be converted into a map structure to establish a corresponding relationship between fields and field values in the key-value pair.

Further, since the second field obtained through the above parsing operation and its corresponding field value (or map structure) are out of order, the number of second fields in it is often more than the number of first fields in the target data table, The first field is the field required by the data table, which requires extracting the field value of the first field from the disordered second field and its corresponding field value based on the first field. In this regard, in step S30, the following steps can be adopted for "performing the mapping operation on the first field and the second field according to the field order", and the method flowchart is shown in Figure 4:

S303: Determine a target field matching the first field in the second field according to the field order.

In the embodiment of the present invention, according to the field order among the first fields in the target data table, the first field currently to be processed is sequentially determined, and then the second field obtained by parsing a piece of data is determined to match the first field field, the target field.

Continue to take the target data table as the user data table for illustration, assuming that the field order among the three fields of "name", "age" and "gender" in the user data table is "name→age→gender". According to the order of the fields, the three fields of "name", "age" and "gender" match the target field. The following description takes the first field as the "Name" field:

Suppose, a piece of data is "http://user.info?name=lucky&age=30&sex=female&occupation=farmer", after analysis, it can be determined that the piece of data contains "name", "age", "sex", "occupation" Four fields and their corresponding field values, namely "lucky" corresponding to the identifier "name", "30" corresponding to the identifier "age", "female" corresponding to the identifier "sex", and identifier "occupation" The corresponding "farmer", obviously, the number of the second field in this piece of data is more than the number of the first field in the user data table. Obviously, by matching the identifier of the first field with the identifiers of the four second fields, the second field that matches the first field can be determined, that is, the "name" field of the first field and the "name" field of the second field can be determined fields match.

S304, determine the field value corresponding to the target field, and write the determined field value into the determined ordered data list in turn, the ordered data list is used to store the field value, and the order of the field value and the field order same.

In the embodiment of the present invention, the target data table is continued to be used as the user data table for description. Continue to explain with the first field as the "name" field:

After it is determined that the "name" field of the second field with which it matches is used as the target field, the field value of the "name" field in the target data can be written into the ordered data list. Therefore, according to the field order of "name→age→gender" in the target data table, the "lucky", "30" and "female" are written into the ordered data list. Obviously, the order of the data in the ordered data list is consistent with the field order (or the field order in the ordered field list). Moreover, the data extracted from a piece of data is also obtained according to the field order, which not only ensures the accuracy of the obtained data, but also ensures that the output data is completely consistent with the field order of the data table, and the output will not be misplaced.

In some other embodiments, the ordered data list can be further output to inform the UDF of the output order of each field value, which is a necessary step in the hive UDF, and the ordered data list is given to the UDF processing method, Do forward output. It should be noted that forward is the output function specified by hive UDF. To use UDF, this function must be used according to the regulations of hive UDF, otherwise hive UDF cannot be used.

Based on this, after the ordered field list and the ordered data list are given to the UDF, the UDF can be registered to the hive platform, so that the hive platform can call the UDF for cleaning and write data to the data table.

Therefore, in the data self-cleaning method provided by the embodiment of the present invention, the metadata information of the data table is read through the hive UDF, and the fields of the unstructured data and the fields of the data table can be automatically kept in the same order to ensure the accuracy of the data. Moreover, after the subsequent expansion of the fields, the expanded fields can be read in real time, and the order of the fields in the data table is guaranteed to be consistent, so as to achieve self-adaptation, which greatly reduces the workload of manual verification and improves the efficiency.

Based on the data self-cleaning method provided by the above-mentioned embodiments, an embodiment of the present invention may also provide a device for executing the data self-cleaning method. The schematic structural diagram of the device is shown in FIG. 5 , including:

A determination module 10, configured to determine the target data table to be processed and the target data, and the target data is unstructured data corresponding to the target data table and contains at least one piece of data;

The obtaining module 20 is used for obtaining the field information corresponding to the target data table, and the field information can represent the first field in the target data table and the field order between the first fields;

The cleaning module 30 is configured to perform a parsing operation on each piece of data in the target data to determine the second field in the piece of data and the field value corresponding to the second field; Perform a mapping operation with the second field to determine the field value corresponding to the first field of the piece of data.

Optionally, obtain the module 10, which is specifically used for:

Obtain the metadata information table corresponding to the target data table, the metadata information table contains at least ordered field metadata information; read the ordered field metadata information in turn to determine the matching field metadata information currently read The determined fields are sequentially written into the determined ordered field list, the ordered field list is used to store the fields, and the order of the fields is the same as the order of the field metadata information in the metadata information table.

Optionally, the acquisition module 10 is also used for:

Output an ordered list of fields.

Optionally, the cleaning module 30 for performing the parsing operation on the piece of data is specifically used for:

Extract the key-value pair in the piece of data, the key in the key-value pair represents the field, and the value in the key-value pair represents the field value; establish the corresponding relationship between the field and the field value in the key-value pair.

Optionally, the cleaning module 30 for performing a mapping operation on the first field and the second field according to the field order is specifically used for:

Determine the target field that matches the first field in the second field according to the field order; determine the field value corresponding to the target field, and write the determined field value into the determined ordered data list in turn, orderly A data list is used to store field values, and the order of the field values is the same as the order of the fields.

Optionally, the cleaning module 30 is also used for:

Output an ordered list of data.

It should be noted that, for the refined functions of each module in the embodiments of the present invention, reference may be made to the corresponding disclosure parts of the foregoing method embodiments, and details are not described herein again.

Based on the data self-cleaning method provided by the above embodiment, the embodiment of the present invention further provides an electronic device, the electronic device includes: at least one memory and at least one processor; the memory stores a program, the processor calls the program stored in the memory, and the program uses For realizing data self-cleaning method.

Based on the data self-cleaning method provided by the foregoing embodiments, an embodiment of the present invention further provides a storage medium, where computer-executable instructions are stored in the storage medium, and the computer-executable instructions are used to execute the data self-cleaning method.

A data self-cleaning method, device, electronic device and storage medium provided by the present invention are described in detail above. Specific examples are used in this paper to illustrate the principles and implementations of the present invention. In order to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. In summary, this specification The contents should not be construed as limiting the present invention.

It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same and similar parts among the various embodiments, refer to each other Can. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

It should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply those entities or operations There is no such actual relationship or order between them. Furthermore, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article, or device of a list of elements is included, inherent to, or is also included for, those processes. , method, article or device inherent elements. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. a data self-cleaning method, is characterized in that, described method comprises: Determine the target data table and target data to be processed, and the target data is unstructured data corresponding to the target data table and contains at least one piece of data; acquiring field information corresponding to the target data table, where the field information can represent the first field in the target data table and the field order between the first fields; For each piece of data in the target data, a parsing operation is performed on the piece of data to determine a second field in the piece of data and a field value corresponding to the second field; A mapping operation is performed between a field and the second field to determine the field value corresponding to the first field of the piece of data. 2. The method according to claim 1, wherein the acquiring the field information corresponding to the target data table comprises: obtaining a metadata information table corresponding to the target data table, where the metadata information table at least contains ordered field metadata information; Read the ordered field metadata information in turn to determine the field matched by the currently read field metadata information; The determined fields are sequentially written into the determined ordered field list, where the ordered field list is used to store the fields, and the order of the fields is the same as the order of the field metadata information in the metadata information table. 3. The method according to claim 2, wherein the method further comprises: The ordered field list is output. 4. The method according to claim 1, wherein the performing a parsing operation on the piece of data comprises: Extract the key-value pair in the piece of data, the key in the key-value pair represents the field, and the value in the key-value pair represents the field value; A corresponding relationship between fields and field values in the key-value pair is established. 5. The method according to claim 1, wherein the performing a mapping operation on the first field and the second field according to the field order comprises: determining a target field matching the first field in the second field according to the field order; Determine the field value corresponding to the target field, and write the determined field value into the determined ordered data list in turn, and the ordered data list is used to store the field value, and the order of the field value is the same as the order of the field value. The fields are in the same order. 6. The method according to claim 5, wherein the method further comprises: The ordered list of data is output. 7. A data self-cleaning device, wherein the device comprises: a determination module, configured to determine a target data table to be processed and target data, where the target data is unstructured data corresponding to the target data table and contains at least one piece of data; an acquisition module, configured to acquire field information corresponding to the target data table, where the field information can represent a first field in the target data table and a field order between the first fields; a cleaning module, configured to perform a parsing operation on each piece of data in the target data to determine a second field in the piece of data and a field value corresponding to the second field; according to the field A mapping operation is performed on the first field and the second field in order to determine the field value corresponding to the first field of the piece of data. 8. The device according to claim 7, wherein the acquisition module is specifically used for: Obtain the metadata information table corresponding to the target data table, where the metadata information table contains at least ordered field metadata information; read the ordered field metadata information in turn to determine the currently read The fields matched by the field metadata information; the determined fields are sequentially written into the determined ordered field list, the ordered field list is used to store the fields, and the order of the fields is consistent with the metadata information table The order of the field metadata information is the same. 9. An electronic device, characterized in that the electronic device comprises: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, and the program is used for A data self-cleaning method according to any one of claims 1-6 is realized. 10. A storage medium, wherein the storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the data self-cleaning method according to any one of claims 1-6.

Images