CN108280226B - Data processing method and related equipment - Google Patents

Abstract

The application provides a data processing method, which comprises the steps of determining a primary table and a secondary table related to a target processing service after the target processing service is determined, extracting data records in the secondary table into a file, sequencing the data records in the file according to a key value field of the primary table, and storing the sequenced data records into a temporary data table, wherein the data records in the temporary data table are sequenced in the same mode as the data records in the primary data table, namely, the data records are sequenced according to the key value field of the primary table. The pre-reading cache mechanism of the database can read data records in the main data table to the memory according to the key value field of the main table, and can also read data records in the same key value field in the temporary data table to the memory, so that the probability of simultaneously accessing the data records in the two data tables in the memory is improved, and the data access efficiency is improved. In addition, the application also provides a data processing device used for ensuring the application and the realization of the method in practice.

Description

Data processing method and related equipment

Technical Field

The present application relates to the field of database technologies, and more particularly, to a data processing method and related device.

Background

A database, a repository built on a computer storage device, is used to organize, store, and manage data according to a data structure. One very important object of database management is a data table, which is stored in a non-volatile storage medium such as a magnetic disk. The data table includes a plurality of data records, and when a service processing program (also referred to as a main program) needs to read the data records in the data table, a read instruction is sent to the database, and the database can read the data records in the nonvolatile storage medium into the memory and process the data records in the memory.

The data required by the business processing program may be stored in fields of a plurality of data tables, wherein the data table containing the fields most relevant to the process processed by the business processing program is called a primary table, and other data tables are called secondary tables. Under the condition of distinguishing the primary table from the secondary table, when a service processing program processes data records in the memory, the primary table is used as the main table, that is, the data records are sequentially processed according to the storage sequence of the data records of the primary table in the memory, and when one data record is read, other fields related to the data record need to be read in the secondary table. However, if other fields in the secondary table are not saved in the memory, the database needs to read these fields from the disk into the memory. Therefore, the current data processing efficiency needs to access the disk for multiple times, and the processing efficiency is low.

Disclosure of Invention

In view of this, the present application provides a data processing method, which is used to solve the technical problem of low data processing efficiency caused by the data asynchronism of the primary table and the secondary table in the data processing process.

In order to achieve the purpose, the technical scheme provided by the application is as follows:

in a first aspect, the present application provides a data processing method, including:

determining a target processing service;

determining a main data table associated with the target processing service, and determining a key value field used for indicating data record sequencing in the main data table;

determining a secondary data table associated with the target processing service, and extracting data records in the secondary data table into a file; wherein the data record comprises a key value field in a main data table;

sorting the data records in the file according to the key value field;

and storing the sorted data records into a target data table.

In a second aspect, the present application provides a data processing apparatus comprising:

the processing service determining module is used for determining a target processing service;

a primary table key value determining module, configured to determine a primary data table associated with the target processing service, and determine a key value field in the primary data table, where the key value field is used to indicate data record sorting;

the secondary data extraction module is used for determining a secondary data table associated with the target processing service and extracting data records in the secondary data table into a file; wherein the data record comprises a key value field in a main data table;

the secondary table data sorting module is used for sorting the data records in the file according to the key value field;

and the secondary table data storage module is used for storing the sorted data records into a target data table.

The application provides a data processing method, which comprises the steps of determining a primary table and a secondary table related to a target processing service after the target processing service is determined, extracting data records in the secondary table into a file, sequencing the data records in the file according to a key value field of the primary table, and storing the sequenced data records into a temporary data table, wherein the data records in the temporary data table are sequenced in the same mode as the data records in the primary data table, namely, the data records are sequenced according to the key value field of the primary table. The pre-reading cache mechanism of the database can read data records in the main table to the memory according to the key value fields of the main table, and can also read data records in the same key value fields in the temporary data table to the memory, so that the probability of simultaneously accessing the data records in the two data tables in the memory is improved, and the data access efficiency is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a flow chart of a data processing method provided herein;

FIG. 2 is a flow chart of a data processing method provided herein;

FIG. 3 is a schematic diagram of Cartesian product processing of data records in a document as provided herein;

FIG. 4 is a schematic diagram of the sorting of data records in a file provided by the present application;

FIG. 5 is a schematic diagram of a data processing apparatus provided in the present application;

fig. 6 is a schematic structural diagram of another data processing apparatus provided in the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

An important processing object in the database is a data table, and the data table is stored in a nonvolatile storage medium such as a magnetic disk. The service processing program can process the service, and the data record in the data table is needed to be used when the service is processed. When the business processing program needs to use the data records in the data table, an instruction is sent to the database so that the database reads the data records from the nonvolatile storage medium to the memory.

The data required by the service processing program may be stored in a plurality of data tables, one of the data tables is most relevant to the service processing program, the data table may be called a main data table (may be simply called a main table), and the other data tables may be called secondary data tables (may be simply called secondary tables). The secondary data table will contain detailed data related to the data in the primary data table.

It should be noted that, in general, a data table has a plurality of fields, wherein a certain field is used to indicate the sorting of data records, and the field may be referred to as a key value field. The data table may store data in some varying order of values of the key-value field. For example, the data table is a customer information table, the key value field is a customer number, and the data table can store data records sequentially according to the order from small to large of the customer number.

The database has a pre-reading cache mechanism, that is, when a certain data record needs to be read to the memory, a plurality of data records including the previous data record and/or the next data record of the data record can be read simultaneously to the memory, so that if the next data record of the data record needs to be read by the business processing program, the next data record can be directly obtained in the memory without accessing the nonvolatile storage medium, and the data reading efficiency can be improved.

The primary data table and the secondary data table have an association relationship, namely, the secondary data table comprises a key value field in the primary data table. The business processing program performs association processing on the primary data table and the secondary data table, sequentially accesses each data record in the primary data table according to the sequence of the key value fields in the primary data table, and when a certain data record in the primary data table is accessed, the data record corresponding to the key value field needs to be extracted from the secondary data table according to the key value field in the data record.

However, the secondary data table itself also has a key value field, and the key value field of the secondary data table is typically different from the key value field of the primary data table. For example, the main data table is an account information table in a banking system, the account information table mainly includes fields of account numbers, interest rates, interest increment, interest due and the like, wherein the account numbers are key value fields, that is, data records in the account information table are sequentially ordered according to the order of the account numbers from small to large; the secondary data table comprises a customer information table, wherein the customer information table mainly comprises fields such as customer numbers, customer names, customer categories, customer addresses, customer tax codes, account numbers and the like, the customer numbers are key value fields, namely data records in the customer information table are sequentially ordered according to the order of the customer numbers from small to large.

In this case, the read-ahead caching mechanism of the database cannot function effectively.

For example, the service processing program needs to access a certain data record in the main data table, the data record may be referred to as a first data record, the database may read all the first data record and 10 data records after the first data record into the memory, and the service processing program directly obtains the first data record from the memory. After the first data record in the main data table is obtained, the data record associated with the data record in the secondary data table also needs to be obtained, and the data record in the secondary data table can be called as a second data record. The service processing program firstly searches a second data record containing the key value field of the first data record in a secondary data table of the memory, and if the second data record is not found, the database is required to read the second data record from the nonvolatile storage medium to the memory.

It should be noted that, based on the read-ahead caching mechanism, the database may not only read the second data record, but also may record 10 data records after the second data record. However, since the key value fields of the primary data table and the secondary data table are different, the data records of the primary data table and the secondary data table read in the memory in advance may not correspond to each other, that is, the key value field of the data record of the primary data table in the memory is most likely not included in the data record of the secondary data table in the memory. Therefore, when the service processing program reads the next data record according to the key value field in the main data table, the data record containing the key value field of the next data record cannot be found in the secondary data table data record in the memory.

To facilitate understanding of the above technical problems, explanation is made in conjunction with specific examples.

For example, the business processing program is a processing program in a banking system, and the business processing content required to be executed is accumulated account interest increment. Because the accumulated account interest increment is the account interest increment, the account interest corresponding to the account numbers needs to be processed after determining which account numbers. The account number is stored in the account information table, and thus the account information table serves as a main data table. When calculating the account interest increment, other information related to the account interest corresponding to the account needs to be accessed based on the requirement of a calculation mode, and a data table for storing the other information is called a slave table. For example, the other information may include customer information, which is stored in a customer information table, and thus the customer information table is a slave data table.

As illustrated in the above example, the key value field in the account information table is an account number, and the business processing program needs to traverse each account number in the account information table from the beginning. Assuming that the service processing program needs to read the data record of the account a1 at present, and the data record does not exist in the memory, the database reads the account a1 from the disk, and reads 5 data records from the account a1 to the account a5 together into the memory. Assuming that the account information table includes 100 pieces of account information as shown in table 1 below, the account information read into the memory is shown in table 2.

TABLE 1

Account number	Interest rate	Increment of interest
			a1	a11	a12
a2	a21	a22
			a3	a31	a32
……	……	……
			a100	a1001	a1002

TABLE 2

Account number	Interest rate	Increment of interest
			a1	a11	a12
a2	a21	a22
			a3	a31	a32
a4	a41	a42
			a5	a51	a52

In this way, the service processing program can directly read the data record with the account a1 from the memory, and needs to read the data record containing the account a1 from the customer information table, where the data record contains the relevant information of the customer corresponding to the account a 1. If the key value field in the customer information table is the customer number, the data records in the customer information table are sorted according to the arrangement sequence of the customer numbers. It is assumed that 200 pieces of customer information are included in the customer information table as shown in table 3 below, and the 200 pieces of customer information are arranged in the order of customer numbers from small to large.

TABLE 3

Customer number	Interest rate	Increment of interest	Account number
				b1	b11	b12	a27
b2	b21	b22	a1
				b3	b31	b32	a73
b4	b41	b42	a6
				b5	b51	b52	a23
……	……	……	……
				b200	b2001	b2002	a55

The database needs to search for the data record with account a1, and assuming that the database will read 5 data records in advance, the data records read into the memory are shown in table 4 below. As can be seen, the customer information read in advance into the memory includes the data record corresponding to the account a1, so the service processing program may read the second data record in table 4 directly from the memory.

TABLE 4

Customer number	Interest rate	Increment of interest	Account number
				b1	b11	b12	a27
b2	b21	b22	a1
				b3	b31	b32	a73
b4	b41	b42	a6
				b5	b51	b52	a23

It should be noted that, the service processing program reads the data record with the account number a2 according to the storage sequence of the account numbers in the account information table, where table 3 in the memory includes the data record but table 4 in the memory does not include the data record with the account number a2, so that table 4 in the memory is not read in advance by the database, and instead, table 4 occupies a part of the memory space, which results in resource waste. The database needs to search for the data record corresponding to the account a2 from the customer information in the disk again, and the data reading efficiency is low.

The main reason why the above technical problem is caused is that the data records in the primary data table and the secondary data table are not sorted according to the same key value field, so the present application provides a data processing method. Referring to fig. 1, a flow of the data processing method provided by the present application is shown, and specifically includes steps S101 to S105.

S101: and determining a target processing service.

The service processed by the service processing program may be directly used as a target processing service, and as described in the above example, the target processing service may be an accumulated account interest increment in a banking system.

S102: a primary data table associated with the target processing service is determined, and a key value field used for indicating the ordering of the data records in the primary data table is determined.

The main data table is associated with the target processing service, and the data table capable of providing basic data required in the target processing service is the main data table. For example, if the target processing service is cumulative account interest increment, it is necessary to determine how many account numbers are present, and then sequentially accumulate the interest increments corresponding to the account numbers, so that the account information table for recording account information is used as a main data table. For another example, if the target processing service is to calculate an average score of chemical scores of a class, it is first necessary to determine how many students are included in the class, and then average the chemical scores of the students after summing up, so that the class information table in which the student numbers are recorded is used as the main data table.

It should be noted that the master data table is determined according to the processing logic of the target processing service, and a certain data table is the master data table in a certain target processing service, but may be the secondary data table in another target processing service.

As mentioned above, a key field is set in each data table, and the key field can provide a basis for sorting the data records. For example, the account information table contains account numbers, which are unique identifiers and can be sorted in order of size, the account information table can use the field of the account numbers as a key value field, and the data records in the account information table are sorted in a certain order of the account numbers. Wherein the key value field can be a primary key in the data table or other fields.

S103: and determining a secondary data table associated with the target processing service, and extracting data records in the secondary data table into a file.

In the application scenario of the present application, a plurality of data tables are involved, and a secondary data table is included in addition to a primary data table. The secondary data table is a data table which contains fields required by the target processing service and has an association relation with the primary data table. The association relationship between the primary data table and the secondary data table is embodied in that the secondary data table is directly or indirectly associated with a key value field of the primary data table.

For example, if the target processing service is cumulative account interest increment, fields such as interest increment and interest rate need to be acquired, and these fields are included in the customer information table, and the customer information table includes key value field account numbers in the account information table.

It should be noted that there may be a plurality of secondary data tables, for example, after an account is found in the primary data table by a certain target processing service, a customer number corresponding to each account needs to be further determined, and if the customer number is stored in the customer information table, the customer information table is one of the secondary data tables, and if the target processing service also needs to further determine an asset condition of a customer, and if the asset condition is stored in the asset information table, the asset information table is another secondary data table.

In the case where there are a plurality of secondary data tables, the association relationship between the secondary data table and the primary data table may be expressed as two types: one is that some secondary data tables can directly contain key value fields in the primary data table, and the customer information contains account numbers in the account information table; some of the secondary data tables may include fields in other secondary data tables, which have a relationship with the key value field in the primary data table indirectly or directly, such as the above-mentioned property information, which may not include the field of the account number, but includes the customer number in the customer information table, and the customer information table includes the key value field of the primary data table, i.e. the account number.

When there are a plurality of secondary data tables, all the secondary data tables do not necessarily include the key value field in the primary data table, and at least one secondary data table may include the key value field.

The data records are extracted in a manner that each data record is extracted into a file according to the key value field of the secondary data table. For example, if the secondary data table is a client information table, all data records in the client information table are extracted into a file according to the client number.

The secondary data table may contain a plurality of fields, some of which may not be related to the target processing service, for example, the target processing service is a cumulative account interest increment, and the customer information table contains fields related to the service, such as an interest increment, and fields unrelated to the service, such as a customer address, a customer name, and the like. Therefore, when extracting the data record, the fields required for the target processing service may be first determined among the fields contained in the secondary data table, and for convenience of description, these fields may be referred to as service fields. And then extracting the data records containing the service fields in the secondary data table into a file. Therefore, the reading amount of data can be reduced, and the service processing efficiency is improved.

It should be noted that, in addition to extracting the service field, fields having an association relationship with the key-value field of the primary data table must be extracted from each secondary data table, and for convenience of description, these fields may be referred to as association fields. For example, the secondary data table includes a customer information table, and the associated field is an account number, because the account number is a key value field of the primary data table. If the secondary data table further includes an asset information table on the basis of the client information table, the associated field is the client number, because the client number is a common field between the asset information table and the client information table.

It should be noted that, the purpose of extracting the association field is that the association field includes a key value field of the main data table, then step S104 may be executed to sort the data records in the file according to the key value field of the main data table. In addition, the reason for extracting data into a file is that writing data into a file is more efficient than writing data into a data table.

S104: data records in the file are sorted according to the key value field.

Based on the above description, it can be known that the data records in the file necessarily include the key value field of the main data table, and the data records in the file can be sorted according to the key value field of the main data table. For example, the data records in the file include { (b1, b11, b12, a 2); (b2, b21, b22, a 1); (b3, b31, b32, a3) }, wherein a1, a2 and a3 are key value fields, and the result after sorting according to the key value fields of the main data table is { (b2, b21, b22 and a 1); (b1, b11, b12, a 2); (b3, b31, b32, a3) }, or the key value field of the main data table may be further turned to the head, i.e., { (a1, b2, b21, b 22); (a2, b1, b11, b 12); (a3, b3, b31, b 32).

S105: and storing the sorted data records into a target data table.

The data records sorted in the file can be stored in the data table because the reading efficiency of the data table is high, so that the target processing service can be conveniently executed by matching the data records in the main data table. For convenience of description, the data table may be referred to as a target data table or a temporary data table.

According to the technical scheme, after the target processing service is determined, the main data table and the secondary data table related to the target processing service are determined, the data records in the secondary data table are extracted into the file, the data records in the file are sorted according to the key value field of the main data table, and the sorted data records are stored in the temporary data table, so that the data records in the temporary data table are sorted in the same mode as the data records in the main data table, namely, the data records are sorted according to the key value field of the main data table. The pre-reading cache mechanism of the database can read data records in the main data table to the memory according to the key value field of the main data table, and can also read data records in the same key value field in the temporary data table to the memory, so that the probability of simultaneously accessing the data records in the two data tables in the memory is improved, and the data access efficiency is improved.

As mentioned above, the secondary data table may be plural, in which case the secondary data table may be associated, so that the present application provides another flow of the data processing method. As shown in fig. 2, the flow of the data processing method specifically includes the following steps S201 to S206.

S201: and determining a target processing service.

S202: a primary data table associated with the target processing service is determined, and a key value field used for indicating the ordering of the data records in the primary data table is determined.

S203: determining a secondary data table associated with the target processing service, and extracting data records in the secondary data table into a file; wherein the data record contains a key value field in the main data table.

If there are multiple secondary data tables, each secondary data table will obtain one file, so that multiple files can be obtained.

S204: and if the number of the secondary data tables is multiple, performing association operation on the data records in the files corresponding to the multiple secondary data tables, and storing the data records after the association operation in the target file.

Each secondary data table is corresponding to obtain a file, and the data records in the files are subjected to correlation operation such as Cartesian product operation.

For example, the file obtained from the secondary data table 1 includes three data records: { (b1, b11, b12, a 2); (b2, b21, b22, a 1); (b3, b31, b32, a3) }, the file obtained from the time data table 2 contains two data records: { (c1, b 2); (c2, b 1). As shown in fig. 3, after the two data records are subjected to the cartesian product operation, the obtained data records are: { (c2, b1, b11, b12, a 2); (c1, b2, b21, b22, a1) }.

S205: and sorting the data records in the target file according to the key value field.

The key value fields in the data records in the target file can be firstly sorted, transposed to the head of the data records, and then sorted according to the key value fields. As shown in fig. 4, the key value fields a2 and a1 in the two data records obtained in fig. 3 are first transposed to the first field, and then the two data records are sorted according to the order from a1 to a2, so as to obtain the final target file.

S206: and storing the sorted data records into a target data table.

It should be noted that, for the description of steps S201 to S203 and S205 to S206, reference may be made to the description in fig. 1, and details are not repeated here.

According to the technical scheme, the data processing method provided by the application can perform association operation on the data records obtained from the plurality of secondary data tables, and store the data records obtained by the association operation in the temporary data table.

In practical applications, after extracting the data records in the secondary data table into a file, the plurality of fields may be preprocessed according to the requirements of the target processing service. Specifically, if the target processing service requires processing results of a plurality of fields in the secondary data table, the data records corresponding to the plurality of fields are subjected to comprehensive processing to obtain a target field and a data record corresponding to the target field, and the target field and the data record corresponding to the target field are stored in a file.

For example, the target processing service is the average score of the computational physiological ensemble, the secondary data table is a score information table, wherein the score information table comprises scores of biological, physical and chemical courses, the data records extracted from the secondary data table into the file also comprise the three fields, and as the target processing service is the average score of the computational ensemble, the total score of each computational physiological ensemble can be calculated first, and then the average scores of all the computational physiological ensembles can be calculated. Therefore, biological, physical and chemical performances in the file can be summed to obtain the target field of the total score, and the target field is saved in the file.

The temporary data table with the data records sorted according to the key value field of the main data table can be obtained, and the completion of the target processing service needs to depend on the main data table and the temporary data table. Therefore, on the basis of the above embodiments, the following steps may be further included:

after receiving a processing instruction of a target processing service, reading data records in the main data table and data records in the target data table into a memory according to the key value field; and processing the data records in the main data table and the data records in the target data table according to the processing mode corresponding to the target processing service.

Specifically, after the processing, a temporary data table (i.e., a target data table) may be obtained, and after a processing instruction of a target service is received, after a data record in the main data table is read to the memory according to the key value field of the main data table, the data record in the temporary data table may also be read to the memory according to the key value field of the main data table. And then, directly obtaining the required data records in the memory, and processing the data records according to the processing mode corresponding to the target processing service. For example, the target processing service is to calculate the average score of the comprehensive scores, the comprehensive scores of all students can be obtained, and then the sum average calculation is performed. The processing method corresponding to different processing services is different, and is not limited herein.

Compared with the mode of directly accessing the data records in the main data table and the secondary data table in the prior art, the method can find the data required by the main data table and the secondary data table in the cache data of the memory more possibly, thereby improving the service processing efficiency.

Referring to fig. 5, a schematic diagram of a data processing apparatus according to the present application is shown. As shown in fig. 5, the structure of the apparatus specifically includes: the system comprises a processing service determining module 501, a primary table key value determining module 502, a secondary table data extracting module 503, a secondary table data sorting module 504 and a secondary table data saving module 505.

A processing service determining module 501, configured to determine a target processing service;

a primary table key value determining module 502, configured to determine a primary data table associated with the target processing service, and determine a key value field in the primary data table, where the key value field is used to indicate data record sorting;

a secondary data extraction module 503, configured to determine a secondary data table associated with the target processing service, and extract data records in the secondary data table into a file; wherein the data record comprises a key value field in a main data table;

a secondary table data sorting module 504, configured to sort the data records in the file according to the key value field;

and a secondary table data saving module 505, configured to store the sorted data records into a target data table.

Fig. 6 shows a schematic structure of another data processing apparatus provided in the present application. The device can also include, on the basis of the device structure shown in fig. 5: data record association module 506.

A data record association module 506, configured to, after the data records in the secondary data table are extracted into a file and before the data records in the file are sorted according to the key value field, perform association operation on the data records in the file corresponding to multiple secondary data tables if there are multiple secondary data tables, and store the data records after the association operation in a target file;

correspondingly, the secondary table data sorting module 504 may specifically include: a secondary table data sorting sub-module 507. And the secondary table data sorting submodule is used for sorting the data records in the target file according to the key value field.

In one implementation, the secondary table data extracting module 504 may specifically include: a sub-module for determining the secondary table and a sub-module for extracting the field.

A secondary table determining submodule for determining a secondary data table associated with the target processing service;

a field extraction submodule, configured to determine, in fields included in the secondary data table, a service field required by the target processing service and an association field having an association relationship with the key value field; and extracting the data records containing the associated fields and the service fields in the secondary data table into a file.

In one implementation, the data processing apparatus may further include: and an intermediate processing module.

And the intermediate processing module is used for performing comprehensive processing on the data records corresponding to the fields to obtain a target field and the data record corresponding to the target field and storing the target field and the data record corresponding to the target field in a file after the data records in the secondary data table are extracted into the file and before the data records in the file are sorted according to the key value field, if the target processing service needs the processing result of the fields in the secondary data table.

In one implementation, the data processing apparatus may further include: and a service processing module.

The service processing module is used for reading the data records in the main data table and the data records in the target data table into the memory according to the key value field after receiving a processing instruction of a target processing service; and processing the data records in the main data table and the data records in the target data table according to the processing mode corresponding to the target processing service.

It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other.

It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the same element.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (8)

1. A data processing method, comprising:

determining a target processing service;

determining a main data table associated with the target processing service, and determining a key value field used for indicating data record sequencing in the main data table;

determining a secondary data table associated with the target processing service, and extracting data records in the secondary data table into a file; wherein the data record comprises a key value field in a main data table;

sorting the data records in the file according to the key value field;

storing the sorted data records into a target data table;

after receiving a processing instruction of a target processing service, reading data records in the main data table and data records in the target data table into a memory according to the key value field;

and processing the data records in the main data table and the data records in the target data table according to the processing mode corresponding to the target processing service.

2. The data processing method according to claim 1, wherein after the extracting the data records in the secondary data table into a file and before the sorting the data records in the file according to the key value field, further comprising:

if the number of the secondary data tables is multiple, performing association operation on data records in files corresponding to the multiple secondary data tables, and storing the data records after the association operation in a target file;

then the sorting the data records in the file according to the key value field includes:

and sorting the data records in the target file according to the key value field.

3. The data processing method of claim 1, wherein extracting the data records in the secondary data table into a file comprises:

determining a service field required by the target processing service and an association field having an association relation with the key value field in fields contained in the secondary data table;

and extracting the data record containing the association field and the service field in the secondary data table into a file.

4. The data processing method of claim 1, wherein after extracting the data records in the secondary data table into a file and before sorting the data records in the file according to the key value field, further comprising:

and if the target processing service needs the processing results of the fields in the secondary data table, performing comprehensive processing on the data records corresponding to the fields to obtain a target field and a data record corresponding to the target field, and storing the target field and the data record corresponding to the target field in a file.

5. A data processing apparatus, comprising:

the processing service determining module is used for determining a target processing service;

a primary table key value determining module, configured to determine a primary data table associated with the target processing service, and determine a key value field in the primary data table, where the key value field is used to indicate data record sorting;

the secondary data extraction module is used for determining a secondary data table associated with the target processing service and extracting data records in the secondary data table into a file; wherein the data record comprises a key value field in a main data table;

the secondary table data sorting module is used for sorting the data records in the file according to the key value field;

the secondary table data storage module is used for storing the sorted data records into a target data table;

the service processing module is used for reading the data records in the main data table and the data records in the target data table into the memory according to the key value field after receiving a processing instruction of a target processing service; and processing the data records in the main data table and the data records in the target data table according to the processing mode corresponding to the target processing service.

6. The data processing apparatus of claim 5, further comprising:

a data record association module, configured to, after the data records in the secondary data table are extracted into a file and before the data records in the file are sorted according to the key value field, perform association operation on the data records in the file corresponding to multiple secondary data tables if there are multiple secondary data tables, and store the data records after the association operation in a target file;

the secondary table data sorting module includes:

and the secondary table data sorting submodule is used for sorting the data records in the target file according to the key value field.

7. The data processing apparatus of claim 5, wherein the secondary table data extraction module comprises:

a secondary table determining submodule for determining a secondary data table associated with the target processing service;

a field extraction submodule, configured to determine, in fields included in the secondary data table, a service field required by the target processing service and an association field having an association relationship with the key value field; and extracting the data records containing the associated fields and the service fields in the secondary data table into a file.

8. The data processing apparatus of claim 5, further comprising:

and the intermediate processing module is used for performing comprehensive processing on the data records corresponding to the fields to obtain a target field and the data record corresponding to the target field and storing the target field and the data record corresponding to the target field in a file after the data records in the secondary data table are extracted into the file and before the data records in the file are sorted according to the key value field, if the target processing service needs the processing result of the fields in the secondary data table.

Images