CN118014098B - Machine learning training data scheduling method and equipment - Google Patents

Abstract

The application provides a machine learning training data scheduling method and equipment, wherein the method comprises the following steps: when a target machine learning algorithm starts training, acquiring a grafting module corresponding to the target machine learning algorithm, and triggering a universal module layer to start through the grafting module; determining a target disk file according to the directed acyclic graph in the universal module layer; forming a machine learning training module subgraph according to the adaptation modules from the target disk file to all the directed paths of the grafting module; and converting the original training data in the target disk file into data required by a target machine learning algorithm based on the machine learning training module subgraph. According to the technical scheme, the training data conversion efficiency of the machine learning algorithm can be improved.

Description

Machine learning training data scheduling method and device

Technical Field

The present application relates to the field of machine learning, and in particular to a method and device for scheduling machine learning training data.

Background Art

With the development of the big data era, mining potentially useful knowledge from massive data has become a common demand in major industries. Machine learning is a subfield of artificial intelligence that has flourished in the context of the big data era. The core problem it solves is to discover knowledge from data and save the learned knowledge into predictive models or explanatory models for subsequent applications. Predictive models and explanatory models are collectively referred to as models. The process of machine learning is generally divided into two stages: the first stage is the learning stage, also known as the training stage, in which data is input and the model is output; the second stage is the application stage (also known as the prediction stage for predictive models), in which the model is applied to new data to obtain the output of the model. The application stage of machine learning is relatively efficient, and generally data can be processed one by one to obtain the corresponding prediction or explanation results for each data. The training stage of machine learning is the performance bottleneck of the overall machine learning process. Generally, all training data must be considered and multiple non-sequential accesses to all data must be performed to obtain a reliable and effective model.

In order to ensure the effectiveness of the training phase, most machine learning algorithms require that all training data be placed in memory for processing. When the entire training data cannot be placed in memory, the machine learning algorithm can manage data that exceeds the memory capacity by using the operating system's disk as virtual memory. This method involves disk access, and the original disk file storing all training data needs to be converted into the data required by the machine learning algorithm. The conversion process requires the computer equipment to have high disk data processing technology. However, the current conversion process for disk data is relatively slow, and the operating system may occasionally crash during the processing process. The training data conversion efficiency of machine learning is low.

Summary of the invention

The present application provides a method and device for scheduling machine learning training data to solve the technical problem of low data conversion efficiency when a computer device performs training data conversion for a machine learning algorithm.

In a first aspect, a method for scheduling machine learning training data is provided, which is applied to a computer device, and the method includes: when a target machine learning algorithm starts training, obtaining a grafting module corresponding to the target machine learning algorithm, and triggering the startup of a general module layer through the grafting module; determining a target disk file according to a directed acyclic graph in the general module layer; constructing a machine learning training module subgraph according to adaptation modules in all directed paths from the target disk file to the grafting module; and converting the original training data in the target disk file into data required by the target machine learning algorithm based on the machine learning training module subgraph.

In a second aspect, a computer device is provided, comprising a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory to implement the method described in the first aspect.

The present application can achieve the following technical effects: the computer device presets a general module layer, and when it detects that the target machine learning algorithm starts training, it obtains the grafting module corresponding to the target machine learning algorithm, and triggers the startup of the general module layer through the grafting module, and then determines the target disk file that the target machine learning algorithm needs to use according to the directed and undirected graph in the general module layer, and forms a machine learning training module subgraph according to the adaptation module of all directed paths from the target disk file to the grafting module, and converts the original training data in the target disk file into the data required by the target machine learning algorithm based on the machine learning training module subgraph. The above method realizes the training data conversion processing through the preset general module layer, ensuring that ordinary computers can also efficiently process the massive training data stored in the disk files, so that the training stage of the machine learning algorithm can be implemented for the massive training data, and improves the training data conversion efficiency of machine learning.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an existing process for converting training data for a machine learning algorithm;

FIG2 is a flow chart of a method for scheduling machine learning training data according to an embodiment of the present application;

FIG3 is an example diagram of a general module layer provided in an embodiment of the present application;

FIG4 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in this application do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

To make it easier to understand this application, the machine learning algorithm is first introduced. Machine learning is a sub-field of artificial intelligence that has flourished in the context of the big data era. Its core is "using algorithms to parse data, learn from it, and then explain or predict new data." In other words, machine learning is to discover knowledge from data and save the learned knowledge in the model for subsequent application. The process of machine learning is generally divided into two stages: the first stage is the learning stage, also known as the training stage, in which data is input and the model is output; the second stage is the application stage (also known as the prediction stage for predictive models), in which the model is applied to new data to obtain the output of the model. The application stage of machine learning is more efficient, and generally data can be processed one by one to obtain the prediction or explanation results corresponding to each data. The training stage of machine learning is the performance bottleneck of the overall process of machine learning. Generally, all training data need to be considered and multiple non-sequential accesses to all data can be used to obtain a reliable and effective model. In order to ensure the effect of the training stage, most machine learning algorithms require that all training data be placed in memory for processing. Therefore, machine learning algorithms have a large demand for memory.

There are two common solutions to the problem of excessive memory requirements of the above-mentioned machine learning algorithms. One is to upgrade the hardware by adding memory bars and setting larger available memory to ensure efficient implementation of the machine learning algorithm training phase; the other is to deploy a distributed system to distribute the training data to multiple machine nodes with limited memory capacity that cannot accommodate all the training data for processing, and to ensure that the machine learning algorithm is trained normally in multiple nodes through an efficient data communication mechanism. However, both of the above solutions involve hardware enhancement, or increasing memory, or adding computing devices and network communication equipment. They cannot solve the problem while maintaining the original machine hardware facilities, and the cost of use is high.

The inventors of this application have found in their in-depth research on the prior art that there are two other solutions in the related art that can achieve the conversion of machine learning training data without changing the hardware facilities of the computer equipment. Specifically, one solution is to upgrade the machine learning algorithm and transform the memory-based algorithm into a data streaming algorithm, an incremental learning algorithm or an iterative learning algorithm that supports disk input. These algorithm processes all require sequential access to training data; the other solution is approximate learning based on random sampling. Before training the model, a training subset suitable for the current memory capacity is randomly sampled from all the training data, and learning is performed based on the training subset, and finally an approximate model close to the standard model obtained by learning all the training data is obtained. Both solutions are non-hardware-dependent solutions, do not require changes to the machine hardware facilities, and are suitable for more application scenarios. However, both solutions involve disk access, and the performance bottleneck lies in the conversion process from the original disk file that stores all the training data to the sequential access data stream required by the machine learning algorithm or the sample set that does not exceed the current memory capacity, as shown in Figure 1. The data conversion process involves view conversion of disk file data, such as adding, deleting, or modifying certain columns or filtering certain rows, and requires efficient view conversion based on disk, and all data cannot be loaded into memory for processing. Therefore, both of the above solutions need to introduce efficient disk data processing technology to implement the data conversion process shown in Figure 1, so as to ensure the overall effectiveness and practicality of the solution.

In view of this, the present application proposes a method and device for scheduling machine learning training data. The data conversion framework of the machine learning algorithm can be constructed through a preset general module layer, and the original training data file can be efficiently converted into the data required in the training phase of the machine learning algorithm, thereby improving the training data conversion efficiency of machine learning.

The basic idea of this application is to design the data conversion process in Figure 1 into a general module layer, which is composed of adapter modules as nodes. The nodes transfer data objects through the configured input and output ports, and the flow routes of these data objects constitute a directed acyclic graph. When the machine learning algorithm that needs to be called starts training, the grafting module connected to the machine learning algorithm initiates the execution of the general module layer to form a machine learning training module subgraph. For the adapter module nodes in the machine learning training module subgraph, they are processed layer by layer from top to bottom according to the original layering. The adapter modules of each layer concurrently execute all adapter modules with disk files or memory data output ports in the layer during processing. When the current layer adapter module is executed, it drives the upper layer adapter modules connected to the file pointer to execute together, and further drives these upper layer adapter modules to sequentially access the input disk file. The above scheme can effectively solve the problem of excessive memory requirements of machine learning algorithms. The scheme does not require the addition or modification of computer hardware facilities, and can ensure that ordinary computers can effectively process the massive training data stored in disk files, so that the training phase of the machine learning algorithm can be implemented for massive training data, and improve the training data conversion efficiency of computer equipment.

The following will specifically describe the method for scheduling machine learning training data shown in the present application. Please refer to FIG2, which is a flow chart of a method for scheduling machine learning training data shown in the present application. The method for scheduling machine learning training data can be applied to a computer device. The method shown in FIG2 includes:

S201. When the target machine learning algorithm starts training, obtain the grafting module corresponding to the target machine learning algorithm, and trigger the startup of the general module layer through the grafting module.

It should be noted that the computer device can store the training data required by multiple different machine learning algorithms, and the target machine learning algorithm refers to the machine learning algorithm detected by the computer device that needs to start training data conversion.

It should also be noted that the general module layer is a virtual module layer preset in advance for the conversion of machine learning algorithm training data. In one embodiment, the general module layer includes at least two adapter modules, one of which is a grafting module corresponding to the target machine learning algorithm; the adapter modules are configured as nodes of the general module layer according to the hierarchy, and are used to transfer data objects, the flow path of the data objects constitutes a directed acyclic graph in the general module layer, wherein the data objects include any one or more of disk files, file pointers, and memory data.

In one embodiment, the adaptation module is provided with at least one input port and at least one output port. The adaptation module receives the data object output by the upper layer node through the input port, and transmits the converted data object to the next layer node through the output port.

In one embodiment, the adaptation module includes any one or more types of a data view generation module, a data view column transformation module, a data view row transformation module, and a data view batch processing module.

The data object input to the data view generation module is a disk file, and the data object output is a file pointer for sequentially accessing the disk file. The data view column transformation module includes functional modules for adding columns, deleting columns, and changing column values in the data view; the data object input to the data view column transformation module is a file pointer, and the data object output is a file pointer. The data view row transformation module includes functional modules for randomly sampling rows in the data view, filtering rows according to specified conditions, filtering duplicate rows, and orderly merging data rows in multiple data views; the data object input to the data view row transformation module is any one or more of a disk file, a file pointer, and memory data, and the data object output is any one or more of a disk file, a file pointer, and memory data.

The data view batch processing module includes a batch processing module for a fixed number of rows and a batch processing module for a fixed number of sequences, wherein the fixed sequence refers to a set of data rows with the same ID; the data object input to the data view batch processing module is a file pointer, and the data object output is batched memory data, the batched memory data is used by an internal process, and the internal process is initiated by the target machine learning algorithm.

In one embodiment, the top-level node of the directed acyclic graph is the data view generation module, the bottom-level nodes of the directed acyclic graph are other types of modules except the data view generation module, and the middle nodes of the directed acyclic graph are other types of modules except the data view batch processing module.

As a feasible implementation method, the computer device can be designed in advance as a general module layer for training data conversion of machine learning algorithms, and the general module layer is composed of adapter modules as nodes. Each adapter module node can have multiple input ports, each input port receives data objects such as disk files, file pointers or memory data, and the node can also have multiple output ports, each port outputs data objects such as disk files, file pointers or memory data. Among them, there is at most one output port for the disk file, and at most one output port for the memory data, and the disk file can be represented by a file path. The nodes of the adapter module can transfer data objects through the input and output ports, and the flow routes of these data objects constitute a directed acyclic graph (DAG graph).

For example, please refer to FIG3, which is an example diagram of a general module layer provided in the present application. The general module layer shown in FIG3 includes three machine learning algorithms to be adjusted (it should be known that in other embodiments, any other number of machine learning algorithms to be adjusted may also be included) and seven adaptation modules. These seven adaptation modules are configured in a hierarchical manner, with adaptation module No. 2 located at the top layer, adaptation modules No. 7 to No. 10 located in the middle layer, and adaptation modules No. 15 and No. 17 located at the bottom layer; adaptation module No. 2 can be a data view generation module, adaptation modules No. 8 and No. 10 can be data view column transformation modules, adaptation modules No. 7, 9 and No. 15 can be data view row transformation modules, adaptation module No. 17 can be a data view batch processing module, the grafting module of machine learning algorithm No. 16 is adaptation module No. 9, the grafting module of machine learning algorithm No. 20 is adaptation module No. 15, and the grafting module of machine learning algorithm No. 21 is adaptation module No. 17.

In one embodiment, the grafting module of the target machine learning algorithm is the data view batch processing module or the output data object is an adaptation module for memory data.

For example, the batch processing module inputs a file pointer and outputs batches of memory data for use by an internal process, where the internal process is initiated by a machine learning algorithm that needs to be called. In this case, the grafting module of the machine learning algorithm can be defined as the corresponding data view batch processing module. The output port of the memory data can be connected to the machine learning algorithm that needs to be called. In this case, the grafting module of the machine learning algorithm can be defined as an adapter module that outputs memory data.

For another example, please refer to Figure 3. Assuming that the machine learning algorithm No. 16 needs to start training, the computer device can use the machine learning algorithm No. 16 as the target machine learning algorithm to be adjusted, and obtain the grafting module connected to the machine learning algorithm No. 16, that is, the adaptation module No. 9, and trigger the startup execution of the entire general module layer through the grafting module.

S202: Determine a target disk file according to the directed acyclic graph in the general module layer.

It should be noted that the target disk file refers to the disk file that can reach the grafting module corresponding to the target machine learning algorithm through a directed path in the directed acyclic graph.

As a feasible implementation method, when the target machine learning algorithm starts training, the grafting module corresponding to the target machine learning algorithm can initiate the execution of the general module layer. The flow routes of various data objects in the general module layer constitute the directed acyclic graph. The computer device can determine that the disk file that can reach the grafting module corresponding to the target machine learning algorithm through a directed path in the directed acyclic graph is the target disk file. The target disk file can be one or more, and the present application does not impose any restrictions on this.

In one embodiment, the target disk file is represented by a data view in the form of a two-dimensional table.

S203. Construct a machine learning training module subgraph according to the adaptation modules in all directed paths from the target disk file to the grafting module.

For example, when machine learning algorithm No. 16 is the target machine learning algorithm, the disk file involved is only No. 1, so its machine learning training module subgraph is composed of adaptation modules No. {1, 2, 6, 9, 13, 16}; similarly, when machine learning algorithm No. 20 is the target machine learning algorithm, the disk files involved are No. 1, 3, 4, and 11, so its machine learning training module subgraph is composed of node sets No. {1, 2, 3, 4, 5, 7, 8, 11, 12, 15, 18}.

S204. Convert the original training data in the target disk file into data required by the target machine learning algorithm based on the machine learning training module subgraph.

As a feasible implementation method, the conversion process from the disk file storing all the training data to the input data required by the machine learning algorithm can be performed in the machine learning training module subgraph of the target machine learning algorithm.

In one embodiment, the S204 step includes: according to the hierarchy of the adaptation module in the machine learning training module subgraph, the adaptation module corresponding to the target disk file is used as the top-level node, and the adaptation module is triggered and started layer by layer from top to bottom to convert the original training data of the target disk file into the data required by the target machine learning algorithm.

In one embodiment, the method also includes, when starting the adaptation module of the current layer, if the input data object of the started adaptation module is the file pointer, then driving the starting of the upper layer adaptation module connected to the file pointer; concurrently executing the linkage adaptation module in the current layer, the linkage adaptation module is an adaptation module whose output port data object is a disk file or memory data.

For example, in the example of the general module layer shown in FIG3, the top-down hierarchical structure of the adapter modules in the machine learning training module subgraph of the machine learning algorithm No. 16 is {2} and {9}; similarly, the top-down hierarchical structure of the adapter modules in the machine learning training module subgraph of the machine learning algorithm No. 20 is {2}, {7,8} and {15}. When executing the machine learning training module subgraph, the machine learning algorithm No. 16 first processes {2}. Since the output of the adapter module No. 2 in the subgraph is not a disk file or memory data, {2} is skipped and {9} is directly processed. The output of the adapter module No. 9 is memory data, so it is triggered to execute, and drives the upper-layer adapter module No. 2 connected to the file pointer to execute together, sequentially access the disk file No. 1, and generate the memory data No. 13. Similarly, when executing the machine learning training module subgraph, the machine learning algorithm No. 20 first processes {2}. Since the output of the adapter module No. 2 in the subgraph is a disk file, it is triggered to execute, and the disk file No. 1 is accessed sequentially to generate the disk file No. 4; then {7, 8} are processed in parallel. Since the output of the adapter module No. 7 in the subgraph is a disk file, and the output of the adapter module No. 8 in the subgraph is not a disk file or memory data, the adapter module No. 8 is jumped out and only the adapter module No. 7 is triggered to execute; since the adapter module No. 7 has no upper-level adapter module connected by the file pointer, it does not drive the upper-level adapter module to run, and only the disk file No. 3 and the disk file No. 4 are accessed sequentially to generate the disk file No. 11; finally, {15} is processed. Since the output of the adapter module No. 15 in the subgraph is memory data, it is triggered to execute, and the upper-level adapter module No. 8 and the adapter module No. 2 connected by the file pointer are successively driven to execute together, and the disk file No. 1 and the disk file No. 11 are accessed sequentially to generate the memory data No. 18.

It should be noted that, in the worst case, the adaptation module of each layer of the above-mentioned general module layer must be triggered to execute, and all the disk files of the upper layer must be accessed sequentially at most once during execution. Therefore, the general module layer can minimize the number of sequential accesses to the original training data files, and through the extraction of machine learning training module subgraphs and the concurrent execution of adaptation modules at each layer, the original training data files can be efficiently converted into the sequential access data stream required for input in the training phase of the machine learning algorithm or a training sample set that does not exceed the current memory capacity limit.

It can be seen that the machine learning training data scheduling method shown in the present application designs the conversion process of the original disk file storing all the training data to the input data required by the machine learning algorithm into a common module layer, so as to maximize the conversion efficiency without changing the hardware facilities of the computer device and minimizing memory usage. When the target machine learning algorithm to be called starts training, its grafted adapter module initiates the execution of the general module layer. The disk files involved are the disk files that can reach the grafted module through directed paths in the directed acyclic graph, and the nodes in all directed paths from these disk files to the grafted modules constitute the machine learning training module subgraph; in the process of data conversion, the adapter module nodes in the machine learning training module subgraph implement the same-layer parallel computing process, that is, the adapter module is processed layer by layer from top to bottom according to the original hierarchy, and the adapter module of each layer concurrently executes all the adapter modules with disk file or memory data output ports in the layer during processing, and the adapter modules without output disk files or memory data are not processed. When the current layer adapter module is executed, it drives the upper layer adapter modules connected to the file pointer to execute together, and further drives these upper layer adapter modules to sequentially access the input disk files. At most, all upper layer disk files can be accessed sequentially once to generate output disk files or memory data, which effectively improves the efficiency of training data conversion.

The above describes the method of the present application, and the following describes the execution device of the present application.

Referring to Fig. 4, Fig. 4 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, wherein the computer device 40 includes a processor 401 and a memory 402. The memory 402 is connected to the processor 401, for example, via a bus.

The processor 401 is configured to support the computer device 40 to perform the corresponding functions in the method in the above method embodiment. The processor 401 can be a central processing unit (CPU), a graphics processing unit (GPU) or a hardware chip. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

The memory 402 is used to store program codes, etc. The memory 402 may include a volatile memory (VM), such as a random access memory (RAM); the memory 402 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); the memory 402 may also include a combination of the above-mentioned types of memories.

The processor 401 may call the program code to perform the following operations:

When the target machine learning algorithm starts training, a grafting module corresponding to the target machine learning algorithm is obtained, and the startup of the general module layer is triggered by the grafting module;

Determine the target disk file according to the directed acyclic graph in the general module layer;

Construct a machine learning training module subgraph according to the adaptation modules in all directed paths from the target disk file to the grafting module;

Based on the machine learning training module subgraph, the original training data in the target disk file is converted into data required by the target machine learning algorithm.

An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory to implement the method described in the above embodiment.

A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims (9)

1. A machine learning training data scheduling method, for application to a computer device, the method comprising:

when a target machine learning algorithm starts training, acquiring a grafting module corresponding to the target machine learning algorithm, and triggering a universal module layer to start through the grafting module, wherein the universal module layer is a virtual module layer which is preset in advance and used for machine learning algorithm training data conversion, and is formed by taking an adaptation module as a node;

The adaptation module is used for transmitting data objects, wherein the data objects comprise any one or more of disk files, file pointers and memory data;

the adaptation module comprises any one or more types of a data view generation module, a data view column conversion module, a data view row conversion module and a data view batch processing module;

the data objects input by the data view generation module are disk files, and the data objects output by the data view generation module are file pointers for sequentially accessing the disk files;

The data view column transformation module comprises a functional module for adding columns, deleting columns and changing column values in the data view; the data object input by the data view column conversion module is a file pointer, and the data object output by the data view column conversion module is a file pointer;

The data view line transformation module comprises a functional module for randomly sampling lines in a data view, filtering the lines under a specified condition, filtering repeated lines and orderly combining the data lines in a plurality of data views; the data objects input by the data view line transformation module are any one or more of disk files, file pointers and memory data, and the data objects output by the data view line transformation module are any one or more of disk files, file pointers and memory data;

The data view batch processing module comprises batch processing with fixed line number and batch processing module with fixed sequence number, wherein the fixed sequence refers to a data line set with the same ID; the data objects input by the data view batch processing module are file pointers, the data objects output by the data view batch processing module are memory data of batches, the memory data of batches are used by an internal process, and the internal process is initiated by the target machine learning algorithm;

determining a target disk file according to the directed acyclic graph in the universal module layer;

Forming a machine learning training module subgraph according to the adaptation modules in all directed paths from the target disk file to the grafting module;

And converting the original training data in the target disk file into data required by the target machine learning algorithm based on the machine learning training module subgraph.

2. The method of claim 1, wherein the generic module layer comprises at least two adaptation modules, wherein one adaptation module is a grafting module corresponding to the target machine learning algorithm;

the adaptation modules are configured as nodes of the generic module layer in a hierarchy, and the flow paths of the data objects form a directed acyclic graph in the generic module layer.

3. The method of claim 2, wherein the adaptation module is provided with at least one input port and at least one output port, and wherein the adaptation module receives the data object output by the node of the previous layer through the input port and passes the converted data object to the node of the next layer through the output port.

4. The method of claim 1, further characterized in that a top level node of the directed acyclic graph is the data view generation module, a bottom level node of the directed acyclic graph is another type of module other than the data view generation module, and an intermediate node of the directed acyclic graph is another type of module other than the data view batch processing module.

5. The method of claim 1, further characterized in that the grafting module of the target machine learning algorithm is the data view batch processing module or an adaptation module of the output data object as memory data.

6. The method of claim 1, further characterized in that the target disk file is represented in a data view in the form of a two-dimensional table; the adaptation module in all directed paths from the target disk file to the grafting module forms a machine learning training module subgraph, which comprises:

The adaptation module of the target disk file in all the directed paths of the grafting module is obtained;

and constructing the machine learning training module subgraph according to the original hierarchical structure of the adaptation module in the universal module layer.

7. The method of claim 1, further characterized in that the converting the raw training data in the target disk file to data required by the target machine learning algorithm based on the machine learning training module subgraph comprises:

and triggering and starting the adaptation module layer by layer from top to bottom by taking the adaptation module corresponding to the target disk file as a top node according to the level of the adaptation module in the machine learning training module subgraph so as to convert the original training data of the target disk file into the data required by the target machine learning algorithm.

8. The method of claim 7, wherein the method further comprises:

When an adaptation module of a current layer is started, if an input data object of the started adaptation module is the file pointer, driving an upper layer adaptation module connected with the file pointer to be started;

And concurrently executing the linkage adaptation module in the current layer, wherein the linkage adaptation module is an adaptation module of which the data object of the output port is a disk file or memory data.

9. A computer device comprising a memory, a processor connected to the processor for executing one or more computer programs stored in the memory, which processor, when executing the one or more computer programs, causes the computer device to implement the method of any of claims 1-8.