CN118428457A - Configurable algorithm component packaging and combined modeling method - Google Patents

Abstract

The invention discloses a configurable algorithm component packaging and combination modeling method, which comprises the following steps: defining model library interface specifications, including model interface specifications, component interface specifications, data exchange standards and module joining standards; 2, packaging the neural network structure and the training process into an easy-to-use interface based on the platform-independent model interface; 3, designing a visual model assembly line to realize on-line assembly of the model based on a task-based distributed assembly mode; 4, carrying out parallel training and prediction on the online assembled model; the invention constructs an efficient, flexible and extensible big data analysis and deep learning platform from the standardized definition of the model interface, the diversification method of the packaging technology, the efficient implementation of the model assembly line, the application of the parallel training technology and the comprehensive optimization of the training and prediction flow together, and strengthens the convenience of users in actual operation and the maintainability of the model. By standardized interfaces and modular design, the system is made easier to update and maintain while maintaining compatibility with new technologies.

Description

Configurable algorithm component packaging and combined modeling method

Technical Field

The invention belongs to the technical field of development of computers and deep learning platforms, and particularly relates to a configurable algorithm component packaging and combination modeling method.

Background

In modern artificial intelligence technology, development and application of a deep learning platform are increasingly important. Such platforms typically require the provision of an open model library that manages the generated or imported analytical models, providing design support and assistance for AI developers to improve modeling efficiency. The open component libraries provided by such platforms are largely divided into two main categories: a deep neural network algorithm component and a specific functional component.

The deep neural network algorithm components include basic algorithm components such as convolvers, pooling devices, activation functions, and the like, as well as more complex algorithm models such as Inception models and ResNet models. These components provide a powerful tool for developers to build and optimize various neural network models.

Specific functional components are then focused on specific applications of the algorithm implementation, such as visualization of neuronal functions, computation of feature distribution, image spectrum analysis, etc. These functional components help developers to understand the internal workings of the model and to output results more deeply.

However, while the provision of these components greatly enriches the developer's tool box, the development of deep learning models still faces some technical challenges. The main problem is the underlying heterogeneity and interface complexity of the deep learning framework, which makes unified integration and combinatorial modeling between different components difficult. Furthermore, the interface and standardization issues between the different frames further increase the complexity of efficiently integrating these tools into a single platform.

Disclosure of Invention

The invention provides a configurable algorithm component packaging and combination modeling method, which is used for making a set of packaging standard and combination modeling method of a model, and allowing various components to be seamlessly integrated on the same platform, so that the flexibility and efficiency of model construction are improved; this not only can help the developer save time, but can also promote broader technological innovations and applications.

The invention is realized by adopting the following technical scheme:

The utility model provides a configurable algorithm component encapsulation and combination modeling method, which is characterized in that the method comprises the following steps:

defining model library interface specifications, including model interface specifications, component interface specifications, data exchange standards and module joining standards;

2, packaging the neural network structure and the training process into an easy-to-use interface based on the platform-independent model interface;

3, designing a visual model assembly line to realize on-line assembly of the model based on a task-based distributed assembly mode;

And 4, performing parallel training and prediction on the online assembled model.

In some embodiments of the present invention, the packaging of step 2 includes:

and (3) packaging: encapsulating the network structure, training process and reasoning process in a class using an object-oriented approach;

Function encapsulation: packaging the deep learning model as a function;

and (3) modularized packaging: decomposing the deep learning model into a plurality of modules, wherein each module is responsible for a specified function;

And (3) interface packaging: the framework allowing different deep learning works in the same calling mode by a unified interface;

Model save and load: such that training or reasoning is performed using the trained model or loading the pre-trained model in different environments;

And (3) packaging a configuration file: the hyper-parameters of the model, network structure and related information are encapsulated using the configuration file, allowing the user to change the behavior of the model by modifying the configuration file.

In some embodiments of the invention, in a task-based distributed component assembly mode, the component implements data parallelism and distributed random gradient descent across multiple GPUs; during the post-assembly model training and optimization, the caller obtains the state of, pauses, checks the state of, or adjusts the parameters of the gradient-based optimizer without modifying downstream components.

In some embodiments of the invention, in a task-based distributed component assembly mode, a remote call API is designed to launch more tasks and support stateful remote classes, and remote functions and classes meet API call requirements by providing remote call abstractions for Python functions and classes.

In some embodiments of the invention, the method further comprises:

The basic unit of the component module in deep learning is denoted by f & ltn & gt & gtH & ltR ^p→(Rⁿ→R^m & gt, and a new multiple-input multiple-output composite module is defined by combination, and a signal path from input to output is packaged in the composite module; the composite module introduces own super parameters and inherits the super parameters of the module serving as independent variables;

where n is the input dimension, H is the set of valid values for the hyper-parameters, p is the number of parameters, and m is the output dimension.

In some embodiments of the present invention, a task-mode-based model assembly line parallel trainer is designed in step 4, comprising an optimizer module for managing tasks of distributed sampling and gradient optimization training;

The parallel trainer embeds a plurality of SGD policies, including selection between synchronous and asynchronous execution, which may be automatically selected based on user specifications or optimizers.

In some embodiments of the invention, to make reasonable use of gradient optimization algorithms, the evaluator is abstract:

Where X is the empirical vector, θ is the model parameter, and ev is the evaluator.

Compared with the prior art, the invention has the advantages and positive effects that: the invention provides a configurable algorithm component packaging and combination modeling method.

Other features and advantages of the present invention will become more apparent from the following detailed description of embodiments of the present invention, which is to be read in connection with the accompanying drawings.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic illustration of an application framework of a configurable algorithm component package and combinatorial modeling approach of the present invention;

FIG. 2 is a schematic diagram of one embodiment of a configurable algorithm component package and a configuration file package in a combined modeling method according to the present invention

FIG. 3 is a schematic diagram of data and multiple cards of components in the configurable algorithm component package and combined modeling method according to the present invention;

FIG. 4 is a schematic overall flow diagram of line training and prediction in the configurable algorithm component packaging and combinatorial modeling method of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The invention aims to formulate a set of packaging standard and a combined modeling method of a model, provide a standard interface and standard specification, allow various components to be seamlessly integrated on the same platform, and enable developers or system developers to conduct algorithm importing and integrated modeling according to the specification, so that the flexibility and efficiency of model construction are improved; this not only can help the developer save time, but can also promote broader technological innovations and applications. Such standardization efforts need to be made with respect to the uniformity of interfaces, the compatibility of data formats, and the coordination of interactions between modules to ensure that components can efficiently cooperate under a unified architecture.

Specifically, firstly, standardized definitions are made for behaviors such as a model interface, a component interface, data exchange, module joining and the like, and corresponding standard specifications are formulated, including:

1. Model interface specification: and (3) formulating standards for generating the data analysis model, including a data set input mode, an analysis result output mode, parameter adjustment, function description, release standards and the like, so that the data analysis model is more unified and standardized, and the portability and reusability of the model are facilitated.

2. Component interface specification: criteria are formulated for related components involved, such as algorithm components, explicit inputs, outputs, parameter settings, and functional descriptions, etc., ensuring seamless integration of the components.

3. Data exchange criteria: standards are formulated for data exchange in the big data analysis process, such as sample data access format, intermediate process data display form, analysis result output format and the like, so that data processing inhibition is maintained, and data compatibility among different systems is promoted.

4. Module addition standard: and the model assembly line is oriented, the standard is formulated for adding the modules, corresponding standards and guidance are provided for the model assembly of developers, the situation that the model design work does not accord with logic is avoided, and the model design work is rapidly and accurately carried out.

Second, packaging of the assembly.

As shown in fig. 1, the encapsulation in the present invention follows a platform-independent model interface, packaging complex neural network structures and training processes into an easy-to-use interface to facilitate flexible invocation and integration of models by developers in various applications. In the illustration Pytorch, caffe, tensorflow, theano, CNTK and kears are some common deep learning frameworks within the field.

The goal of encapsulation is to improve the maintainability, scalability and reusability of the model, making it easier to deploy and share in different environments, including the following encapsulation modes:

1. And (5) packaging.

The network structure, training process and reasoning process are encapsulated in a class in an object-oriented manner. This encapsulation means helps organize the code, hide the internal implementation details, and provide a clear user interface for the user to invoke.

Membership functions of classes include initialization methods, training methods, reasoning methods, etc., such as:

class MyModel:

def__init__(self,input_size,output_size)；

Model # initialization

def train(self,train_data,epochs)；

Training model #

def predict(self,input_data)；

Model # prediction

2. And (5) function encapsulation.

The deep learning model is packaged as a function in a simple and direct mode, and a user can obtain corresponding output only by calling the function and transmitting input data. The packaging mode is suitable for small projects or rapid prototyping development, and reduces some complexity of similar packaging.

For example: defmy_model (input_data);

Model definition #

Model training

Return prediction result #

return prediction；

3. And (5) modularized packaging.

The deep learning model is broken down into a plurality of modules, each of which is responsible for a particular function, such as data processing, network construction, training and reasoning. Such modular packaging allows a user to selectively replace or adjust one of the modules as desired, enhancing the flexibility and customizable nature of the model.

For example: # Module 1: data processing

def preprocess_data(data)；

# Data preprocessing

# Module 2: network construction

def build_model()；

Definition of # network structure

# Module 3: training

def train_model(model,train_data)；

Model training

# Module 4: reasoning

def predict(model,input_data)；

Model # prediction

4. And (5) packaging an interface.

Providing a unified interface enables models of different deep learning frameworks to work in the same call mode. The encapsulation enables a user to seamlessly switch different frameworks, and improves portability of the model.

For example: def train_and_ predict (model, train_data, input_data);

Training model #

trained_model＝train(model,train_data)；

Model # prediction

prediction＝predict(trained_model,input_data)；

return prediction

5. Model save and load.

When the model is packaged, the storage and loading functions of the model are considered, so that a user can conveniently store the trained model or load the pre-trained model in different environments for further training or reasoning.

For example: defsave_model (model filepath);

# save model parameters

def load_model(filepath)；

# Load model parameters

6. And (5) packaging configuration files.

The configuration file is used to encapsulate the hyper-parameters of the model, network structure and other relevant information. This approach allows the user to change the behavior of the model by simply modifying the configuration file without having to go deep into the internal implementation of the model, as in the embodiment shown in fig. 2.

By these encapsulation methods, the use of models is facilitated, while the organization and maintenance of code is facilitated. And a proper packaging method is selected according to project requirements, so that the readability and maintainability of codes can be obviously improved, and meanwhile, the wide application and sharing of the model are promoted. These measures, while ensuring technical and operational specifications, represent a technical advancement and innovation of the embodiments of the present invention.

Third, model combinatorial modeling.

Through the function of a visual model assembly line, a developer can perform online assembly design of a model according to the requirements of big data analysis projects, and can select a deep neural network algorithm component or a functional component from an open component library, and can also select the algorithm library in a knowledge space and an existing algorithm model in the open model library, so that the design of the model is realized; the method comprises the following technical key points:

1. task-based distributed component assembly model.

With the development of deep learning, a large number of model components are presented, which are based on heterogeneous frameworks and have an indefinite component boundary. Moreover, these components may require parallelism across clusters, i.e., recursively issuing calls to other components using neural networks defined by a deep learning framework, or interfacing with black box third party simulators. In essence, many of these heterogeneous and distributed properties present key challenges to their parallel reasoning, making it difficult to package parallelism and resource requirements into a single component. Thus, reusing these distributed components requires inserting the appropriate control points in the program.

It is critical to achieve scalable modeling capability by combining and reusing existing components. For this reason, distributed algorithm components need to be built around the parallel encapsulation principle. This will enable one parallel or distributed component to call another, possibly in parallel, without any modification to the called component. Compared with the existing general distributed framework, the invention adopts a task-based distributed component assembly mode.

This pattern may enable data parallelism and distributed random gradient descent across multiple GPUs, which is illustrated in fig. 3 at multiple levels. During the post-assembly system model training and optimization process, the invoker of the gradient-based optimizer may wish to obtain its state, pause the optimizer, check its state, or adjust its parameters, and the traditional MPI distributed messaging framework requires explicit algorithm modifications to insert these coordination points. In this mode, however, for components that encapsulate parallelism and resource requirements, the caller can issue these instructions without modifying downstream components, making it easier to reuse, expand, and combine existing components with these design rules.

In addition, parallelizing the entire program using a framework such as MPI typically requires explicit instruction intervention to insert coordination points when trying to combine two programs or components together, which limits the ability to rapidly prototype new distributed applications, while building component libraries on top of task-based schemas, eases this requirement, and allows remote machines to schedule and execute asynchronously.

The task-based distributed component assembly model requires some primitives, and in addition, in order to allow more tasks to be initiated, the model must provide flexible remote call APIs; second, to provide optimal performance, it is necessary to support stateful remote classes; finally, the schema must support efficient communication between remote tasks and classes. The remote functions and classes satisfy the API call requirements by providing remote call abstractions for Python functions and classes. Any Python function can be run as a remote task and any Python class can be built remotely. In order to achieve efficient communication, standard communication primitives such as aggregation and broadcasting need to be provided, and zero-copy sharing of large data objects is achieved through a shared memory.

2. The model is assembled with the hyper-parametric model.

The basic unit of the modular assembly of components in deep learning can be expressed by the following formula:

f:n→(H→(R^p→(Rⁿ→R^m)) (1)；

Where n is the dimension of the input, H is the set of valid values for the hyper-parameters, p is the number of parameters, and m is the dimension of the output. The set H may be structured or may simply be a cross product of a scalar super parameter set, i.e. h=h ¹*H²……H^h, where H is the number of scalar super parameters, assuming that set H is discrete in both cases.

In equation (1), the number of parameters p and the output dimension m may both be a function of the input dimension n and the selected hyper-parameter value H e H. For example, the output dimension of a fully connected module with h hidden units is m=h, the number of parameters p is (n+1) h: the weight matrix W ε R ^h Xn and the bias vector b ε R ^h. A convolution module may also make similar reasoning: the number of parameters p depends on the input dimension, the number of filters and the size of the filters; the dimension of the output m depends on the input dimension, the number of filters, the size of the filters, the step size and the filling scheme, p and m being a function of the input dimension and the chosen hyper-parameter value.

The basic modules can be defined by the above formulas, and the basic modules perform some well-defined transformations, such as affine, batch normalization, dropout, etc., are examples of basic modules. On this basis, new multiple-input multiple-output composite modules can be defined by combining, which encapsulate many signal paths from input to output. Composite modules are defined in terms of other (composite or base) modules, i.e., instantiations of composite modules take other modules as arguments. For example, for a residual module, it is a single input before skipping connection splitting and a single output after skipping connection merging. Composite modules may introduce their own superparameters and inherit the superparameters of the modules as arguments. For example, the branching module obtains a list of modules and selects one of the modules to use. It introduces a discrete hyper-parameter for which module and selects values for the hyper-parameters of the selected module; the hyper-parameters available are conditioned on the selection of the module to be used. The characterization capability of the composite module comes from the componentization of the composite module and the base module.

The assembly model may be composed of a plurality of basic or composite modules, as shown below:

(Concat

(Conv2D[32,64][3,5][1])

(MaybeSwap BatchNormalization ReLU)

(Optional(Dropout[0.5,0.9]))

(Affine[10]))

This example is defined using three composite modules (Concat, maybeSwap and Optional) and five base modules (Conv 2D, batchNormalization, reLU, dropout and Affine). Concat does not introduce additional hyper-parameters, but it must specify all modules that have been delegated to it; maybeSwap introduces a binary hyper-parameter for encoding the order of whether pairs are swapped. The number of modules; optional introduces a binary hyper-parameter for coding whether a module is included. The behavior of the basic module in this example is relatively simple: conv2D acquires a list of possible values for the number of filters, the size of the filters and the step size; batchNormalization and ReLU have no hyper-parameters; dropout takes a list of possible values for the loss probability; affine (Affine) obtains a list of possible values for the number of hidden units.

Selecting different values for the superparameters of the composite module may affect the structure of the resulting architecture, whereas selecting different values for the superparameters of the base module may only affect the structure of the corresponding local transformations, which the module may characterize by its inputs, outputs, parameters and superparameters. The composite modules have different granularities, e.g., given an Empty base module that has no superparameters or parameters, but performs identity transformation, and an or composite module that introduces an additional superparameter to encode the selection of a particular module in its list, the composite modules Optional and MaybeSwap can be defined as (Optional B) = (or Empty B) and (MaybeSwap-B1B 2) = (or (Concat B1B 2), (Concat B B1)).

3. Parallel training and prediction of model assembly lines.

Another advantage of task model based combinable assembly is that training is performed faster. It will be appreciated that considering a training task that runs continuously until completion, it does not provide the caller of the task with an opportunity to adjust the training unless the task is terminated. The task explicitly defines the way to run an iterative training, between iterations, control decisions can be made, such as whether to continue training, check points, adjust parameters, or stop. Iterations typically call multiple remote classes that maintain state between calls to improve efficiency.

Accordingly, the present invention proposes a task-mode based model assembly line parallel trainer that includes an optimizer module for managing the common tasks required for distributed sampling and gradient optimization training. The trainer takes the form of a number of built-in SGD policies, including selection between synchronous and asynchronous execution, which the user can specify, and which the optimizer can automatically select. To make reasonable use of these gradient-optimized algorithms, an evaluator Evaluator abstraction needs to be implemented.

The evaluator is a class defining the following method, where X is an empirical vector, θ is a model parameter, and ev is an evaluator:

Any object implementing these interfaces is Evaluator. Typical SGD programming APIs implement a single step training step (L (θ), X, θ) →θ _opt. The parallel trainer takes as input a local Evaluator and remote evaluator set, implementing distributed parameter updating: step (E _local,E_1..n,θ)→θ_opt).

Evaluator abstraction has the advantage that, first, the optimizer can make decisions based on the location information from Evaluator. Second, this API encapsulates the SGD policy, enabling the user to choose between synchronous and asynchronous implementations without modifying the algorithm code. Finally, the Evaluator interface fully encapsulates the model graph and the penalty function, allowing the SGD implementation to be improved and reused in different deep learning frameworks. All gradient-based algorithms use this flexible SGD interface. In the task-based model, it is easy to extract distributed SGD abstractions because the parallelism of SGD components can be fully encapsulated, and also by providing pluggable gradient descent SGD modules that allow all deep learning libraries with composable components to enable rapid development and prototyping of new algorithms using high performance components, facilitating rapid development of new neural network architectures.

Through the technical means of the three aspects, the design of the model assembly line not only improves the flexibility and expansibility of big data analysis projects, but also ensures the high efficiency and operability of the model under various environments.

4. Model assembly line training and predicting the overall process.

The training process is shown in fig. 4, and comprises a metadata collection and mapping stage, a data preprocessing stage, a model training stage, a model storing and loading stage and a prediction stage; what is particularly needed is metadata mapping to apply exactly the same preprocessing to the input data at the time of prediction, while preserving model weights and superparameters to load the same exact model obtained during training. The preprocessed data can be selectively cached in the HDF5 file, and the HDF5 file is used next time when the same data is accessed, so that the time required by preprocessing is saved. The innovation of this approach is that each component is dynamically built from declarative task definitions, from preprocessing to modeling to training loops.

The preprocessing flow is shown at the bottom of the figure, it uses the metadata obtained during the training phase to preprocess the new input data, the loading model reads its hyper-parameters and weights, and uses it to obtain predictions that are mapped back into the data space by the post processor using the same mapping obtained during training.

Thus far, specific embodiments of the entire model library interface specification, packaging techniques, and model combination modeling techniques have been described in detail.

From standardized definition of model interfaces, diversified methods of packaging technology, efficient implementation of model assembly lines, application of parallel training technology, and comprehensive optimization of training and prediction processes, the embodiments together construct an efficient, flexible and extensible big data analysis and deep learning platform.

In addition, the implementation of the techniques not only improves the technical performance and adaptability of the model, but also strengthens the convenience of users in actual operation and the maintainability of the model. By standardized interfaces and modular design, the system is made easier to update and maintain while maintaining compatibility with new technologies.

Finally, the system design also considers the actual application demands of users, and supports the specific demands under different scenes by providing flexible model combination and assembly options, so that the platform can serve a wider application field, such as various big data and AI fields of financial analysis, image processing, natural language processing and the like.

It should be noted that, in the specific implementation process, the above method part may be implemented by executing, by a processor in a hardware form, computer execution instructions in a software form stored in a memory, which is not described herein, and the program corresponding to the executed action may be stored in a computer readable storage medium of the system in a software form, so that the processor invokes and executes the operations corresponding to the above modules.

The computer readable storage medium above may include volatile memory, such as random access memory; but may also include non-volatile memory such as read-only memory, flash memory, hard disk, or solid state disk; combinations of the above types of memories may also be included.

The processor referred to above may be a general term for a plurality of processing elements. For example, the processor may be a central processing unit, or may be other general purpose processors, digital signal processors, application specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or may be any conventional processor or the like, but may also be a special purpose processor.

It should be noted that the above description is not intended to limit the invention, but rather the invention is not limited to the above examples, and that variations, modifications, additions or substitutions within the spirit and scope of the invention will be within the scope of the invention.

Claims (7)

1. A configurable algorithm component package and combinatorial modeling method, comprising:

defining model library interface specifications, including model interface specifications, component interface specifications, data exchange standards and module joining standards;

2, packaging the neural network structure and the training process into an easy-to-use interface based on the platform-independent model interface;

3, designing a visual model assembly line to realize on-line assembly of the model based on a task-based distributed assembly mode;

And 4, performing parallel training and prediction on the online assembled model.

2. The configurable algorithm component package and combinatorial modeling method of claim 1, wherein the package of step 2 comprises:

and (3) packaging: encapsulating the network structure, training process and reasoning process in a class using an object-oriented approach;

Function encapsulation: packaging the deep learning model as a function;

and (3) modularized packaging: decomposing the deep learning model into a plurality of modules, wherein each module is responsible for a specified function;

And (3) interface packaging: the framework allowing different deep learning works in the same calling mode by a unified interface;

Model save and load: such that training or reasoning is performed using the trained model or loading the pre-trained model in different environments;

And (3) packaging a configuration file: the hyper-parameters of the model, network structure and related information are encapsulated using the configuration file, allowing the user to change the behavior of the model by modifying the configuration file.

3. The configurable algorithmic component packaging and combinatorial modeling method of claim 1, wherein in a task-based distributed component assembly mode, the component implements data parallelism and distributed random gradient descent across multiple GPUs; during the post-assembly model training and optimization, the caller obtains the state of, pauses, checks the state of, or adjusts the parameters of the gradient-based optimizer without modifying downstream components.

4. The configurable algorithmic component packaging and combinatorial modeling method of claim 1, wherein in a task-based distributed component assembly model, a remote call API is designed to launch more tasks and support stateful remote classes, and remote functions and classes meet API call requirements by providing remote call abstractions for Python functions and classes.

5. The configurable algorithm component packaging and combinatorial modeling method of claim 1, further comprising:

By using A basic unit representing a component module in deep learning, wherein a new multiple-input multiple-output composite module is defined by combination, and a signal path from input to output is packaged in the composite module; the composite module introduces own super parameters and inherits the super parameters of the module serving as independent variables;

where n is the input dimension, H is the set of valid values for the hyper-parameters, p is the number of parameters, and m is the output dimension.

6. The configurable algorithm component packaging and combinatorial modeling method of claim 1, wherein in step 4, a task model based model assembly line parallel trainer is designed, comprising an optimizer module for managing tasks of distributed sampling and gradient optimization training;

The parallel trainer embeds a plurality of SGD policies, including selection between synchronous and asynchronous execution, which may be automatically selected based on user specifications or optimizers.

7. The configurable algorithm component packaging and combinatorial modeling method of claim 6, wherein to make rational use of gradient optimization algorithms, the evaluator is abstracted:

; where X is the empirical vector, θ is the model parameter, and ev is the evaluator.