CN115114470B

CN115114470B - A model search method, device, equipment, storage medium, and program product

Info

Publication number: CN115114470B
Application number: CN202210505507.7A
Authority: CN
Inventors: 柳露艳; 郑冶枫
Original assignee: Tencent Healthcare Shenzhen Co Ltd
Current assignee: Tencent Healthcare Shenzhen Co Ltd
Priority date: 2022-05-10
Filing date: 2022-05-10
Publication date: 2025-05-13
Anticipated expiration: 2042-05-10
Also published as: CN115114470A

Abstract

The application provides a model searching method, a device, equipment, a storage medium and a program product; the method comprises the steps of searching a first weight of network granularity and a plurality of second weights of operator granularity for a plurality of network units in a target super network respectively based on a search dataset, extracting at least two target network units from the plurality of network units of the target super network according to the first weight, connecting the at least two target network units to obtain a target network structure, generating a target operator corresponding to each target network unit of the target network structure based on the plurality of second weights and a plurality of operation operators, constructing and obtaining a target network model by using the target operator and each target network unit, and completing model searching. The application can improve the performance of model searching.

Description

Model searching method, device, equipment, storage medium and program product

Technical Field

The present application relates to computer vision technology in the field of artificial intelligence, and in particular, to a model searching method, apparatus, device, storage medium, and program product.

Background

The model search refers to a process of searching the most suitable network model from the super network for the image processing task so as to improve the processing effect of the image processing task. The model search can be widely applied to scenes such as image segmentation, image recognition and the like.

The model searching mode in the related technology is to search for an operation operator, so that the network model obtained by searching has structural limitation, the difficulty of searching for the optimal network model is increased, and the performance of model searching is influenced.

Disclosure of Invention

The embodiment of the application provides a model searching method, device and equipment, a computer readable storage medium and a program product, which can improve the performance of model searching.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides a model searching method, which comprises the following steps:

Searching a first weight of network granularity and a plurality of second weights of operator granularity for a plurality of network units in a target super network respectively from a search space based on a search dataset, wherein the network granularity is the weight granularity influencing the external structure of a search model, the operator granularity is the weight granularity influencing operators in the search model, and the target super network is a set formed by all candidate network structures;

Extracting at least two target network units from a plurality of network units of the target super network according to the first weight, and connecting the at least two target network units to obtain a target network structure;

Generating a target operator corresponding to each target network element of the target network structure based on a plurality of second weights and a plurality of operation operators;

And constructing and obtaining a target network model by using the target operator and each target network unit, and completing model searching.

The embodiment of the application provides a model searching device, which comprises:

the weight searching module is used for searching a first weight of network granularity and a plurality of second weights of operator granularity for a plurality of network units in a target super-network respectively from a search space based on a search dataset, wherein the network granularity is the weight granularity influencing the external structure of a search model, the operator granularity is the weight granularity influencing the operator of the search model, and the target super-network is a set formed by all candidate network structures;

the structure determining module is used for extracting at least two target network units from the plurality of network units of the target super network according to the first weight, and connecting the at least two target network units to obtain a target network structure;

an operator generating module, configured to generate a target operator corresponding to each target network element of the target network structure based on a plurality of second weights and a plurality of operation operators;

And the model construction module is used for constructing and obtaining a target network model by utilizing the target operator and each target network unit to finish model searching.

In some embodiments of the present application, the weight search module is further configured to construct a sparse super-network based on the search space, wherein the depth and the width of the target super-network are greater than those of the sparse super-network, iteratively update the sparse super-network by the search dataset, determine a candidate space from the search space, iteratively update the target super-network by the search dataset, and search the candidate space for each network element of the target super-network for the first weight of a network granularity and the second weights of an operator granularity, respectively.

In some embodiments of the present application, the search dataset includes training image data and verification image data, the weight search module is further configured to update model parameters of a first initial super-network of a kth round of iteration based on the training image data to obtain a first temporary super-network of the kth round of iteration, wherein k is a positive integer, the first initial super-network of the 1 st round of iteration is the sparse super-network, update weights of network granularity of each network element of the first temporary super-network of the kth round of iteration based on the verification image data to obtain a first intermediate super-network of the kth round of iteration, update weights of operator granularity of each network element of the first intermediate super-network of the kth round of iteration based on the verification image data to obtain a first updated super-network of the kth round of iteration, and use the first updated super-network of the kth round of iteration as a first initial super-network of the k+1 th round of iteration, and determine that the weights of each network of the first updated super-network of the kth round of iteration are candidates of the first iteration from the space when k M reaches the first candidate operator of the k round of iteration.

In some embodiments of the present application, the weight searching module is further configured to perform difference calculation with respect to a candidate weight of an operator granularity in the search space and a weight of an operator granularity of each network element of the first update super-network of the mth round of iteration to obtain a weight difference, and reject, from the search space, the candidate weight of the operator granularity with the weight difference greater than a difference threshold to obtain the candidate space.

In some embodiments of the present application, the weight search module is further configured to segment the region of the object of interest of the training image data through a first initial super-network of a kth iteration to obtain a first segmented region, and update model parameters of the first initial super-network of the kth iteration by using the first segmented region and a loss value of the object of interest between labeling regions in the training image data to obtain a first temporary super-network of the kth iteration.

In some embodiments of the present application, the weight searching module is further configured to segment the region of the object of interest of the verification image data through the first temporary super network of the kth round of iteration to obtain a second segmented region, and update the weight of the network granularity of each network element of the first temporary super network of the kth round of iteration by using the second segmented region and the loss value of the object of interest between the labeling regions in the verification image data to obtain the first intermediate super network of the kth round of iteration.

In some embodiments of the present application, the weight searching module is further configured to segment the region of the object of interest of the verification image data through the first intermediate super network of the kth round of iteration to obtain a third segment region, and update the weight of the operator granularity of each network element of the first intermediate super network of the kth round of iteration by using the third segment region and the loss value of the object of interest between the labeling regions in the verification image data to obtain the first updated super network of the kth round of iteration.

In some embodiments of the present application, the search dataset includes training image data and verification image data, the weight search module is further configured to update a model parameter of a second initial super-network of an ith round of iteration based on the training image data to obtain a second temporary super-network of the ith round of iteration, i is a positive integer, the second initial super-network of the 1 st round of iteration is a target super-network, update a weight of a network granularity of each network element of the second temporary super-network of the ith round of iteration based on the verification image data to obtain a second intermediate super-network of the ith round of iteration, update a weight of an operator granularity of each network element of the second intermediate super-network of the ith round of iteration based on the verification image data to obtain a second updated super-network of the ith round of iteration, and use the second updated super-network of the ith round of iteration as a second initial updated network of the ith+1 th round of iteration, and determine that the weight of each network element of the second updated super-network of the ith round of iteration is the second operator granularity of the second iteration is N when the weight of each network element of the ith round of iteration reaches N.

In some embodiments of the application, the model searching device further comprises a data set construction module, wherein the data set construction module is used for acquiring image data sets of a plurality of fields before searching a first weight of network granularity and a plurality of second weights of operator granularity for a plurality of network units in a target super network respectively in a search space based on a search data set, determining a corresponding extraction proportion for the image data sets of each field based on a preset probability distribution, extracting image data to be mixed of each field from the image data sets of each field according to the extraction proportion, and integrating the image data to be mixed of the plurality of fields into the search data set.

In some embodiments of the present application, the operator generating module is further configured to fuse output layers of a plurality of operation operators based on a plurality of the second weights, so as to obtain a target operator corresponding to each target network element of the target network structure, where the plurality of operation operators include at least a horizontal fusion operator, an up-sampling operator, and a down-sampling operator;

The model construction module is further configured to add the target operator to each target network element of the target network structure, obtain a target network model, and complete model searching.

In some embodiments of the present application, the model searching device further includes an image segmentation module, and the image segmentation module is configured to construct a target network model by using the target operator and each target network unit, and segment, by using the target network model after the model searching is completed, an interest region where the object of interest is located from the acquired medical image data.

The embodiment of the application provides model searching equipment, which comprises the following steps:

a memory for storing executable instructions;

and the processor is used for realizing the model searching method provided by the embodiment of the application when executing the executable instructions stored in the memory.

The embodiment of the application provides a computer readable storage medium which stores executable instructions for realizing the model searching method provided by the embodiment of the application when being executed by a processor.

The embodiment of the application provides a computer program product, which comprises a computer program or instructions, wherein the computer program or instructions realize the model searching method provided by the embodiment of the application when being executed by a processor.

The embodiment of the application has the advantages that the model searching equipment can search and obtain the first weight of the network granularity and the second weight of the operator granularity for each network element in the target super network at the same time, and the target network elements required by the network model are determined through the first weight so as to be connected and obtain the external network structure, namely the target network structure is determined, and the target operators of each network element are constructed through the second weight and different operation operators, so that the network level searching and the operator level searching are simultaneously carried out during the model searching, and the structure of the network model obtained by searching is more diversified, thereby improving the performance of the model searching.

Drawings

Fig. 1 is a schematic diagram of an encoding-decoding network;

FIG. 2 is a schematic diagram of a variant network that generates an encoding-decoding network based on model searching;

FIG. 3 is a schematic diagram of a model search system according to an embodiment of the present application;

FIG. 4 is a schematic diagram of the structure of the server in FIG. 3 according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of a model searching method according to an embodiment of the present application;

FIG. 6 is another schematic flow chart of a model searching method according to an embodiment of the present application;

FIG. 7 is a schematic flow chart of a model searching method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a virtual dataset provided by an embodiment of the present application;

fig. 9 is a schematic structural diagram of a woven network according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a conventional unit provided by an embodiment of the present application;

fig. 11 is a comparison chart of effects of segmenting medical image data according to an embodiment of the present application.

Detailed Description

The present application will be further described in detail with reference to the accompanying drawings, for the purpose of making the objects, technical solutions and advantages of the present application more apparent, and the described embodiments should not be construed as limiting the present application, and all other embodiments obtained by those skilled in the art without making any inventive effort are within the scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.

In the following description, the terms "first", "second", "third" and the like are merely used to distinguish similar objects and do not represent a particular ordering of the objects, it being understood that the "first", "second", "third" may be interchanged with a particular order or sequence, as permitted, to enable embodiments of the application described herein to be practiced otherwise than as illustrated or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.

Before describing embodiments of the present application in further detail, the terms and terminology involved in the embodiments of the present application will be described, and the terms and terminology involved in the embodiments of the present application will be used in the following explanation.

1) Artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) is the theory, method, technique, and application system that simulates, extends, and extends human intelligence using a digital computer or a machine controlled by a digital computer, perceives the environment, obtains knowledge, and uses the knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.

The artificial intelligence technology is comprehensive learning, and relates to the technology with wide fields, hardware level and software level. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, deep learning/deep learning, intelligent driving and other directions.

2) Computer Vision (CV) is a science of how to "look" at a machine, and more specifically, to replace a camera and a Computer to perform machine Vision such as identifying and measuring a target by human eyes, and further perform graphic processing, so that the Computer is processed into an image more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D techniques, virtual reality, augmented reality, synchronous positioning, and map construction, among others, as well as common biometric recognition techniques such as face recognition, fingerprint recognition, and others.

3) Machine learning (MACHINE LEARNING, ML) is a multi-domain interdisciplinary, involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and the like. It is specially studied how a computer simulates or implements learning behavior of a human to acquire new knowledge or skills, and reorganizes existing knowledge structures to continuously improve own performance. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, confidence networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like.

4) Model search (Nerral Architecture Search, NAS) is a strategy for automatically designing neural networks. By setting a certain search space, a search strategy is designed to find the network structure in the search space that performs best on the verification dataset.

5) Continuous relaxation is a method of serializing a discrete space, essentially transforming a sequence from discrete space to continuous space using the Softmax function.

6) Super networks (supernetworks) refer to the collection of all possible subnetworks involved in the model search process. Based on the set search space, a super-network may be generated that includes a plurality of sub-networks that are trained to be used for the performance metrics being evaluated.

7) A network unit for stacking modules (blocks) generating a network structure.

8) The operation operator, a small network module formed by combining a plurality of convolution layers, pooling layers and other basic network layers, can be used for performing operation processing on an image (or a feature map), for example, performing feature combination on the feature map, or performing compression on the feature map, and the like. The operator operators can be classified into downsampling operators, upsampling operators, standard operators (i.e., operators with the same input and output sizes), compression operators, multi-scale operators, and the like, according to the operation implemented.

With research and advancement of artificial intelligence technology, artificial intelligence technology has been developed for research and application in various fields, such as intelligent homes for scenes, intelligent wearable devices, virtual assistants, intelligent sound boxes, intelligent marketing, unmanned driving, automatic driving, unmanned vehicles, robots, intelligent medical treatment, intelligent customer service, etc. It is believed that with the development of artificial intelligence technology, artificial intelligence technology will find application in more fields and play a great role.

In the related art, the model searching is to search for a model operator aiming at each network element in the network structure, namely, the model searching is that the network element in the network structure searches for a proper operator from a search space, and the operator and the existing network structure are utilized to integrate to obtain a final network model.

However, in the related art model searching mode, searching is performed with respect to an operator, so that the structure of the network model obtained by searching is always limited by the network structure used for performing the model searching, that is, the network model has structural limitation.

Illustratively, fig. 1 is a schematic diagram of an encoding-decoding network. An encoding-decoding (Encoder-decoder) network 1-1 is composed of encoding paths 1-11 and decoding paths 1-12, and between the encoding paths 1-11 and decoding paths 1-12, there is a jump 1-13. In the encoding paths 1-11, the computational effort is reduced, the receptive field is increased, and the robustness to small input fluctuations is improved, reducing the overfit by downsampling operations, and in the decoding paths 1-12 the pixel loss during downsampling is restored by upsampling operations to perform the end-to-end image segmentation task. However, the encoding-decoding network 1-1 cannot recover the spatial information loss generated during the downsampling process, and therefore, the skip connection 1-13 is introduced, so that the low-order features (the low-order features have more spatial structure information) on the encoding path and the high-order features (the high-order features lose the spatial structure information) on the decoding path are fused, so that the fused features integrate more bottom-layer features, and the segmented feature map has more accurate edge information.

FIG. 2 is a schematic diagram of a variant network that generates an encoding-decoding network based on model search. Referring to fig. 2, the variant network 2-2 searches for an appropriate operator 2-21 from standard operators (Normal Cell), compression operators (Reduce Cell) and multiscale operators (Multi-SCALE CELL) for each network element 2-22 in the encoding-decoding network 2-1, so that the structure of the network element 2-22 of the variant network 2-2 is more diversified to adapt to the image segmentation tasks of different fields.

Therefore, the variant network generated by the model search has great similarity with the structure of the decoding-encoding network, namely the structure of the network model obtained by the search is limited, and the optimal network model is difficult to search, so that the performance of the model search is influenced.

In addition, the data set used for model search in the related art is mostly in the field of natural images (for example, images of living scenes, images of animals and plants, and the like). When the model search is applied to the image segmentation field, particularly the medical image segmentation field, since the medical image data is very limited and has a strong Domain Gap (Domain Gap), the model search for the medical image segmentation task is more prone to selecting a parameter-free operation, which affects the performance of the searched network model, that is, the performance of the model search is poor.

The embodiment of the application provides a model searching method, device and equipment, a computer readable storage medium and a program product, which can improve the performance of model searching. The exemplary application of the model search device provided by the embodiment of the present application is described below, and the model search device provided by the embodiment of the present application may be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device), and other various types of terminals, and may also be implemented as a server. In the following, an exemplary application in which the model search apparatus is implemented as a server will be described.

Referring to fig. 3, fig. 3 is a schematic architecture diagram of a model search system according to an embodiment of the present application. To enable support for a model search application, in model search system 100, terminal 400 (terminal 400-1 and terminal 400-2 are illustratively shown) is connected to server 200 via network 300, and network 300 may be a wide area network or a local area network, or a combination of both. In the model search system 100, a database 500 is also provided to provide data support to the server 200. Database 500 may be integrated into server 200 or may be independent of server 200. Fig. 1 shows a case where the database 500 is independent of the server 200.

The terminal 400-1 is used to generate a search data set and transmit the search data set to the server 200 through the network 300.

The server 200 is configured to search a first weight of a network granularity and a plurality of second weights of operator granularities for a plurality of network elements in a target super-network from a search space based on a search dataset, wherein the network granularity is the weight granularity influencing an external structure of a search model, the operator granularity is the weight granularity influencing an operator inside the search model, the target super-network is a set formed by all candidate network structures, extract at least two target network elements from the plurality of network elements of the target super-network according to the first weight and connect the at least two target network elements to obtain a target network structure, generate a target operator corresponding to each target network element of the target network structure based on the plurality of second weights and a plurality of operation operators, and construct and obtain a target network model by using the target operator and each target network element to complete the model search.

The server 200 is further configured to issue the target network model to the terminal 400-2. The terminal 400-2 is configured to segment the region of interest for the medical image data using the target network model and display the region of interest in the graphical interface 410-2.

In some embodiments, the server 200 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart home appliance, a vehicle-mounted terminal, a medical image analysis device, etc. The terminal and the server may be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiment of the present application.

Referring to fig. 4, fig. 4 is a schematic diagram of the structure of the server (an implementation of the model search device) in fig. 3 according to an embodiment of the present application, and the server 200 shown in fig. 4 includes at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in server 200 are coupled together by bus system 240. It is understood that the bus system 240 is used to enable connected communications between these components. The bus system 240 includes a power bus, a control bus, and a status signal bus in addition to the data bus. But for clarity of illustration the various buses are labeled as bus system 240 in fig. 4.

The Processor 210 may be an integrated circuit chip having signal processing capabilities such as a general purpose Processor, such as a microprocessor or any conventional Processor, a digital signal Processor (DSP, digital Signal Processor), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like.

The user interface 230 includes one or more output devices 231, including one or more speakers and/or one or more visual displays, that enable presentation of media content. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard drives, optical drives, and the like. Memory 250 optionally includes one or more storage devices physically located remote from processor 210.

Memory 250 includes volatile memory or nonvolatile memory, and may also include both volatile and nonvolatile memory. The non-volatile Memory may be a Read Only Memory (ROM) and the volatile Memory may be a random access Memory (RAM, random Access Memory). The memory 250 described in embodiments of the present application is intended to comprise any suitable type of memory.

In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules and data structures, or subsets or supersets thereof, as exemplified below.

An operating system 251 including system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and handling hardware-based tasks;

A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include bluetooth, wireless compatibility authentication (Wi-Fi), and universal serial bus (USB, universal Serial Bus), among others;

A presentation module 253 for enabling presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 (e.g., a display screen, speakers, etc.) associated with the user interface 230;

an input processing module 254 for detecting one or more user inputs or interactions from one of the one or more input devices 232 and translating the detected inputs or interactions.

In some embodiments, the model searching apparatus provided by the embodiments of the present application may be implemented in software, and fig. 4 shows the model searching apparatus 255 stored in the memory 250, which may be software in the form of a program, a plug-in, and the like, including software modules including a weight searching module 2551, a structure determining module 2552, an operator generating module 2553, a model constructing module 2554, a dataset constructing module 2555, and an image dividing module 2556, which are logical, so that any combination or further splitting may be performed according to the implemented functions. The functions of the respective modules will be described hereinafter.

In other embodiments, the model searching apparatus provided in the embodiments of the present application may be implemented in hardware, and by way of example, the model searching apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor that is programmed to perform the model searching method provided in the embodiments of the present application, for example, the processor in the form of a hardware decoding processor may employ one or more Application Specific Integrated Circuits (ASICs), DSPs, programmable logic devices (PLDs, programmable Logic Device), complex Programmable logic devices (CPLDs, complex Programmable Logic Device), field Programmable Gate Arrays (FPGAs), field-Programmable GATE ARRAY), or other electronic components.

In some embodiments, a server (an implementation of the model searching apparatus) may implement the model searching method provided by the embodiment of the present application by running a computer program. For example, the computer program may be a Native program or a software module in an operating system, a Native Application (APP), i.e., a program that needs to be installed in the operating system to run, such as a model search APP, an applet, i.e., a program that needs to be downloaded only to a browser environment to run, or an applet that can be embedded in any APP. In general, the computer programs described above may be any form of application, module or plug-in.

The embodiment of the application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation, vehicle-mounted, medical treatment and the like. The model searching method provided by the embodiment of the present application will be described below in connection with exemplary applications and implementations of the model searching apparatus provided by the embodiment of the present application.

Referring to fig. 5, fig. 5 is a schematic flow chart of a model searching method according to an embodiment of the present application, and the steps shown in fig. 5 will be described.

S101, based on a search data set, searching a first weight of network granularity and a plurality of second weights of operator granularity for a plurality of network units in a target super network respectively from a search space.

The embodiment of the application is realized in the scene of performing model search for an image processing task, for example, performing model search for a medical image segmentation task and performing model search for an object recognition task. After the model searching process starts, firstly, the model searching device acquires a searching data set from a database or a storage space of the model searching device, then the searching data set is input into a built target super-network, the searching data set is utilized to search the weight of each network element in the target super-network on the network granularity in the searching space, finally, the first weight of each network element is determined, the searching is carried out on the weight of each network element on the operator granularity, and the respective second weights of a plurality of operation operators are determined for each network element, so that a plurality of second weights are obtained.

It should be noted that, the network granularity is a weight granularity that affects the external structure of the search model, the operator granularity is a weight granularity that affects the operators inside the search model, and the target super-network is a set formed by all candidate network structures. That is, from the target supernetwork, a plurality of different network models can be created according to the weights of the network granularity and the weights of the operator granularity of each network element. The first weight is the weight of the searched preferred network granularity, the second weight is the weight of the searched preferred operator granularity, the model searching device influences the external structure of the searching model (such as whether jump connection exists among different network units, how many network units the model shares, etc.) according to the first weight of each network unit, and influences operators in the searching model (such as whether a certain network unit is an up-sampling operator or a down-sampling operator) according to a plurality of second weights of each network unit, so that a more reasonable network model can be created for an image processing task.

In some embodiments, the model searching device may perform update iteration on the target super network through the search dataset to obtain the first weight and the second weight by searching in the search space, that is, obtain the first weight and the second weight by one search.

In other embodiments, the model searching device may further construct a super network having a smaller scale than the target super network, then update and iterate the super network through the search dataset, determine a smaller candidate space from the search space, and search the target super network by using the search dataset to update and iterate the target super network, thereby obtaining the first weight and the second weight through two-stage searching.

It will be appreciated that each network element in the search space having a supernetwork has candidate weights that are selectable at the network granularity, and that each element has candidate weights that are selectable at the operator granularity.

S102, extracting at least two target network units from a plurality of network units of the target super network according to the first weight, and connecting the at least two target network units to obtain a target network structure.

The model searching device can determine whether each network element in the target super-network can bring about improvement of processing performance (such as improvement of accuracy, acceleration of reasoning speed, and the like) for the image processing task by using the first weight obtained by searching, so as to extract the network element which brings about improvement of processing performance for the image processing task from a plurality of network elements in the target super-network, and the extracted network element is used as the target network element. In this way, the model search device is able to obtain at least two target network elements. Then, the model searching device connects at least two target network units according to the sequence of the target network units in the target super network or the sequence reverse to the sequence of the target network units in the target super network, so as to obtain a target network structure.

In some embodiments, the model search device may determine a network element with a first weight greater than a weight threshold as the target network element. In other embodiments, the model searching apparatus may further determine the N network elements with the largest first weights as the target network elements. The embodiments of the present application are not particularly limited herein.

It should be noted that the operation operator is not yet added to the target network element determined in this step, so that only the external network structure of the searched model, for example, the number of network elements, the location of each network element in the target structure model, and so on, are characterized by using the target network structure spliced by the plurality of target network elements.

S103, generating a target operator corresponding to each target network element of the target network structure based on the second weights and the operators.

In some embodiments, the model searching device may weight the plurality of operators according to the plurality of second weights, and use the weighted operators to perform fusion, so as to obtain a target operator of each target network element.

In other embodiments, the model searching apparatus may compare the plurality of second weights with weight thresholds of operator granularities, respectively, and then extract operators with the second weights greater than or equal to the weight thresholds from the plurality of operators, and determine the extracted operators as target operators.

S104, constructing a target network model by using the target operators and each target network unit, and completing model searching.

After the target operators corresponding to each target network unit are obtained through fusion, the model searching device may add the target operators to each target network unit, so that each target network unit can process an image (or a feature map) to obtain a target network model capable of being used for performing an image processing task, or may perform pruning, compression and other processing on some target operators, and add the processed target operators to corresponding target network units to obtain the target network model.

It can be understood that, compared with the method that the model search in the related art searches the operation operators, so that the structure of the network model obtained by the search is limited, in the embodiment of the application, the model search device can search for each network element in the target super network to obtain the first weight of the network granularity and the second weight of the operator granularity, and determine the target network element required by the network model through the first weight so as to connect and obtain the external network structure, namely determine the target network structure, and construct the target operator of each network element through the second weight and different operation operators, thereby realizing the simultaneous search of the network level and the operator level during the model search.

Based on fig. 5, referring to fig. 6, fig. 6 is another flow chart of a model searching method according to an embodiment of the present application. In some embodiments of the present application, based on the search dataset, searching the first weight of the network granularity and the second weights of the operator granularity for the network elements in the target super-network, respectively, from the search space, i.e., the specific implementation process of S101, may include S1011-S1013 as follows:

s1011, constructing a sparse super network based on the search space.

And S1012, carrying out iterative updating on the sparse super network through searching the data set, and determining a candidate space from the search space.

It should be noted that the depth and width of the target super-network are larger than those of the sparse super-network. That is, the model searching device reconstructs a sparse super network with smaller scale for the search space, initializes the weight of the network granularity and the weight of the operator granularity of each network unit of the sparse super network from the search space, and performs update iteration on the weight of the network granularity and the weight of the operator granularity obtained by initializing each network unit of the sparse super network by utilizing the search dataset to screen candidate weights of the network granularity and the candidate weights of the operator granularity with good reasoning effect for the sparse super network on the search dataset in the search space (can be determined by the weight of the network granularity and the weight of the operator granularity selected for the sparse super network after the update iteration is completed), and eliminates the candidate weights of the network granularity and the candidate weights of the operator granularity with poor reasoning effect for the sparse super network on the search dataset in the search space, namely, extracts more excellent candidate space from the search space.

And S1013, carrying out iterative updating on the target super network through searching the data set, and respectively searching a first weight of network granularity and a plurality of second weights of operator granularity for each network unit of the target super network from the candidate space.

After determining the candidate space from the search space, the model search device initializes the weights of the network granularity and the weights of the operator granularity for each network element of the target super network from the candidate space, and then performs update iteration on the target super network by using the search data set so as to continuously update the weights of the network granularity of each network element in the target super network until the update iteration is finished, taking the weights of the updated network granularity of the last round as a first weight, and obtaining a plurality of second weights for each network element in the same manner.

It can be understood that in the embodiment of the application, because the scale of the sparse super network is smaller, the parameters required to be calculated are fewer, the time required for performing iterative updating is less, and when the model searching device performs the first-stage search based on the sparse super network, the candidate space can be determined from the search space by using less time, that is, the scale of the search space can be effectively reduced by using shorter time, so that the first weight and the second weight can be directly searched from the candidate space with smaller scale for the target super network, and the efficiency of model searching can be effectively improved.

In some embodiments of the application, the search dataset includes training image data and verification image data. The iterative updating of the sparse super network by searching the dataset determines candidate space from the search space, i.e. the specific implementation process of S1012 may include S1012a-S1012d as follows:

And S1012a, updating model parameters of a first initial super-network of the kth iteration based on training image data to obtain the first temporary super-network of the kth iteration, wherein the initial super-network of the 1 st iteration is a sparse super-network.

If the positive integer k is used to represent the round of updating iteration for the sparse super-network, the model searching device updates the model parameters of the first initial super-network of the kth round of iteration by using training image data in the searching data set when the kth round of iteration is performed, and determines the super-network obtained after the model parameters are updated as the temporary super-network of the kth round of iteration. Wherein k is a positive integer,

It should be noted that, the model searching device uses the constructed sparse super-network as the first initial super-network of the 1 st iteration, uses the super-network obtained after the 1 st iteration is completed as the first initial super-network of the next iteration, and can realize the iteration update of the sparse super-network by repeating the steps.

It will be appreciated that for any one super-network, the model parameters are the parameters of the lowest layer thereof, which can be understood as parameters of the network layers constituting the operator, such as the convolutional layer, the pooling layer, etc.

And S1012b, updating the weight of the network granularity of each network unit of the first temporary super-network of the kth iteration based on the verification image data to obtain the first intermediate super-network of the kth iteration.

After the first temporary super-network of the k-th iteration is obtained, the model searching device inputs verification image data into the first temporary super-network of the k-th iteration, so that the reasoning result in the verification image data and the difference of labels of the verification image data are aimed at through the first temporary super-network of the k-th iteration, the weight of the network granularity of each network unit of the first temporary super-network of the k-th iteration is updated, and the super-network obtained after the weight updating of the network granularity is completed is the first intermediate super-network of the k-th iteration.

It should be noted that, the training image data and the verification image data in the search data set have corresponding labels, where the label of the training image data indicates the labeling area of the target object in the training image data, and the label of the verification image data indicates the labeling area of the target object in the verification image data.

And S1012c, updating the weight of the operator granularity of each network unit of the first intermediate super network of the k-th round of iteration based on the verification image data to obtain the first updated super network of the k-th round of iteration, and taking the first updated super network of the k-th round of iteration as the first initial super network of the k+1-th round of iteration.

After the first intermediate super-network of the kth iteration is obtained, the model searching device inputs the verification image data into the first intermediate super-network of the kth iteration again, updates the weight of the operator granularity of each network element of the first intermediate super-network of the kth iteration according to the difference between the reasoning result of the verification image data and the label of the verification image data through the first intermediate super-network of the kth iteration, determines the super-network obtained after the update is finished as the first update super-network of the kth iteration, and finally takes the first update super-network of the kth iteration as the start of the kth+1 iteration so as to continuously start the model parameter update process of the kth+1 iteration.

S1012d, when k is reached to M, determining a candidate space from the search space based on the weights of the operator granularities of each network element of the first update super-network of the mth round of iterations.

Note that M is the total number of first iterations. When k reaches M, the model search device enters the last iteration round for the sparse super network, i.e., the mth iteration round, to obtain the first updated super network of the mth iteration round. Then, the model searching device uses the weight of the operator granularity of each network element of the first update super-network of the mth round of iteration to reject the weight of the operator granularity which does not perform well in the search space, so as to obtain a candidate space.

In the embodiment of the application, the model searching equipment can update the model parameters of the super network, the weights of the network granularity and the operator granularity weights through searching training image data and verification image data in the data set in the kth round of iteration, and the searching space is reduced by utilizing the weights of the operator granularity of the last round of iteration, so that the parameter quantity of the searching space is reduced.

In some embodiments of the present application, determining a candidate space from the search space, i.e., a specific implementation of S1012d, based on the weights of the operator granularities of each network element of the first update super-network of the mth round of iteration may include S201-S202 as follows:

S201, performing difference calculation on candidate weights of operator granularity in a search space and weights of operator granularity of each network unit of the first updating super-network of the Mth round of iteration to obtain weight differences.

It may be understood that the model searching device may obtain the weight difference by making a difference between the candidate weight of the operator granularity in the search space and the weight of the operator granularity of each network element of the first updated super-network of the mth round of iteration, or may obtain the weight difference by comparing the candidate weight of the operator granularity with the weight of the operator granularity of each network element of the first updated super-network of the mth round of iteration.

S202, eliminating candidate weights with the operator granularity, wherein the weight difference is larger than the difference threshold value, from the search space to obtain a candidate space.

The model searching equipment acquires a difference threshold value, then compares the weight difference with the difference threshold value, eliminates candidate weight values of operator granularity corresponding to the weight difference larger than the difference threshold value from the search space, and determines the search space with the elimination operation as a candidate space.

It may be appreciated that the difference weight may be set manually, or may be determined after the model searching device analyzes the field of the search data set or the similarity of each image data in the search data set by using an artificial intelligence technology, which is not limited herein.

In some embodiments of the present application, updating the model parameters of each network element of the first initial super-network of the kth iteration based on the training image data to obtain the first temporary super-network of the kth iteration, i.e. the specific implementation process of S1012a, may include S203-S204 as follows:

s203, performing region segmentation of the object of interest on the training image data through a first initial super network iterated in the kth round to obtain a first segmented region.

The model searching device inputs training image data into a first initial network of a kth iteration to divide the region where the object of interest is located from the training image data by using the first initial network of the kth iteration to obtain a first division region.

It is understood that the object of interest may be a lesion, organ, etc. in the medical field, or may be an object commonly found in daily life, such as a tree, a traffic sign on a roadside, etc.

S204, updating model parameters of the first initial super-network of the kth iteration by using the first segmentation region and loss values of the interest objects among the labeling regions in the training image data to obtain the first temporary super-network of the kth iteration.

The model searching equipment carries out loss calculation on the first segmentation area and the labeling area where the interest object is actually located in the training image data to obtain a loss value of the labeling area corresponding to the first segmentation area and the training image data, then solves the partial derivative value of the original model parameter in the first initial super-network of the kth round of iteration aiming at the loss value, calculates an updating component of the model parameter by utilizing the product between the partial derivative value and the learning rate, and updates the model parameter of the first initial super-network of the kth round of iteration by utilizing the updating component, so that the first temporary super-network of the kth round of iteration can be obtained after the updating is finished.

Illustratively, equation (1) is a calculation process for updating model parameters of the first initial super-network of the kth round of iterations:

Where w _k is the first initial supernetwork original model parameter for the kth iteration, η _w represents the learning rate of the model parameter, Representing training image data in the search dataset, alpha _k being the weight of the operator granularity of each network element of the first initial super-network of the kth round of iterations, beta _k being the weight of the network granularity of each network element of the first initial super-network of the kth round of iterations,Is the loss value of the first segmentation region and the labeling region corresponding to the training image data,Is the bias operation, w _k+1 is the model parameters updated by the first initial super-network of the kth iteration. The model search device is able to complete updating of the model parameters of the first initial super-network for the kth round of iterations, as per equation (1).

In some embodiments of the present application, updating the weight of the network granularity of each network element of the first temporary super-network of the kth round of iteration based on the verification image data to obtain the first intermediate super-network of the kth round of iteration, namely, the specific implementation process of S1012b, may include S205-S206 as follows:

S205, performing region segmentation of the object of interest on the verification image data through a first temporary super network iterated in the kth round to obtain a second segmented region.

S206, updating the weight of the network granularity of each network unit of the first temporary super-network of the kth iteration by using the second segmentation region and the loss value of the interest object between the labeling regions in the verification image data to obtain the first intermediate super-network of the kth iteration.

The model searching device inputs the verification image data into a first temporary super-network of the kth round of iteration, and the region where the object of interest is located is segmented from the verification image data by using the first temporary super-network of the kth round of iteration, and the region is determined to be a second segmentation region. Then, the model searching device calculates the loss value of the second segmentation region and the interest object in the region where the verification image data is located, namely the labeling region corresponding to the verification image data, calculates the partial derivative of the weight of the network granularity for the loss value, for example, calculates the partial derivative and the learning rate of the weight of the network granularity, determines the updating component of the weight of the network granularity, and then updates the weight of the network granularity of each network element of the first temporary super network of the kth round of iteration by using the determined updating component, so that the first intermediate super network of the kth round of iteration is obtained after the updating is completed.

The process of updating the weight of the network granularity of each network element of the first temporary super-network of the kth round of iteration in the embodiment of the present application may be expressed as formula (2):

where β _k is the weight of the network granularity of each network element of the first temporary super-network of the kth round of iterations, w _k+1 is the model parameters of the first temporary super-network of the kth round of iterations (i.e., the model parameters after the first initial super-network update of the kth round of iterations), a _k is the weight of the operator granularity of each network element of the first initial super-network of the kth round of iterations, (w _k+1;α_k,β_k) is the first temporary super-network of the kth round of iterations, Is to verify the image data and,Is the loss of the second segmentation region and the labeling region corresponding to the verification image data, eta _β is the learning rate corresponding to the weight of the network granularity, beta _k+1 is the weight of the network granularity after each network element of the first temporary super-network of the kth iteration is updated,Is a deviation-solving operation. The model search device is able to complete updating of the weights of the network elements of the first intermediate super-network at the network granularity for the kth round of iterations, according to equation (2).

In some embodiments of the present application, updating the weight of the operator granularity of each network element of the first intermediate super-network of the kth iteration based on the verification image data, to obtain the first updated super-network of the kth iteration, i.e. the specific implementation process of S1012c may include S207-S208 as follows:

S207, performing region segmentation of the interest object on the verification image data through the k-th round of iterative first update super network to obtain a third segmented region

And S208, updating the weight of the operator granularity of each network unit of the first intermediate super-network of the kth iteration by using the third segmentation region and the loss value of the interest object between the labeling regions in the verification image data to obtain a first updated super-network of the kth iteration.

The model searching device calculates a loss value of a third segmentation region and a labeling region corresponding to verification image data, calculates a partial derivative of a weight value of an operator granularity for the loss value, for example, calculates a partial derivative value and a learning rate of the weight value of the operator granularity, determines an updating component of the weight value of the operator granularity, updates the weight value of the operator granularity of each network unit of a first intermediate super-network of a kth round of iteration by using the determined updating component, and determines the super-network obtained after the updating as the first updating super-network of the kth round of iteration.

The process of updating the weight of the operator granularity of each network element of the first update super-network of the kth round of iteration in the embodiment of the present application can be expressed as formula (3):

Where a _k is the weight of the operator granularity of each network element of the first intermediate super-network of the kth round of iterations, β _k+1 is the weight of the network granularity of each network element of the first intermediate super-network of the kth round of iterations (i.e., the weight of the network granularity of each network element of the first intermediate super-network of the kth round of iterations after updating), (w _k+1;α_k,β_k+1) represents the first intermediate super-network of the kth round of iterations, Is the loss value of the labeling area corresponding to the third segmentation area and the verification image data, eta _α is the learning rate corresponding to the weight of the operator granularity,Is a bias operation, and α _k+1 is a weight of the operator granularity updated by each network element of the first intermediate super-network of the kth iteration. The model search device is able to complete the updating of the weights at the operator granularity for the network elements of the first intermediate super-network of the kth round of iterations, according to equation (3).

In some embodiments of the present application, when the search dataset includes training image data and verification image data, iteratively updating the target super-network by the search dataset, searching for a first weight of network granularity and a plurality of second weights of operator granularity, respectively, from each network element of the target super-network in the candidate space, i.e., the specific implementation process of S1013 may include S1013a-S1013d, as follows:

and S1013a, updating model parameters of the second initial super-network of the ith iteration based on the training image data to obtain the second temporary super-network of the ith iteration.

Wherein i is a positive integer, and the second initial super-network of the 1 st iteration is a target super-network.

S1013b, updating the weight of the network granularity of each network unit of the second temporary super-network of the ith round of iteration based on the verification image data to obtain a second intermediate super-network of the ith round of iteration

S1013c, updating the weight of the operator granularity of each network unit of the second intermediate super network of the ith round of iteration based on the verification image data to obtain a second updated super network of the ith round of iteration, and taking the second updated super network of the ith round of iteration as a second initial super network of the (i+1) th round of iteration.

The specific implementation of S1013a to S1013c is similar to the specific implementation of S1012a to S1012c, and the description thereof will not be repeated here.

S1013d, when i reaches N, determining the weight of each network unit network granularity of the second updated super network of the nth iteration as a first weight, and determining the weight of each operator granularity of the second updated super network of the nth iteration as a second weight.

Where N is the total number of second iterations. When i reaches N, namely the total number of second iterations is reached, the model searching equipment carries out the last iteration update on the target super-network to obtain a second updated super-network of the nth iteration. And then, the model searching equipment extracts the weight of each network element of the second updating super network in the N round of iteration at the network granularity to obtain a first weight, and extracts the weight of each network element at the operator granularity to obtain a plurality of second weights. The model search device thus completes the determination of the first weight and the second weight.

Based on fig. 6, referring to fig. 7, fig. 7 is a schematic flowchart of a model searching method according to an embodiment of the present application. In some embodiments of the present application, based on the search dataset, before searching the first weight of the network granularity and the second weights of the operator granularity, respectively, from the search space for the plurality of network elements in the target super network, i.e. before S101, the method may further comprise S105-S108 as follows:

s105, acquiring image data sets of a plurality of fields.

S106, based on the preset probability distribution, corresponding extraction proportion is determined for the image data set of each field.

The model searching device may acquire image datasets of a plurality of different fields, for example, a dataset of a medical image segmentation field and a dataset of a medical image classification field, from a database or a network, and then allocate the corresponding extraction ratio to the image dataset of each field by using a preset probability distribution.

It is to be understood that the preset probability distribution may be a Beta distribution or a gaussian distribution, and the embodiment of the present application is not limited herein.

For example, when the preset probability distribution is a Beta distribution, i.e. Beta (μ, μ), the model search device may incorporate the super parameter μ e (0, + -infinity), determine lambda ₁,λ₂,…,λ_a, then the normalization processing is carried out for { lambda ₁,λ₂,…,λ_a }, obtaining the extraction ratio of each field to obtain Where a is the number of fields.

And S107, extracting the image data to be mixed of each field from the image data set of each field according to the extraction proportion.

S108, integrating the data sets to be mixed in the multiple fields into a search data set.

The model search apparatus calculates the number of image data to be extracted from the image data set of each domain, using the total number of image data contained in the image data set of each domain and the extraction ratio of each domain, then extracts image data to be mixed randomly or sequentially in the image data set of each domain according to the number, thereby obtaining image data to be mixed of each domain, and integrates the image data to be mixed of each domain into one image data set, which is the search data set.

For example, when the extraction ratios of the a fields are respectivelyWhen the model search apparatus may obtain a search dataset according to equation (4) for the image dataset of each domain:

Wherein, Is a search dataset and its labels, { (x ₁,y₁),(x₂,y₂),…,(x_a,y_a) } is image data and corresponding labels extracted from image datasets of multiple fields.

In the embodiment of the application, the model searching equipment can mix the image data in the image data sets in a plurality of different fields to obtain the searching data set, so that the searching data set can overcome the field gaps among the different fields, thereby improving the characteristic generalization capability during the model searching, helping the model searching to search the optimal model with fully aggregated characteristics and further improving the performance of the model searching.

In some embodiments of the present application, based on the plurality of second weights and the plurality of operation operators, the specific implementation process of generating the target operator corresponding to each target network element of the target network structure, that is, S103, may include S1031 as follows:

s1031, based on the second weights, fusing the output layers of the operation operators to obtain a target operator corresponding to each target network element of the target network structure.

The model searching device weights the output layer of each operator by using the second weight corresponding to each operator, and fuses the weighted output layers into one (which can be understood as that the weighted output layer is accessed into the output layer of the next network unit) to obtain the target operator corresponding to each target network unit. That is, in this step, the model search device fuses a plurality of operators into the target operators, instead of selecting one operator from the plurality of operators as the target operator corresponding to each target network element, so that the processing of the target operator corresponding to each target network element for the input image (or the feature map) is sufficient, and the feature generalization capability of the target operator is enhanced.

In some embodiments of the present application, the plurality of operators includes at least a horizontal fusion operator, an upsampling operator, and a downsampling operator. The horizontal fusion operator is an operator with the same size of the input feature and the same size of the output feature, the up-sampling operator is an operator with the size of the input feature smaller than the size of the output feature, and the down-sampling operator is an operator with the size of the input feature smaller than the size of the output feature. The horizontal fusion operator, the upsampling operator, and the downsampling operator can all be modeled as Directed Acyclic Graphs (DAGs).

In some embodiments of the present application, the building of the target network model by using the target operator and each target network element to complete the model search, that is, the specific implementation process of S104 may include:

S1041, adding a target operator to each target network element of the target network structure to obtain a target network model, and completing model searching.

The model searching device can take the input of the target operator as the input of the target network structure and the output of the target operator as the input of the target network structure, so that the target operator can be added into a corresponding target network unit, and a target network model capable of processing an input image (or a feature map) can be obtained.

In some embodiments of the present application, the target operator and each target network element are utilized to construct a target network model, and the model search is completed, that is, after S104, the method may further include S109, as follows:

s109, segmenting an interest area where the interest object is located from the acquired medical image data by using the target network model.

That is, the model searching apparatus may perform region segmentation of the object of interest with respect to the medical image data by acquiring the medical image data after searching for the target network model to label the region of interest from the medical image data to assist the diagnosis process of the medical personnel through the region of interest.

Of course, the model search device may also send the target network model to a specialized electronic medical device to enable the electronic medical device to use the target network model to image segment the medical image data information into tasks.

In the following, an exemplary application of the embodiment of the present application in a practical application scenario will be described.

The embodiment of the application is realized in a scene of carrying out model search on an image segmentation task of medical image data by a server, so that a region of interest, such as a region where a focus is located, a region where an organ is located and the like, can be marked from the medical image data through the searched model.

The server (model search device) will first construct a virtual dataset (search dataset) for the model search process. The virtual data set is mixed with medical image data of a plurality of data sets, and the problem of generalization of an image segmentation task for the medical image data in model searching can be solved.

Wherein the virtual dataset can be constructed by formula (4). Fig. 8 is a schematic diagram of a virtual data set according to an embodiment of the present application. The server determines the extraction ratio of the images for the medical data set 8-1, the medical data set 8-2 and the medical data set 8-3 (image data sets in multiple fields), randomly extracts the image data from the three medical data sets according to the extraction ratio, and forms a virtual data set 8-4 by using the extracted image data, so that the medical image segmentation model (target network model) is searched from the searchable woven network 8-5 (super network) by using the virtual data set 8-4.

The searchable woven network consists of an up-sampling unit (up-sampling operator), a down-sampling unit (down-sampling operator) and a conventional unit (horizontal fusion operator) which respectively represent feature fusion in the top-down, bottom-up and horizontal directions.

Fig. 9 is a schematic structural diagram of a woven network according to an embodiment of the present application. The woven network of fig. 9 has a depth of 6 (i.e., 0 to 5) and a width of 10 (i.e., 0 to 9), and the profile of each network element (i.e., hexagon in fig. 9) is obtained by a weighted sum of the outputs of up-sampling element 9-1, down-sampling element 9-2, and regular element 9-3, e.g., profile X ^2,8 in fig. 9, weighted by the output of 9-2 (input as profile X ^1,7), up-sampling element 9-1 (input as profile X ^3,7), and the output of regular element 9-3 (input as profiles X ^2,6 and X ^2,4) passing through the down-sampling element. Depth maps 9-4 are outputs of preprocessing network elements (e.g., STEM elements) for images.

The server will search the search space at both the unit level (operator granularity) and the network level (network granularity) to obtain the optimal combination of different network elements and the operations corresponding to each network element. In the search space, optional operations (regular-operations) with ownership values at the network level and at the unit level are taken by the regular unit, downsampling operations (down-ops) are taken by the downsampling unit in addition to the regular operation, and upsampling operations (up-ops) are taken by the upsampling unit in addition to the regular operation. Taking the conventional cell as an example, it can be modeled as a directed acyclic graph. Illustratively, FIG. 10 is a schematic diagram of a conventional unit provided by an embodiment of the present application, and as can be seen from FIG. 10, the conventional unit 10-1 can be connected as a directed acyclic graph by the outputs 10-11 of other units, as well as the network layers 10-12 within.

The optimization function of the server when searching the woven network can be as shown in the formula (5):

Where α is the blending weight (operator granularity weight) of the unit level operation, β is the blending weight (network granularity weight) of the network level, and w is the network weight (model parameter). For loss functions AndIt is indicated that it may be calculated based on several depth maps, e.g. based on X ^1,5、X^1,7、X^1,9 in fig. 9, which are determined by the network weights w and the structural parameters (α and β), respectively. Thus, for the optimization of equation (5), it is a three-level optimization problem, where the optimization parameter α is the uppermost optimization variable, β is the middle optimization variable, and w ^* is the bottommost optimization variable. Therefore, prior to optimizing α and β, it is necessary to solve for w ^* according to the constraints described above.

In order to make the searching process more efficient, the server will first use the complete searching space, set a shallower woven network (sparse network) with depth of 5 and width of 8, search for a certain epoch, halve the searching space according to the parameters of unit-level operation after the searching in the first stage is finished, i.e. reject the operation set with poor performance, then reconstruct the woven network with depth of 6 and width of 10, and perform the final searching in the searching space (candidate space) after halving, through which searching the searching space and efficiency can be balanced.

Next, a search process of the server will be described.

First, a virtual data set is acquired and divided into a training set (training image data) and a verification set (verification map data), and a learning rate at a unit level, a learning rate at a network level, and a learning rate of a network weight are set.

In the first search stage, initializing a super-network from an operation set and a shallower woven network, then performing iterative training on the network weight of the super-network by utilizing a training set, obtaining the super-network with the better network weight after a certain number of iterations, and then performing iteration by utilizing a virtual data set based on the super-network with the better network weight as an iterative initial network. At this time, the server updates the network weights by using the training set according to the calculation process of formula (1), updates the network-level mixed weights by using the verification set according to the method of formula (2), and updates the unit-level mixed weights by using the verification set according to the method of formula (3) until the first search stage is completed.

After the first search phase is completed, the server will halve the operation set, i.e., for example, the operation set of the regular unit is O ₁, the operation set of the up-sampling unit is O ₂, the operation set of the down-sampling unit is O ₃, and the server will start the second search phase based on halve (O ₁)、halve(O₂) and halve (O ₃).

In the second search phase, the server initializes the super network (target super network) from the halved operation set and the deeper woven network, and then completes the second search phase in a similar manner to the first search phase.

Next, performance of model search provided by the embodiment of the present application will be described.

The server runs the model searched by the model searching method provided by the embodiment of the application and the manually designed model on three data sets of the ISIC2018, the CVC and the CHASS-CT respectively. The results of the operation are shown in Table 1:

TABLE 1

Therefore, the model searched by the model searching method provided by the embodiment of the application can obtain higher performance than manual design except the model based on CVC by searching the model obtained by searching the ISIC data set and the CVC data set CHASS-CT data set. This is mainly due to the smaller CHASS-CT dataset, which is more prone to choosing no parameter operation when searching for models, so that the performance of the searched models is limited. On the basis of the model searching method of the embodiment of the application, the virtual data sets are continuously overlapped, namely, the consistency and the similarity of the models obtained based on the virtual data sets (the models with mu=0, mu=0.5 and mu=1) are better, so that the characteristic generalization capability of the model searching can be improved through the virtual data sets.

Referring to fig. 11, fig. 11 is a graph showing the effect of segmenting medical image data according to an embodiment of the present application. The model 11-1, the model 11-2 and the model 11-3 are models obtained by searching on a virtual data set based on the model searching method provided by the embodiment of the application, and the model 11-4 is a model designed manually. For image 11-5 of the CVC dataset, image 11-6 of the ISIC dataset, and image 11-7 of the CHASS-CT dataset, models 11-1, 11-2, and 11-3 can each achieve smoother, sharper segmentation results than model 11-4, closer to true value 11-8. Therefore, by applying the model searching method provided by the embodiment of the application to the virtual data set, a model with good segmentation effect on medical image data can be searched, so that the performance of model searching is improved.

It will be appreciated that in embodiments of the present application that related data, such as medical image data, is relevant to user information, user permissions or consent may be required when embodiments of the present application are applied to a particular product or technology, and that the collection, use and processing of the relevant data may be required to comply with relevant laws and regulations and standards of the relevant country and region.

Continuing with the description below of an exemplary architecture of the model search device 255 implemented as a software module provided by an embodiment of the present application, in some embodiments, as shown in fig. 4, the software modules stored in the model search device 255 of the memory 250 may include:

The weight searching module 2551 is configured to search, based on the search dataset, a first weight of a network granularity and a plurality of second weights of an operator granularity for a plurality of network elements in a target super-network respectively in a search space, where the network granularity is a weight granularity that affects an external structure of a search model, the operator granularity is a weight granularity that affects an operator inside the search model, and the target super-network is a set formed by all candidate network structures;

the structure determining module 2552 is configured to extract at least two target network elements from the plurality of network elements of the target super-network according to the first weight, and obtain a target network structure by using connection of the at least two target network elements;

An operator generating module 2553, configured to generate, based on a plurality of the second weights and a plurality of operation operators, a target operator corresponding to each of the target network elements of the target network structure;

The model construction module 2554 is configured to construct a target network model by using the target operator and each target network element, and complete model searching.

In some embodiments of the present application, the weight searching module 2551 is further configured to construct a sparse super-network based on the search space, wherein the depth and the width of the target super-network are greater than those of the sparse super-network, iteratively update the sparse super-network by the search dataset, determine a candidate space from the search space, iteratively update the target super-network by the search dataset, and search the candidate space for the first weight of a network granularity and the second weights of an operator granularity for each network element of the target super-network, respectively.

In some embodiments of the present application, the search dataset includes training image data and verification image data, the weight search module 2551 is further configured to update model parameters of a first initial super-network of a kth round of iteration based on the training image data to obtain the first temporary super-network of the kth round of iteration, where k is a positive integer, the first initial super-network of the 1 st round of iteration is the sparse super-network, update weights of network granularity of each network element of the first temporary super-network of the kth round of iteration based on the verification image data to obtain a first intermediate super-network of the kth round of iteration, update weights of operator granularity of each network element of the first intermediate super-network of the kth round of iteration based on the verification image data to obtain the first updated super-network of the kth round of iteration, and use the first updated super-network of the kth round of iteration as the first initial super-network of the k+1 th round of iteration, and determine that the weights of each network of the first updated super-network of the kth round of iteration are candidates of the first iteration from the first iteration space when k M is reached.

In some embodiments of the present application, the weight searching module 2551 is further configured to perform difference calculation for a candidate weight of an operator granularity in the search space and a weight of an operator granularity of each network element of the first update super-network of the mth round of iteration to obtain a weight difference, and reject, from the search space, the candidate weight of the operator granularity with the weight difference greater than a difference threshold to obtain the candidate space.

In some embodiments of the present application, the weight searching module 2551 is further configured to segment the region of the object of interest of the training image data through a first initial super-network of a kth iteration to obtain a first segmented region, and update model parameters of the first initial super-network of the kth iteration by using the first segmented region and a loss value of the object of interest between labeling regions in the training image data to obtain a first temporary super-network of the kth iteration.

In some embodiments of the present application, the weight searching module 2551 is further configured to segment the region of the object of interest of the verification image data through the first temporary super-network of the kth round of iteration to obtain a second segmented region, and update the weight of the network granularity of each network element of the first temporary super-network of the kth round of iteration by using the second segmented region and the loss value of the object of interest between the labeling regions in the verification image data to obtain the first intermediate super-network of the kth round of iteration.

In some embodiments of the present application, the weight searching module 2551 is further configured to segment the region of the object of interest of the verification image data through the first intermediate super-network of the kth round of iteration to obtain a third segment region, and update the weight of the operator granularity of each network element of the first intermediate super-network of the kth round of iteration by using the third segment region and the loss value of the object of interest between the labeling regions in the verification image data to obtain the first updated super-network of the kth round of iteration.

In some embodiments of the present application, the search dataset includes training image data and verification image data, the weight search module 2551 is further configured to update a model parameter of a second initial super-network of an ith round of iteration based on the training image data to obtain the second temporary super-network of the ith round of iteration, i is a positive integer, the second initial super-network of the 1 st round of iteration is a target super-network, update a weight of a network granularity of each network element of the second temporary super-network of the ith round of iteration based on the verification image data to obtain a second intermediate super-network of the ith round of iteration, update a weight of an operator granularity of each network element of the second intermediate super-network of the ith round of iteration based on the verification image data to obtain a second updated super-network of the ith round of iteration, and use the second updated super-network of the ith round of iteration as a second initial updated network of the i+1 th round of iteration, and determine that the weight of each network element of the second updated super-network of the ith round of iteration of the N round of iteration is the first weight of the second operator network of the second iteration per round of iteration.

In some embodiments of the present application, the model searching device 255 further includes a dataset construction module 2555, the dataset construction module 2555 is configured to acquire image datasets of a plurality of domains from before searching for the first weights of the network granularity and the second weights of the operator granularity for the plurality of network elements in the target super network in the search space, respectively, based on the search dataset, determine a corresponding extraction ratio for the image dataset of each domain based on a preset probability distribution, extract image data to be mixed of each domain from the image dataset of each domain according to the extraction ratio, and integrate the image data to be mixed of the plurality of domains into the search dataset.

In some embodiments of the present application, the operator generating module 2553 is further configured to fuse output layers of a plurality of operation operators based on a plurality of the second weights, to obtain a target operator corresponding to each of the target network units of the target network structure, where the plurality of operation operators include at least a horizontal fusion operator, an upsampling operator, and a downsampling operator, where the horizontal fusion operator is an operator with a size of an input feature and a size of an output feature being the same, and the upsampling operator is an operator with a size of the input feature being smaller than a size of the output feature, and the downsampling operator is an operator with a size of the input feature being smaller than a size of the output feature;

The model construction module 2554 is further configured to add the target operator to each of the target network elements of the target network structure, obtain a target network model, and complete a model search.

In some embodiments of the present application, the model searching device 255 further includes an image segmentation module 2556, and the image segmentation module 2556 is configured to construct a target network model by using the target operator and each target network element, and segment, after the model searching is completed, a region of interest where the object of interest is located from the acquired medical image data by using the target network model.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the model searching method according to the embodiment of the present application.

Embodiments of the present application provide a computer-readable storage medium storing executable instructions that, when executed by a processor, cause the processor to perform a model search method provided by embodiments of the present application, for example, a model search method as shown in fig. 5.

In some embodiments, the computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM, or various devices including one or any combination of the above.

In some embodiments, the executable instructions may be in the form of programs, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and they may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

As an example, the executable instructions may, but need not, correspond to files in a file system, may be stored as part of a file that holds other programs or data, for example, in one or more scripts stored in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

As an example, the executable instructions may be deployed to be executed on one computing device (an implementation of a model search device), or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

In summary, through the embodiment of the application, the model searching device can search and obtain the first weight of the network granularity and the second weight of the operator granularity for each network unit in the target super network at the same time, and determine the target network unit required by the network model through the first weight so as to connect and obtain the external network structure, namely determine the target network structure, and construct the target operator of each network unit through the second weight and different operation operators, thereby realizing the simultaneous search of the network level and the operator level during the model searching, so that the structure of the network model obtained by the search is more diversified, thereby improving the performance of the model searching, and the candidate space can be determined from the search space by utilizing less time, namely the scale of the search space is effectively reduced by utilizing shorter time, so that the first weight and the second weight can be directly searched from the candidate space with smaller scale for the target super network, the search efficiency is effectively improved, and the image data set in a plurality of different fields can be mixed to obtain the image data sets in the search data sets, thereby further overcoming the characteristics of the search data sets in the search field, and further overcoming the characteristics of the best model can be fully found.

The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A model search method, the method comprising:

Performing iterative updating on a sparse super-network constructed based on a search space based on a search dataset comprising image data, and performing difference calculation on candidate weights of operator granularity in the search space and weights of operator granularity of each network unit of the first updated super-network obtained by iteration when the first total number of iterations is reached, so as to obtain weight differences;

removing candidate weights of the operator granularity, the weight difference of which is larger than a difference threshold value, from the search space to obtain a candidate space;

the method comprises the steps of carrying out iterative updating on a target super-network through a searching dataset comprising image data, and searching a first weight of network granularity and a plurality of second weights of operator granularity for each network unit of the target super-network in the candidate space respectively, wherein the network granularity is the weight granularity influencing the external structure of a searching model, the operator granularity is the weight granularity influencing operators in the searching model, and the target super-network is a set formed by all candidate network structures;

And constructing a target network model for the image processing task by using the target operator and each target network unit, and completing model searching.

2. The method of claim 1, wherein the step of determining the position of the substrate comprises,

The depth and width of the target super network are greater than the depth and width of the sparse super network.

3. The method of claim 1, wherein the search dataset comprises training image data and verification image data, wherein the iteratively updating the sparse super network constructed based on the search space comprises:

updating model parameters of a first initial super-network of a kth iteration based on the training image data to obtain a first temporary super-network of the kth iteration, wherein k is a positive integer, and the first initial super-network of the 1 st iteration is the sparse super-network;

Updating the weight of the network granularity of each network unit of the first temporary super-network of the kth iteration based on the verification image data to obtain a first intermediate super-network of the kth iteration;

And updating the weight of the operator granularity of each network unit of the first intermediate super network of the kth iteration based on the verification image data to obtain the first updated super network of the kth iteration, and taking the first updated super network of the kth iteration as the first initial super network of the (k+1) th iteration.

4. A method according to claim 3, wherein updating the model parameters of the first initial super-network of the kth iteration based on the training image data to obtain the first temporary super-network of the kth iteration comprises:

Performing region segmentation of the object of interest on the training image data through a first initial super network iterated in a kth round to obtain a first segmentation region;

And updating the model parameters of the first initial super-network of the kth iteration by using the first segmentation region and the loss value of the interest object between the labeling regions in the training image data to obtain the first temporary super-network of the kth iteration.

5. A method according to claim 3, wherein updating the weights of the network granularity of each network element of the first temporary super-network of the kth round of iterations based on the verification image data, to obtain the first intermediate super-network of the kth round of iterations, comprises:

performing region segmentation of the object of interest on the verification image data through a first temporary super network iterated in a kth round to obtain a second segmented region;

And updating the weight of the network granularity of each network unit of the first temporary super-network of the kth iteration by using the second segmentation region and the loss value of the interest object between the labeling regions in the verification image data to obtain the first intermediate super-network of the kth iteration.

6. A method according to claim 3, wherein updating the weights of the operator granularity of each network element of the first intermediate super-network of the kth round of iterations based on the verification image data, results in the first updated super-network of the kth round of iterations, comprising:

Performing region segmentation of the object of interest on the verification image data through a first intermediate super network iterated in a kth round to obtain a third segmentation region;

And updating the weight of the operator granularity of each network unit of the first intermediate super-network of the kth iteration by using the third segmentation region and the loss value of the interest object between the labeling regions in the verification image data to obtain the first updated super-network of the kth iteration.

7. The method of any one of claims 2 to 6, wherein the search dataset includes training image data and verification image data, the method further comprising:

updating model parameters of a second initial super-network of the ith round of iteration based on the training image data to obtain a second temporary super-network of the ith round of iteration, wherein i is a positive integer, and the second initial super-network of the 1 st round of iteration is a target super-network;

updating the weight of the network granularity of each network unit of the second temporary super-network of the ith round of iteration based on the verification image data to obtain a second intermediate super-network of the ith round of iteration;

Updating the weight of the operator granularity of each network unit of the second intermediate super network of the ith round of iteration based on the verification image data to obtain a second updated super network of the ith round of iteration, and taking the second updated super network of the ith round of iteration as a second initial updated network of the (i+1) th round of iteration;

when i reaches N, determining the weight of each network unit network granularity of the second updating super network of the nth iteration as the first weight, and determining the weight of each network unit operator granularity of the second updating super network of the nth iteration as the second weight, wherein N is the total number of second iterations.

8. The method according to any one of claims 1 to 6, wherein before iteratively updating the sparse super network constructed based on the search space, the method further comprises:

Acquiring image datasets of a plurality of fields;

based on a preset probability distribution, determining a corresponding extraction proportion for the image dataset of each field;

Extracting image data to be mixed of each field from the image data set of each field according to the extraction proportion;

integrating the image data to be mixed in a plurality of fields into a searching data set comprising the image data.

9. The method according to any one of claims 1 to 6, wherein generating a target operator corresponding to each of the target network elements of the target network structure based on the plurality of second weights and the plurality of operation operators comprises:

Based on the second weights, fusing output layers of the operation operators to obtain a target operator corresponding to each target network unit of the target network structure, wherein the operation operators at least comprise a horizontal fusion operator, an up-sampling operator and a down-sampling operator;

The horizontal fusion operator is an operator with the same size of the input feature and the same size of the output feature, the up-sampling operator is an operator with the size of the input feature smaller than the size of the output feature, and the down-sampling operator is an operator with the size of the input feature smaller than the size of the output feature;

constructing a target network model for an image processing task by using the target operator and each target network unit, and completing model searching, wherein the method comprises the following steps:

And adding the target operator into each target network unit of the target network structure to obtain a target network model for the image processing task, and completing model searching.

10. The method according to any one of claims 1 to 6, wherein said constructing a target network model for an image processing task using said target operator and each of said target network elements, after completing a model search, further comprises:

And segmenting an interest area where the interest object is located from the acquired medical image data by utilizing the target network model.

11. A model search apparatus, the apparatus comprising:

The weight searching module is used for carrying out iterative updating on the sparse super network constructed based on the search space based on the search dataset comprising the image data, and carrying out difference calculation on candidate weights of operator granularity in the search space and the weights of operator granularity of each network unit of the first updated super network obtained by iteration when the first iterative total times are reached to obtain weight differences; iteratively updating a target super-network through a search dataset comprising image data, and searching a first weight of network granularity and a plurality of second weights of operator granularity for each network unit of the target super-network in the candidate space respectively, wherein the network granularity is the weight granularity influencing the external structure of a search model, the operator granularity is the weight granularity influencing operators in the search model, and the target super-network is a set formed by all candidate network structures;

And the model construction module is used for constructing a target network model for the image processing task by utilizing the target operator and each target network unit to finish model searching.

12. A model search apparatus, characterized in that the model search apparatus comprises:

a memory for storing executable instructions;

a processor for implementing the model search method of any one of claims 1 to 10 when executing executable instructions stored in the memory.

13. A computer readable storage medium storing executable instructions which when executed by a processor implement the model search method of any one of claims 1 to 10.

14. A computer program product comprising a computer program or instructions which, when executed by a processor, implements the model search method of any one of claims 1 to 10.