CN118096498B - Image generation method and electronic device - Google Patents

Abstract

An image generation method and an electronic device relate to the technical field of neural network models. In a first time interval, the electronic device uses the second feature vector of the image data to be processed as input, runs the control module, and obtains the first feature map of the image data to be processed. In a second time interval, the electronic device uses the first feature vector of the demand text corresponding to the image data to be processed as input, runs the first encoder, and obtains the potential code of the demand text. There is an overlapping time interval between the second time interval and the first time interval. The electronic device uses the potential code and the first feature map as input, runs the first decoder, and generates target image data. In this way, the electronic device can run the control module and the first encoder in parallel, shortening the time consumption of the process of the electronic device generating the target image data and improving the generation efficiency of the target image data.

Description

Image generation method and electronic device

Technical Field

The embodiments of the present application relate to a neural network model, and more particularly to an image generation method and an electronic device.

Background Art

AI Generated Content (AIGC) technology can be used to generate text, images or videos. In the fields of painting, design, film and television, AIGC technology can automatically generate image data based on user input requirements, reducing manual production costs.

In the existing technology, the use of AIGC technology to generate image data (also known as AI painting) takes a lot of time and cannot meet the user's high efficiency needs.

Summary of the invention

An embodiment of the present application provides an image generation method and an electronic device, wherein the electronic device runs a control module in a control network and an encoder of a U-type network in an SD model in parallel, thereby shortening the time consumed by the electronic device in generating images using the control network, improving the efficiency of the electronic device in generating images, and meeting the user's high-efficiency requirements.

To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

In a first aspect, an embodiment of the present application provides an image generation method, which is applied to an electronic device, wherein a control module and a stable diffusion SD model are configured in the electronic device, and the SD model includes a first encoder and a first decoder.

The image generation method includes: in a first time interval, the electronic device uses the second feature vector of the image data to be processed as input, runs the control module, and obtains the first feature map of the image data to be processed. In a second time interval, the electronic device uses the first feature vector of the requirement text corresponding to the image data to be processed as input, runs the first encoder of the SD model, and obtains the potential code of the requirement text. The requirement text can be used to describe the content requirements and/or quality requirements of the target image data, and there is an overlapping time interval between the second time interval and the first time interval. In this way, the electronic device can run the control module and the first encoder in parallel in the overlapping time interval. Then, the electronic device uses the potential code and the first feature map as input, runs the first decoder of the SD model, and generates the target image data. For example, the requirement text may include "a student graduation photo" to indicate the content of the target image data, or the requirement text may include "a clear student graduation photo" to indicate the content and quality of the target image data.

The image data to be processed includes image data that needs to be processed by the control module and the SD model. For example, the image data to be processed may be the initial image data collected by the camera in the electronic device, or the image data selected by the user from the gallery of the electronic device, or the authorized image data obtained (or received) by the electronic device from a third-party platform. The third-party platform includes but is not limited to other electronic devices that are connected to the electronic device for communication.

The image data to be processed and the required text are a set of input data. The image data to be processed and the required text corresponding to the image data to be processed are both input data used in the process of the electronic device executing the image data generation method. The electronic device also includes a text encoder and/or a variational automatic VAE encoder. The electronic device takes the required text as input and runs the text encoder to obtain a first feature vector of the required text. The electronic device takes the image data to be processed as input and runs the VAE encoder to obtain a second feature vector of the image data to be processed.

In a possible implementation, the electronic device takes the first feature vector of the demand text and the second feature vector of the image data to be processed as input, runs the control module, and obtains the first feature map of the image data to be processed. In this way, the first feature map can include the content carried in the demand text, so that the image feature information carried in the first feature map is richer.

By adopting the above method, the electronic device can simultaneously run the control module and the first encoder within a certain overlapping time interval. In this way, compared with the electronic device running the control module and the first encoder sequentially, the time consumed by the electronic device to run the control module and the first encoder can be reduced, thereby shortening the time consumed by the electronic device to generate target image data and improving the generation efficiency of target image data.

In a possible implementation of the first aspect, the SD model includes a U-type network and a VAE decoder, and the U-type network includes an encoder and a decoder. The first decoder includes a first sub-decoder and a second sub-decoder. The first encoder may indicate an encoder of the U-type network, the first sub-decoder may indicate a decoder of the U-type network in the SD model, and the second sub-decoder may indicate a VAE decoder in the SD model.

Specifically, the electronic device takes the potential code and the first feature map as input, runs the first sub-decoder to obtain the second feature map, and takes the second feature map as input, runs the second sub-decoder to generate target image data.

In a possible implementation of the first aspect, the electronic device takes the first feature vector of the requirement text corresponding to the image data to be processed as input, runs the first encoder, and obtains the potential code of the requirement text. The method also includes: the electronic device obtains a flag parameter, and the flag parameter is used to indicate the operating state of the control module. Among them, the operating state includes the end of operation and the end of operation. In this way, the electronic device can determine the operating state of the control module according to the flag parameter after the first encoder ends. In response to the flag parameter indicating that the operating state of the control module is the end of operation, the electronic device can use the potential code output by the first encoder and the first feature map output by the control module as input data, run the first decoder, and generate target image data.

With the above implementation, the electronic device can obtain the flag parameter after the first encoder is finished. If the flag parameter indicates that the operation state of the control module is finished, the electronic device can run the first decoder to process the potential code output by the first encoder and the first feature map output by the control module to obtain the target image data output by the first decoder. In this way, the electronic device can only run the first decoder after the first encoder and the control module have finished running, so that the first decoder can correctly receive the potential code and the first feature map and process them, thereby obtaining the target image data.

In a possible implementation, the flag parameter can be stored in the memory of the electronic device, so that the electronic device can obtain the flag parameter from the memory at any time after the first encoder is finished. It can be understood that before each execution of the process of the image generation method, the flag parameter stored in the memory of the electronic device indicates that the operation status of the control module is not completed, and only after the control module is completed, the flag parameter will be updated to indicate that the operation status of the control module is completed, so that the electronic device can determine the operation status of the control module in a timely and accurate manner.

In a possible implementation of the first aspect, the electronic device may periodically obtain a flag parameter, and each time the electronic device obtains the flag parameter, the method further includes: in response to the flag parameter indicating that the operation state of the control module is not completed, the electronic device does not run the first decoder, and the electronic device needs to wait for the operation of the control module to end. In response to the flag parameter indicating that the operation state of the control module is completed, the electronic device uses the potential code and the first feature map as input, runs the first decoder, and generates target image data.

It is understandable that when the flag bit parameter obtained by the electronic device indicates that the operation status of the control module is the end of operation, the electronic device may no longer continue to obtain the flag bit parameter.

Specifically, in general, the operation time of the first encoder in the SD model is greater than the operation time of the control module. Therefore, when the electronic device obtains the flag parameter for the first time, the flag parameter has indicated that the operation state of the control module is the end of operation. In this case, the electronic device may no longer continue to obtain the flag parameter, and the electronic device may use the potential code and the first feature map as input, run the first decoder, and generate target image data.

By adopting the above implementation, the electronic device periodically obtains the flag parameter, can determine the state of the control module according to the flag parameter, and promptly runs the first decoder when the operation state of the control module is updated to the end of operation, processes the potential code and the first feature map, and obtains the target image data. When the operation state of the control module is not yet completed, the electronic device does not run the first decoder. In this way, the first decoder can run after correctly receiving the potential code output by the first encoder and the first feature map output by the control module.

In a possible implementation of the first aspect, before the first time interval, the flag parameter in the electronic device indicates that the operation state of the control module is not completed. In this way, before the control module ends, the indication information of the flag parameter can accurately indicate that the control module has not ended, and the electronic device will not run the first decoder. In addition, after the electronic device runs the control module and obtains the first feature map of the image data to be processed, the electronic device can determine that the control module has ended. At this time, the electronic device can update the flag parameter to indicate that the operation state of the control module is ended.

By adopting the above implementation, the flag parameter in the electronic device will be updated to the end of operation only after the control module ends. In this way, after the first encoder ends, the electronic device will only run the first decoder when the control module ends, avoiding the first decoder in the electronic device from being unable to accurately receive the first characteristic map output by the control module due to an error in the indication of the flag parameter. In other words, by adopting the above method, the first decoder can be operated after correctly receiving the first characteristic map output by the control module.

In a possible implementation manner of the first aspect, after the electronic device runs the first decoder to generate target image data, the method further includes: the electronic device updates the flag bit parameter to indicate that the operating state of the control module is not completed.

By adopting the above implementation, the electronic device can update the flag parameter in time after the target image data is generated, so that the flag parameter indicates that the operation status of the control module is not completed. This avoids the error situation that when the electronic device reads the flag parameter again, if the control module has not completed the operation, the electronic device reads the flag parameter indicating that the operation status of the control module is completed.

In a possible implementation of the first aspect, in a first time interval, the electronic device uses the second feature vector of the image data to be processed as input, runs the control module, and obtains the first feature map of the image data to be processed. The method also includes: the electronic device runs a camera application in the electronic device in the foreground, and the camera of the electronic device collects the image data to be processed. In this way, in the first time interval, the electronic device can use the second feature vector of the image data to be processed collected by the camera as input, run the control module, and obtain the first feature map of the image data to be processed. In the second time interval, the electronic device uses the first feature vector of the demand text corresponding to the image data to be processed as input, runs the first encoder, and obtains the potential code of the demand text. The electronic device uses the potential code and the first feature map as input, runs the first decoder, and generates the target image data. Then, the electronic device displays a first interface, and the first interface includes a shooting preview area, and the shooting preview area includes the target image data.

With the above implementation, when the electronic device runs the camera application in the foreground, the camera collects the image to be processed. The electronic device can use the control module and the SD model to process the image to be processed collected by the camera, generate target image data corresponding to the image data to be processed and the required text, and display it through the first interface. In this way, the electronic device can improve the efficiency of generating the target image data by running the control module and the first encoder in parallel, so as to display the target image data in real time in the shooting preview display interface of the camera application, thereby improving the efficiency of shooting of the electronic device.

In a possible implementation of the first aspect, the electronic device includes a plurality of target texts and a plurality of target feature vectors corresponding to the plurality of target texts. The plurality of target texts and the plurality of target feature vectors can be stored in a memory of the electronic device. The target feature vector corresponding to each target text is a feature vector obtained after the electronic device runs a text encoder and performs feature extraction processing on each target text.

In the second time interval, the electronic device uses the first feature vector of the requirement text corresponding to the image data to be processed as input, runs the first encoder, and obtains the potential encoding of the requirement text. The method also includes: the electronic device receives the user's setting operation, and can determine the requirement text that matches the user's setting operation from multiple target texts, as well as the first feature vector corresponding to the requirement text. Among them, the user's setting operation may include the content requirements and/or quality requirements of the target image data set by the user through clicking, sliding, dragging, etc. on the display interface of the camera application in the electronic device. Alternatively, the user's setting operation also includes the option of the content requirements or quality requirements for the target image data selected by the user from multiple target text options of the electronic device before the electronic device runs the camera application in the foreground.

Exemplarily, the multiple target texts include texts such as "beautify", "whiten", "thin face", and "remove freckles". Then the electronic device stores texts such as "whiten", "thin face", "clear, bright", and the feature vector corresponding to "whiten", the feature vector corresponding to "thin face", and the feature vector corresponding to "clear and bright". If the electronic device receives a user setting operation of clicking and selecting the "thin face" option on the display interface of the camera application, and the user's demand is "thin face", the electronic device can use "thin face" in the target text as the demand text and the feature vector corresponding to "thin face" as the first feature vector.

Using the above implementation method, the electronic device pre-stores multiple target texts and multiple feature vectors corresponding to the multiple target texts. After the camera captures the image data, the electronic device can directly determine the required text and the corresponding first feature vector according to the user setting operation, without the need for a text encoder to perform feature extraction processing on the required text, thereby shortening the time for generating target image data and improving the efficiency of the electronic device in generating target image data.

In a possible implementation, the multiple target texts in the electronic device also include a default requirement text, and the multiple target feature vectors corresponding to the multiple target texts also include a feature vector corresponding to the default requirement text. When the electronic device does not receive a setting operation from the user, the electronic device may use the default requirement text in the multiple target texts as the requirement text, and use the feature vector corresponding to the default requirement text as the first feature vector.

Exemplarily, the default requirement text is "clear, bright". When the electronic device does not receive the user's setting operation, the electronic device can use "clear, bright" as the requirement text and the feature vector corresponding to "clear, bright" as the first feature vector.

By adopting the above implementation method, the electronic device can directly obtain the default requirement text and the first feature vector corresponding to the default text without receiving the user's setting operation, and directly process the image data collected by the camera to obtain the target image data without re-obtaining the requirement text, thereby improving the efficiency of generating the target image data.

In a possible line of sight mode, the input data of the control module and the SD model also include iteration number requirements. The electronic device can store multiple iteration number requirements and feature vectors of iteration number requirements corresponding to the multiple iteration number requirements. When the electronic device receives the user's setting operation, it can also determine the iteration number requirement that matches the user's setting operation from the multiple iteration number requirements, and the third feature vector corresponding to the iteration number requirement. The electronic device can use the third feature vector and the second feature vector as input, run the control module, and obtain the first feature map. The electronic device can use the third feature limit and the first feature vector as input, run the first encoder, and obtain the potential code.

In a possible implementation manner of the first aspect, the electronic device includes a first processor, a second processor, and a third processor, the control module is deployed on the third processor, and the SD model is deployed on the second processor.

Before the first time interval, the method also includes: the first processor generates a first control instruction and a second control instruction, the first control instruction includes a first feature vector of the requirement text corresponding to the image data to be processed, and the second control instruction includes a second feature vector of the image data to be processed, the first processor sends the first control instruction to the second processor, and sends the second control instruction to the third processor.

In a first time interval, the electronic device uses the second feature vector of the image data to be processed as input, runs the control module, and obtains a first feature map of the image data to be processed, including: in the first time interval, the third processor uses the second feature vector as input based on the second control instruction, runs the control module, and obtains the first feature map of the image data to be processed.

In the second time interval, the electronic device takes the first feature vector of the required text corresponding to the image data to be processed as input, runs the first encoder, and obtains the latent code of the required text, including: in the second time interval, the second processor takes the first feature vector as input based on the first control instruction, runs the first encoder, and obtains the latent code of the required text.

The electronic device uses the potential code and the first feature map as input, runs the first decoder, and generates target image data, including: the second processor obtains the first feature map, uses the potential code and the first feature map as input, runs the first decoder, and generates target image data. The first processor may include a CPU. The second processor may include a GPU, and the third processor may include an NPU. Alternatively, the second processor may include an NPU, and the third processor may include a GPU.

Specifically, when the third processor stores the first characteristic map after the control module is finished running, the second processor can obtain the first characteristic map from the third processor. After the third processor finishes running the control module, the first characteristic map can also be stored in the memory, in which case the second processor can obtain the first characteristic map from the memory.

By adopting the above implementation method, the electronic device can deploy the control module and the SD model on different processors, and control the operating status of the second processor and the third processor through the first processor, so as to realize the parallel operation of the encoder of the U-type network in the SD model on the second processor and the control module on the third processor, thereby reducing the time consumption of the process of the electronic device generating target image data and improving the efficiency of the electronic device in generating target image data.

In a possible implementation of the first aspect, after obtaining the first feature map of the image data to be processed, the method further includes: in response to the control module running out of operation, the first processor updates the flag parameter to indicate that the operating state of the control module is running out of operation.

After obtaining the latent code of the requirement text, the method further includes: a second processor obtains a flag parameter, the flag parameter is used to indicate the operation status of the control module, and the operation status includes operation completion and non-operation completion.

The second processor obtains the first feature map, takes the potential code and the first feature map as input, runs the first decoder, and generates target image data, including: in response to the flag parameter indicating that the operating status of the control module is ended, the second processor obtains the first feature map, takes the potential code and the first feature map as input, runs the first decoder, and generates target image data.

After generating the target image data, the method further includes: the first processor updates the flag parameter to indicate that the operation status of the control module is not completed.

Specifically, when the operation of the control module ends, the first processor in the electronic device promptly updates the flag parameter to indicate that the operation status of the control module is not completed. After the operation of the first encoder ends, the second processor obtains the flag parameter, and the flag parameter can indicate the operation status of the control module. If the flag parameter indicates that the operation status of the control module is not completed, the second processor does not run the first decoder. If the flag parameter indicates that the operation status of the control module is completed, the second processor runs the first decoder. After the second processor generates the target image data, the first processor can update the flag parameter to indicate that the operation status of the control module is not completed.

With the above implementation, when the encoder of the U-type network in the SD model ends its operation, the second processor promptly obtains the flag parameter to determine the operation state of the control module. When the flag parameter indicates that the operation state of the control module is at the end of operation, the second processor promptly responds, runs the first decoder, processes the first feature map output by the control module and the potential code output by the encoder of the U-type network, and obtains the target image data. When the flag parameter indicates that the operation state of the control module is not at the end of operation, the second processor does not run the first decoder. The first decoder is allowed to run after correctly receiving the first feature map output by the control module at the end of operation and the potential code output by the encoder of the U-type network.

In a second aspect, an embodiment of the present application provides an electronic device, which includes a memory and one or more processors, and the memory is coupled to the processor; computer program code is stored in the memory, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the method in the above-mentioned first aspect and any possible design method thereof.

Specifically, the one or more processors may be a first processor, a second processor, and a third processor.

In a third aspect, an embodiment of the present application further provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed on an electronic device, the electronic device executes the method in the first aspect and any possible design thereof.

In a fourth aspect, an embodiment of the present application further provides a computer program product, including computer instructions, which, when the computer program product runs on an electronic device, enables the electronic device to execute the method in the first aspect and any possible design thereof.

It can be understood that the beneficial effects that can be achieved by the electronic device, computer-readable storage medium, and computer program product provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an AI painting scene provided in an embodiment of the present application;

FIG2 is one of the schematic diagrams of an image generation scenario provided in an embodiment of the present application;

FIG3 is one of the flowcharts of a control network provided in an embodiment of the present application;

FIG4 is a second schematic diagram of a flow chart of a control network provided in an embodiment of the present application;

FIG5 is a second schematic diagram of an image generation scenario provided in an embodiment of the present application;

FIG6 is a schematic diagram of a control network provided in an embodiment of the present application;

FIG7 is a schematic diagram of the structure of a control network provided in an embodiment of the present application;

FIG8 is a hardware structure diagram of an electronic device provided in an embodiment of the present application;

FIG9 is a software architecture diagram of an electronic device provided in an embodiment of the present application;

FIG. 10 is an implementation interaction diagram of an image generation method provided in an embodiment of the present application.

DETAILED DESCRIPTION

The technical solution in the embodiment of the present application will be described below in conjunction with the drawings in the embodiment of the present application. In the description of the present application, unless otherwise specified, "/" means or, for example, A/B can mean A or B; "and/or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like are only used for descriptive purposes, and do not limit the quantity and execution order, nor can they be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present embodiment, unless otherwise specified, the meaning of "multiple" is two or more. And the words "first", "second" and the like are not necessarily different. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present embodiment, unless otherwise specified, "at least one" means one or more, and the meaning of "multiple" is two or more.

In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

AI Generated Content (AIGC) technology is a technology that uses artificial intelligence to generate content related to user needs. It can generate content related to needs based on the information contained in the input data. Among them, the types of content generated by AIGC technology include but are not limited to text, images or videos. Exemplarily, the user can enter a short demand text, and AIGC technology analyzes the demand text to generate a picture corresponding to the demand text. For example, the user enters "output an image of a female character", and the AI generated content technology can output an image containing a female character.

It should be noted that the actual input data of AIGC technology only includes text type. When the input data type is non-text type (such as picture, audio or video), the text extraction model can extract the required text from the input data of picture, audio or video type and input it into AIGC technology. AIGC technology can analyze the required text and generate a picture corresponding to the required text. For example, when the input data type is a picture, the text extraction model can extract the text "a female character image" from the picture as the required text. AIGC technology analyzes "a female character image" and outputs an image data containing a female character.

The process of generating image data by AIGC technology can also be called AI painting. AI painting is usually implemented using machine learning models, including stable diffusion (SD) models. The SD model includes a text encoder, a U-type network (Unet), and a variational automatic decoder (VAE decoder). The input data of the SD model includes the demand text. Among them, the text encoder can perform feature extraction processing on the demand text to obtain a feature vector 1 corresponding to the demand text. Unet is used to perform iterative noise reduction processing on the feature vector 1, and further extract features to obtain a feature vector 2 corresponding to the demand text. The VAE decoder is used to convert the feature vector 2 into image data 1. In this way, the SD model can output image data 1 corresponding to the demand text.

Referring to FIG1 , the user inputs the requirement text “Draw a student graduation photo” into the SD model, and the text encoder performs feature extraction processing on the requirement text “Draw a student graduation photo” and outputs feature vector 1. The U-shaped network performs iterative noise reduction processing on feature vector 1 to obtain feature vector 2. The VAE decoder processes feature vector 2 to obtain image data 101 containing a student graduation scene.

However, the generation of image data 1 by the SD model is based on the extraction of feature vector 1 of the requirement text, and the requirement text can only indicate the content that image data 1 should contain, but cannot indicate image features (for example, three-channel RGB values, texture, etc.). In other words, although the image data 1 generated by the SD model can be close to the information contained in the requirement text at the content level, the quality of image data 1 is not high and cannot fully match the user's needs. Continuing with the example of Figure 1 above, image data 101 contains relevant content of the "student" feature in the requirement text "Draw a student graduation photo" (such as the character image contained in Figure 1), but image data 101 does not include information related to "graduation" and does not match the "graduation" feature in the requirement text "Draw a student graduation photo".

In view of the above problems, a control network (ControlNet) is proposed in some implementations. ControlNet adds a variational autoencoder (VAE) encoder and a control (Control) module on the basis of the SD model architecture. The input data of the VAE encoder is image data 2, and the VAE encoder performs feature extraction processing on image data 2 to obtain feature vector 3 and input it into the Control module. The Control module can downsample feature vector 1 and feature vector 3 to obtain a feature map, which can also be called feature vector 4 or a first feature map, and input feature vector 4 into the SD model in ControlNet. The SD model in ControlNet can optimize feature vector 4 according to the requirement text to obtain image data 3 that matches the requirement text. In this way, the user can control the content of the input data (i.e., image data 2) of the Control module to adjust the feature vector 4 output by the Control module, thereby adjusting the content of the image data 3 generated by ControlNet. In addition, ControlNet can extract image feature information from image data 2 to generate image data 3 containing the above image feature information. In this way, compared with the image data 1 generated by the SD model based only on the requirement text, the image data 3 generated by ControlNet has higher quality.

Referring to FIG. 2 , the user inputs the demand text “Draw a student graduation photo” into the SD model in the control network, inputs the image data 201 into the VAE encoder in the control network, and the feature vector 1 output by the text encoder in the SD model and the feature vector 3 output by the VAE encoder are input into the control module together. The control module inputs the feature vector 4 into the SD model in the control network. The U-shaped network of the SD model in the control network outputs the feature vector 2 based on the feature vector 1 and feature vector 4 corresponding to “Draw a student graduation photo”, and the VAE decoder of the SD model generates the image data 202 based on the feature vector 2. The image data 202 includes the image feature information carried by the image data 201 (such as the university buildings and the natural scenery such as the sun, clouds, etc. and the texture information contained in the image data 201), that is, the image data 202 contains more content related to the “graduation” feature in the demand text. Compared with the image data 101 shown in FIG. 1 , the image data 202 is more matched with the demand text “Draw a student graduation photo”.

Specifically, Unet in ControlNet includes two parts: an encoder and a decoder. The encoder is used to downsample feature vector 1 to obtain a latent variable (latentcode) corresponding to the required text. The latent variable can also be called feature vector 5, or can also be called potential code. After the Control module downsamples feature vector 3 to obtain the corresponding feature vector 4, the decoder can receive feature vector 5 output by the encoder and feature vector 4 output by the Control module, and upsample feature vector 4 and feature vector 5 to obtain feature vector 2. Then, the VAE decoder generates the corresponding image data 3 according to feature vector 2. In the process of ControlNet generating image data 3 according to the required text and image data 2, after the text encoder in the SD model outputs feature vector 1 and the VAE encoder outputs feature vector 3, based on the relationship between the input data of the decoder in Unet and the output data of the Control module, ControlNet should first run the Control module to obtain feature vector 4 and then run Unet in the SD model. In this way, the decoder in Unet can receive the feature vector 5 output by the encoder in Unet and the feature vector 4 output by the Control module after the encoder runs, and generate a feature map (Feature Map), which can also be called feature vector 2 or the second feature map.

Referring to Figure 3, the text encoder in the SD model performs feature extraction on the required text and outputs feature vector 1. The VAE encoder performs feature extraction on the image data 2 and outputs feature vector 3. Then, ControlNet first runs the control module in the control network to obtain feature vector 4, and then inputs feature vector 1 and feature vector 4 into the U-type network. The encoder and decoder in the U-type network and the VAE decoder in the control network are sequentially run to output image data 3.

However, the above-mentioned method of first running the Control module and then running the Unet in the SD model causes ControlNet to take a long time in the iteration phase, which results in a long time for ControlNet to generate image data 3, and cannot meet the user's high timeliness requirements for the shooting function.

For example, the running time of the Control module is 100ms, and the running time of the encoder and decoder in Unet is 150ms. It takes 400ms for ControlNet to run the Control module first and then run Unet, which cannot meet the high efficiency requirements of users.

In view of the above problems, an embodiment of the present application proposes an image generation method. In the process of generating image data 3 according to the required text and image data 2, after the text encoder in the SD model outputs feature vector 1 and the VAE encoder outputs feature vector 3, ControlNet can run the Control module in ControlNet and the encoder in Unet in parallel. After the Control module and the encoder in Unet are finished running, ControlNet runs the decoder in Unet again. The decoder in Unet can receive the feature vector 4 output by the Control module and the feature vector 5 output by the encoder, and generate feature vector 2 according to feature vector 4 and feature vector 5, and the VAE decoder generates the corresponding image data 3 according to feature vector 2. In this way, the time consumption of ControlNet to generate image data 3 can be reduced. For the sake of convenience, the encoder in Unet can also be called the first encoder, the feature vector 1 can also be called the first feature vector, the feature vector 3 can also be called the second feature vector, the image data 2 can also be called the image data to be processed, and the image data 3 can also be called the target image data. The time interval for running the Control module in ControlNet can also be called the first time interval. The time interval for running the encoder in Unet can also be called the second time interval. The time interval for running the Control module in ControlNet and the encoder in Unet in parallel may also be referred to as an overlapping time interval between the first time interval and the second time interval.

Exemplarily, FIG4 provides a flow chart of a control network. As shown in FIG4, the user inputs the demand text and image data 2, the text encoder in the SD model performs feature extraction processing on the demand text to obtain feature vector 1, and the VAE encoder performs feature extraction processing on the image data 2 to obtain feature vector 3, and then the control network runs the encoder and control module in the U-type network in parallel. Among them, the encoder in the U-type network performs downsampling processing on feature vector 1 to obtain feature vector 5, and the control module performs downsampling processing on feature vector 3 to obtain feature vector 4. After the encoder and control module in the U-type network are finished running, the control network runs the decoder in the U-type network. The decoder in the U-type network performs upsampling processing on feature vector 4 and feature vector 5 to obtain feature vector 2. Finally, the VAE decoder in the control network processes feature vector 2 to obtain image data 3.

Generally, the running time of the Control module is shorter than the running time of the encoder in Unet. Therefore, the time consumed by ControlNet to run the Control module and the encoder in parallel can be regarded as the running time of the encoder in Unet. In other words, ControlNet running the Control module and the encoder in parallel can save the running time of the Control module and improve the generation efficiency of ControlNet image data 3.

For example, the running time of the Control module is 100ms, the running time of the encoder and decoder in Unet is 150ms, the parallel running time of the Control module and the encoder is 150ms, and the running time of the decoder in Unet is 150ms. In this way, the total running time of the Control module and Unet is 300ms, and the process of ControlNet generating image data 3 is shortened by 150ms.

Based on ControlNet, it has the ability to generate image data 3 that matches user needs based on the required text and image data 2. In practical applications, ControlNet can also be embedded into electronic devices (such as mobile phones, tablet computers, desktop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), augmented reality (AR) or virtual reality (VR) devices, etc., and the specific form of the electronic device is not particularly limited in the embodiments of the present application) as an image optimization model.

Specifically, the electronic device inputs the initial image data as image data 2 into ControlNet, and inputs the requirement text into ControlNet. The electronic device can use ControlNet to optimize the content of the initial image data according to the requirement text to obtain image data 3. Among them, the initial image data can be the initial image data obtained by the electronic device driving the camera to shoot. Or, the initial image data can be image data input by the user. Alternatively, the initial image data can be image data stored in a gallery in the electronic device. Alternatively, the initial image data can be authorized image data obtained (or received) by the electronic device from a third-party platform. The third-party platform may include other electronic devices that communicate with the electronic device. In this way, the electronic device can generate high-quality image data 3 that matches the user's needs, such as the camera application in the electronic device shoots to generate beauty images, optimizes the image data of the gallery application in the electronic device, or the electronic device generates AI painting images, etc.

It should be understood that in the shooting scene, the electronic device uses ControlNet to optimize the content of the initial image data collected by the camera according to the demand text, and can quickly generate high-quality image data 3 and use it as the photo of the preview interface of the electronic device, thereby improving the efficiency of the electronic device in taking high-quality photos. In particular, when the user zooms in to take photos of scenes with a larger range or farther away, and when the user takes photos of darker scenes, the electronic device uses ControlNet's powerful image generation capabilities to quickly take clearer, brighter, and high-quality photos.

For example, the general requirements of users for taking photos with electronic devices are clarity and brightness. Correspondingly, the electronic device can use clarity and brightness as the default requirement text. In this way, the electronic device uses ControlNet to optimize the initial image data taken by the camera, and outputs clearer and brighter image data 3 as the photo taken by the electronic device preview interface.

Taking a mobile phone as an example, as shown in FIG5 , the camera application in the mobile phone drives the camera to shoot and obtains initial image data 301, which has the problem of being unclear. The control network deployed in the mobile phone optimizes the initial image data 301 based on the "clear and bright" requirement text 302, and can obtain clear and bright image data 303 as the photo displayed on the preview interface of the camera application in the mobile phone.

In some embodiments, after ControlNet is deployed in an electronic device, the Control module and Unet in ControlNet can be deployed on different processors respectively. The electronic device can control two processors to run the Control module in ControlNet and the encoder in Unet in parallel. In this way, the electronic device can generate high-quality images using ControlNet while improving the efficiency of generating images.

Exemplarily, the electronic device deploys the Control module on a neural-network processing unit (NPU) and deploys Unet on a graphics processing unit (GPU). In another exemplary embodiment, the electronic device deploys the Control module on a GPU and deploys Unet on an NPU.

Referring to Figure 6, the electronic device shown in Figure 6 deploys the control module on the NPU and the U-shaped network on the GPU. After the electronic device finishes running the text encoder in the SD model on the GPU and the VAE encoder ends running. The electronic device can control the NPU and the GPU to run the control module and the encoder in the U-shaped network in parallel (the solid arrows shown in Figure 6 indicate the sequential running order). After the encoder in the U-shaped network ends running, the U-shaped network in the GPU pauses running, and after the control module also ends running (as shown by the dotted arrows in Figure 6, the running sequence of pausing after the encoder in the U-shaped network ends running and waiting for the control module to end running before continuing to run), the electronic device controls the GPU to run the decoder in the U-shaped network and the VAE decoder.

Specifically, the electronic device can use a cross-hardware platform deployment model library on the central processing unit (CPU) to implement the operation of controlling the parallel operation of the Control module on the NPU and the Unet on the GPU. Taking the cross-hardware platform deployment model library as NNAdapter as an example, the electronic device uses NNAdapter to send parallel operation control instructions to the NPU and GPU respectively to adapt the logic of the electronic device to run the Control module on the NPU and the Unet on the GPU in parallel. For ease of explanation, the CPU can be called the first processor, the GPU can be called the second processor, the NPU can be called the third processor, or the GPU can be called the third processor and the NPU can be called the second processor.

In some embodiments, the NNAdapter includes a "model manager" parameter, and the "model manager" parameter can be used to control the operating state of the processor. The CPU can set different "model manager" parameters in the NNAdapter, "model manager=GPU" indicates that the controlled processor is a GPU, and "model manager=NPU" indicates that the controlled processor is an NPU. The CPU generates a first control instruction according to the "model manager=GPU" parameter, and generates a second control instruction according to the "model manager=NPU" parameter. The NNAdapter on the CPU sends the first control instruction to the GPU and the second control instruction to the NPU respectively. When the GPU receives the first control instruction, it starts to run Unet, and when the NPU receives the second control instruction, it starts to run the Control module. In this way, the CPU can implement the control operation of the operating state of the two processors, the GPU and the NPU.

Furthermore, the electronic device can set an interrupt waiting mechanism during the operation of Unet, so that the electronic device can enter the interrupt waiting state when the encoder in Unet ends, and the electronic device can run the decoder in Unet after the Control module ends. In addition, the electronic device can also set a flag parameter, which is used to indicate the operating status of the Control module. The flag parameter can include "0" or "1". Among them, when the flag parameter is "0", it indicates that the operating status of the Control module is not completed, and when the flag parameter is "1", it indicates that the end status of the Control module is completed.

It should be understood that after each generation of the image data 3, the electronic device sets the flag parameter to indicate that the running state of the Control module is not completed.

Specifically, in the process of running Unet, the running framework of Unet generally receives input data only once, and then executes each module in Unet in sequence until the Unet runs to the end and outputs the feature vector 2. In order to run the Control module and the encoder in Unet in parallel, the electronic device can set an interrupt waiting mechanism in the running framework of Unet. The running framework of Unet enters the interrupt waiting state after the encoder in Unet runs to the end, and when the control module runs to the end, the feature vector 4 output by the control module and the feature vector 2 output by the encoder in Unet are used as input data to run the decoder in Unet.

Taking the GPU image (Fast Artificial Intelligence Technology by Honor, FAITH) framework as an example, the electronic device can set an interrupt waiting mechanism in the FAITH framework. After the text encoder in the SD model of the electronic device is finished running and the VAE encoder is finished running, the NNAdapter in the electronic device simultaneously sends a first control instruction carrying "model manager = GPU" to the GPU and a second control instruction carrying "model manager = NPU" to the NPU. Then, the GPU controls the FAITH framework to run Unet based on the first control instruction, and the NPU runs the Control module based on the second control instruction. After the encoder in Unet is finished running, the FAITH framework enters the interrupt waiting state, obtains the flag bit parameter, and determines the running state of the Control module according to the flag bit parameter. Among them, the process of GPU running the encoder in Unet and the process of NPU running the Control module have an overlapping time interval. When the Control module runs, the electronic device sets the flag bit parameter to indicate that the running state of the Control module is running. When the flag bit parameter shows that the running state of the Control module is running, the FAITH framework starts running the decoder in Unet, and then the GPU runs the VAE encoder in the SD model, and finally outputs image data 3. The FAITH framework continues to interrupt the waiting state when the flag parameter shows that the running status of the Control module is not completed, until the flag parameter shows that the running of the Control module is completed.

In some embodiments, the electronic device records the operation process of the control network in an offline log.

Exemplarily, the offline log can be used to record the processor that runs each module in the control network and the running time of each module in the control network. For example, the offline log includes that the control module in the control network runs on the NPU and the running time is 10:00:00-10:00:20, the encoder of the U-type network in the control network runs on the GPU and the running time is 10:00:00-10:00:30, and the decoder of the U-type network in the control network runs on the GPU and the running time is 10:00:30-10:01:00.

In some embodiments, the control network further includes an iteration number encoder, and the input data of the control network further includes an iteration number requirement. The iteration number encoder performs feature extraction processing on the iteration number requirement, obtains a feature vector 6 corresponding to the iteration number requirement, and then inputs it into the SD model and the control module. For ease of explanation, the feature vector 6 may also be referred to as a third feature vector.

In some embodiments, the input data of the control module in the control network also includes random variables (random), and the control module performs downsampling processing on the random variables to obtain a feature vector 2 containing richer image feature information. In other words, inputting the random variables into the control module in the control network can further improve the quality of the image data 3 generated by the control network.

Referring to FIG. 7 , an embodiment of the present application provides a structural diagram of a control network. The control network includes an SD model and a control module. The input data of the SD model includes a requirement text (Prompt) and a number of iterations required (Time). The text encoder in the control network performs feature extraction processing on the requirement text to obtain a corresponding feature vector 1. The VAE encoder performs feature extraction processing on the image data 2 to obtain a feature vector 3 corresponding to the image data 2.

The iteration requirement indicates the number of iterations required by the encoder (Unet encoder) and decoder (Unet decoder) in the U-type network, and the iteration encoder (Time Encoder) in the control network performs feature extraction processing on the iteration requirement to obtain a feature vector 6 corresponding to the iteration number. For example, if the iteration requirement is 20, the encoder and decoder in the U-type network need to iterate 20 times, that is, the U-type network can be denoised after 20 iterations to obtain a feature vector 2.

The encoders in the U-type network include the first encoder of the SD model (SD Encoder Block1), the second encoder of the SD model (SD Encoder Block2), the third encoder of the SD model (SD Encoder Block3), and the fourth encoder of the SD model (SD Encoder Block4). The output data of the first encoder of the SD model is a 64*64 matrix, the output data of the second encoder of the SD model is a 32*32 matrix, the output data of the third encoder of the SD model is a 16*16 matrix, and the output data of the fourth encoder of the SD model is an 8*8 matrix. In this way, the encoders in the U-type network can extract the feature vector 5 through step-by-step noise reduction.

The decoders in the U-shaped network include the SD model first decoder (SD Decoder Block1), the SD model second decoder (SD Decoder Block2), the SD model third decoder (SD Decoder Block3), the SD model fourth decoder (SD Decoder Block4) and the SD model fifth decoder (SD Middle Block). The SD model first encoder corresponds to the SD model first decoder, the SD model second encoder corresponds to the SD model second decoder, the SD model third encoder corresponds to the SD model third decoder, and the SD model fourth encoder corresponds to the SD model fourth decoder, and the output data of each encoder can be used as the input data of the decoder corresponding to it.

The input data of the control module includes feature vector 3 (Condition), feature vector 1 corresponding to the requirement text, and feature vector 6 corresponding to the iteration number requirement. Among them, a zero convolution layer (zero convolution) convolves feature vector 3 to obtain feature vector 7, which is input into the SD model encoder A (SD EncoderBlock A) of the control module. In some possible embodiments, the input data of the control module also includes random variables.

The control module includes an SD model encoder A, an SD model encoder B (SD Encoder Block B), an SD model encoder C (SD Encoder Block C), an SD model encoder D (SD Encoder Block D) and an SD model encoder E (SDMiddle Block). In the control module, the SD model encoder A corresponds to the first decoder of the SD model in the U-type network, the SD model encoder B corresponds to the second decoder of the SD model in the U-type network, the SD model encoder C corresponds to the third decoder of the SD model in the U-type network, the SD model encoder D corresponds to the fourth decoder of the SD model in the U-type network, and the SD model encoder E corresponds to the fifth decoder of the SD model in the U-type network. Among them, the output data of the SD model encoder A is a 64*64 matrix, the output data of the SD model encoder B is a 32*32 matrix, the output data of the SD model encoder C is a 16*16 matrix, the output data of the SD model encoder D is an 8*8 matrix, and the output data of the SD model encoder E is an 8*8 matrix. In this way, the control module can further extract the feature vector 8 corresponding to the feature vector 1, the feature vector 7 and the feature vector 6 through the multi-layer encoder step-by-step noise reduction.

The output data (including feature vector 8) of each encoder in the control module is processed by a zero convolution layer to obtain feature vector 4, which is input into the decoder corresponding to each one in the U-type network as input data.

After the encoder and control module in the U-shaped network finish running, each decoder in the U-shaped network upsamples the feature vector 1 output by each encoder in the U-shaped network corresponding to each other, and the feature vector 4 output by each encoder in the control module corresponding to each other, to obtain the feature vector 2.

Then, the feature vector 2 is decoded by the VAE decoder to obtain the corresponding image data 3.

In this way, the electronic device uses ControlNet to optimize the image data 2 according to the text requirements, and can generate high-quality image data 3 that matches the user's requirements. In addition, the electronic device runs the Control module and the encoder in Unet in parallel, which can shorten the time it takes for the electronic device to use ControlNet to generate the image data 3, and improve the efficiency of generating the image data 3.

The image generation method provided in the present application can be executed on an electronic device including the following hardware and software structures. See Figure 8, which is a hardware structure diagram of an electronic device provided in an embodiment of the present application.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) connector 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a camera module 193, a display screen 191, and a subscriber identification module (SIM) card interface 192. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, an ambient light sensor 180C, and the like.

It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

The processor can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

The processor 110 may also be provided with a memory for storing instructions and data.

In some embodiments, the processor 110 may include one or more interfaces. The processor 110 may be connected to a touch sensor, an audio module, a wireless communication module, a display, a camera and other modules through at least one of the above interfaces. It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The USB connector 130 is an interface that complies with USB standard specifications and can be used to connect the electronic device 100 and peripheral devices. Specifically, it can be a Mini USB connector, a Micro USB connector, a USB Type C connector, etc.

The charging management module 140 is used to receive a charging input from a charger.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 191, the camera module 193, and the wireless communication module 160.

The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

The electronic device 100 can realize the display function through a GPU, a display screen 191, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 191 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

The display screen 191 is used to display images, videos, etc. The display screen 191 includes a display panel.

The electronic device 100 can realize the camera function through the camera module 193, ISP, video codec, GPU, display screen 191, application processor AP, neural network processor NPU, etc.

The camera module 193 can be used to collect color image data and depth data of the photographed object. The ISP can be used to process the color image data collected by the camera module 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera module 193.

In some embodiments, the electronic device 100 may include one or more camera modules 193 .

In some embodiments, the CPU or GPU or NPU in the processor 110 can process the image data collected by the camera module 193. In some embodiments, the NPU can determine the location of the face of the person being photographed through a face recognition algorithm, and then beautify the face image of the person being photographed through ControlNet to generate a beautified face image. In some embodiments, the CPU or GPU or NPU can also be used to confirm the body shape of the person being photographed (such as body proportions, fatness and thinness of body parts between bone points) based on the depth data collected by the camera module 193 (which can be a 3D sensing module) and the identified bone points, and then ControlNet processes the position corresponding to the body of the person being photographed in the captured image, so that the body shape of the person being photographed in the captured image is beautified to obtain a beautified image of the person.

NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as: image generation, image recognition, face recognition, voice recognition, text understanding, etc.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external memory card. Or files such as music and videos are transferred from the electronic device to the external memory card.

The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional methods or data processing of the electronic device 100 by running instructions stored in the internal memory 121, and/or instructions stored in a memory provided in the processor.

The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

The sensor module 180 is used to collect device data or collect user behavior data after authorization by the user, for example, collecting the user's frequently input demand texts such as "slim face" and "slim body".

The key 190 may include a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

The software system in the above electronic device can adopt a layered architecture, an event-driven architecture or a cloud architecture. The embodiment of the present application takes the electronic device as an Android™ system with a layered architecture as an example to illustrate the software structure of the electronic device.

Referring to FIG9 , a software architecture diagram of an electronic device provided in an embodiment of the present application is shown. As shown in FIG9 , the layered architecture can divide the software of the electronic device into several layers, each layer having a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android™ system is divided into four layers, namely, from top to bottom, the application layer (Application layer), the application framework layer, the system library, and the kernel layer.

The application layer may include a variety of application packages, such as calls, text messages, and gallery.

In some embodiments, the application package may also include applications with image generation functions, such as AI painting, camera, beauty camera, etc. Exemplarily, AI painting may receive the required text input by the user, and determine the image data 2 from the image data stored in the gallery in the electronic device according to the user operation. AI painting runs ControlNet to process the image data 2 and the required text, and generates image data 3 that meets the user's requirements.

The camera may use "clear and bright" as the default requirement text and use the initial image data captured by the camera as image data 2. The camera runs ControlNet to generate clearer and brighter image data 3 as the image data captured by the camera.

The beauty camera can determine the corresponding demand text according to different beauty modes, and use the initial image data taken by the camera or the image data stored in the gallery of the electronic device as image data 2. The beauty camera runs ControlNet to generate image data 3 corresponding to the beauty mode as the image data taken by the beauty camera. For example, if the beauty mode is "face thinning", the beauty camera runs ControlNet to optimize the position of the face in image data 2 to achieve the effect of "face thinning", thereby generating the corresponding image data 3 as the image data taken by the beauty camera.

The application framework layer provides an application programming interface (API) and a programming framework for the APP in the application layer. The application framework layer includes some predefined functions. Exemplarily, the application framework layer may include a window manager, a resource manager, a view system, etc.

The system library may include multiple functional modules, such as a 3D graphics processing library (such as OpenGL ES) and a media library.

The kernel layer is the layer between hardware and software. The kernel layer can include various hardware drivers such as display driver, camera driver, audio driver, etc., which are used to drive the corresponding hardware work.

It should be understood that the layering of the software architecture shown in FIG. 9 is merely exemplary, and the software architecture may include more or fewer software layers.

Taking the Control module deployed on the GPU, Unet deployed on the NPU, and the camera application as an example, in some embodiments, combined with the software and hardware structure of the electronic device, this embodiment can implement the image generation method through the steps shown in Figure 10.

S900: The camera application receives operation a.

Specifically, the user performs operation a on the interface of the camera application, where operation a is a shooting operation, and the camera application receives operation a.

S901 . In response to receiving operation a, the camera application sends a first frame request to the camera, where the first frame request is used to instruct the camera to collect image data 2 .

Specifically, the camera application generates a first frame request in response to the received operation a. The first frame request is used to instruct the camera to collect image data 2. The image data 2 can be used as input data of the Control module in ControlNet.

S902: The camera collects image data 2 in response to the first frame request.

Specifically, the camera starts and collects initial image data as image data 2 in response to the first frame request.

S903: The camera sends image data 2 to the camera application.

Specifically, after the camera collects the image data 2 , it sends the image data 2 to the camera application, so that the camera application instructs the CPU to run ControlNet to optimize the image data 2 , thereby obtaining the image data 3 .

S904 : In response to receiving operation a, the camera application reads feature vector 1 and feature vector 6 from the memory.

Specifically, after the user performs operation a, the camera application needs to obtain the requirement text in response to receiving operation a. The electronic device memory can pre-store multiple requirement texts, feature vectors 1 that have a one-to-one correspondence with the multiple requirement texts, multiple iteration number requirements, and feature vectors 6 that have a one-to-one correspondence with the multiple iteration number requirements. In this way, after the camera application receives operation a, it can determine the requirement text parameters and iteration number requirement parameters carried by operation a, and then directly read the corresponding feature vector 1 and feature vector 6 from the memory. For the sake of convenience, the above-mentioned multiple requirement texts can also be referred to as multiple target texts, and the above-mentioned feature vector 1 that has a one-to-one correspondence with the multiple requirement texts can also be referred to as multiple target feature vectors that correspond one-to-one to the multiple target texts.

Specifically, operation a can carry the requirement text parameter and the iteration number requirement parameter set by the user. For example, the user selects the requirement text parameter and the iteration number requirement parameter in the interface of the camera application, or enters the setting requirement text parameter and the iteration number requirement parameter in the interface of the camera application. Among them, the requirement text parameter can be a specific requirement text content or an ID corresponding to the requirement text stored in the memory, and the iteration number requirement parameter can include the value of the iteration number, or the ID corresponding to the corresponding feature vector 6 iteration number requirement. For example, the requirement text parameter is the requirement text "thin face", or the requirement text parameter is "XQ001", which is used to indicate the requirement text "thin face" stored in the memory. For example, the iteration number requirement parameter can include an iteration number requirement of 15, or the iteration number requirement parameter is "DD001", which is used to indicate that the iteration number requirement stored in the memory is 15 times.

Exemplarily, the memory stores feature vectors 1 that correspond one-to-one to the requirement text "thin face" with ID XQ001, the requirement text "enlarge eyes" with ID XQ002, and the requirement text "clear and bright" with ID XQ003. And the memory stores feature vectors 6 that correspond one-to-one to the iteration number requirement "15" with ID DD001, the iteration number requirement "20" with ID DD002, and the iteration number requirement "30" with ID DD003. After the user performs operation a in the camera application interface, operation a carries the requirement text parameter "thin face" and the iteration number requirement parameter with ID "DD001". In response to receiving operation a, the camera application can read feature vector 1 corresponding to the requirement text "thin face" with ID XQ001 and feature vector 6 corresponding to the iteration number requirement "15" with ID DD001 from the memory.

Specifically, the multiple requirement texts stored in the electronic device include a default requirement text, and the multiple iteration number requirements include a default iteration number requirement. When operation a does not carry a requirement text parameter and an iteration number requirement parameter, the camera application can directly read a feature vector 1 having a corresponding relationship with the default requirement text and a feature vector 6 having a corresponding relationship with the default iteration number requirement from the memory.

For example, in general, the user's requirement for taking pictures and videos with a camera application is "clear and bright", so the electronic device can use "clear and bright" as the default requirement text of the camera application. In order to improve the efficiency of image generation, further, the electronic device can perform feature extraction processing on the default requirement text "clear and bright" through the text encoder in the SD model in advance, obtain feature vector 1 and store it in the memory. In this way, when operation a does not carry the requirement text parameter, the camera application can directly read the feature vector 1 from the memory. It can be understood that the default requirement text of the camera application can also be specifically set according to actual needs. The above-mentioned "clear and bright" as the default requirement text of the camera application is only an example, and the default requirement text of the camera application is not specifically limited here. The iteration number requirement of ControlNet can also set a default value. For example, the iteration number requirement is 20 times, and the electronic device uses the iteration number encoder to perform feature extraction processing on the 20 iteration number requirements to obtain feature vector 6 and store it in the memory. In this way, when operation a does not carry the iteration number requirement parameter, the camera application can directly read the feature vector 6 from the memory.

The electronic device stores feature vectors 1 of multiple demand texts in memory in advance. During the shooting process, the camera application can directly read the feature vector 1 matching the user demand from the memory according to the user's operation a, without the need for a text encoder to perform feature extraction processing on the demand text corresponding to operation a, thereby further reducing the time consumption of the electronic device's shooting process.

It is understandable that both step S901 and step S904 are steps executed by the camera application in response to receiving operation a, and the execution order of the above two steps is not limited to the order shown in FIG10. Similarly, the steps associated with steps S901 and S904 are not limited to the order shown in FIG10. For example, in response to receiving operation a, the camera application first executes step S904 and then executes step S901. In this case, the execution order of steps S902-S903 associated with step S901 is still after step S901.

S905 . After receiving feature vector 1 and feature vector 6 , the camera application generates an image generation instruction, where the image generation instruction includes feature vector 1 , feature vector 6 , and image data 2 .

Specifically, after receiving feature vector 1, feature vector 6, and image data 2, the camera application generates a corresponding image generation instruction based on operation a. The image generation instruction includes feature vector 1, feature vector 6, and image data 2. The image generation instruction is used to instruct the CPU to run ControlNet to process feature vector 1, feature vector 6, and image data 2 to obtain corresponding image data 3.

S906: The camera application sends an image generation instruction to the CPU.

Specifically, the camera application sends an image generation instruction to the CPU to instruct the CPU to run ControlNet according to the image generation instruction.

S907 . In response to receiving the image generation instruction, the CPU generates a third control instruction, where the third control instruction includes image data 2 .

Specifically, the CPU starts running ControlNet in response to receiving an image generation instruction, and generates a third control instruction based on the image generation instruction. The third control instruction includes image data 2. The third control instruction is used to control the NPU to run the VAE encoder and optimize the image data 2 to obtain a feature vector 3.

S908. The CPU sends a third control instruction to the NPU.

Specifically, the CPU sends a third control instruction to the NPU.

S909 . In response to receiving the third control instruction, the NPU runs the VAE encoder in the control network.

Specifically, in response to receiving the third control instruction, the NPU runs the VAE encoder in the control network according to the third control instruction, and the VAE encoding establishment thread performs feature extraction processing on the image data 2 to obtain a feature vector 3 corresponding to the image data 2.

It should be understood that the VAE encoder is deployed on the NPU and the VAE decoder is deployed on the GPU as shown in FIG10 . This is merely an example of the deployment of the VAE encoder and VAE decoder and does not constitute a specific limitation on the VAE encoder and VAE decoder.

For example, the VAE encoder can also be deployed on the GPU, and not on the same processor as the control module in the control network. In this case, the GPU needs to send the VAE encoder operation results (including feature vector 3) to the NPU after the VAE encoder is finished running, so that the NPU can use feature vector 3 as input to run the control module in the control network.

S910, VAE encoder operation ends.

Specifically, when the VAE encoder finishes running, the NPU jumps out of the current thread, and the CPU can determine that the VAE encoder has finished running. It can be understood that at this time, the third running result of the VAE encoder (including feature vector 3) is stored in the NPU, and when the NPU runs the control module, the third running result of the VAE encoder can be used as input data of the control module.

S911. The CPU generates a first control instruction and a second control instruction. The first control instruction includes feature vector 1 and feature vector 6, and the second control instruction includes feature vector 1 and feature vector 6.

Specifically, the CPU generates a first control instruction and a second control instruction, the first control instruction includes feature vector 1 and feature vector 6, and is used to instruct the GPU to run the encoder in the U-shaped network in the control network to downsample feature vector 1 and feature vector 6. The second control instruction includes feature vector 1 and feature vector 6, and the second control instruction is used to instruct the NPU to run the control module in the control network to downsample feature vector 1, feature vector 3, and feature vector 6.

S912: The CPU sends a first control instruction to the GPU.

S913: The CPU sends a second control instruction to the NPU.

Specifically, after the CPU sends the first control instruction to the GPU, it sends the second control instruction to the NPU, so that the process of the GPU running the encoder of the U-type network and the process of the NPU running the control module have an overlapping time interval. Compared with the method of running the encoder of the U-type network after the control module is finished running, the operation time of the control network can be reduced, thereby improving the generation efficiency of the image data 3.

Step S912 and step S913 are both steps executed after the CPU generates the first control instruction and the second control instruction, and the execution order of the above two steps is not limited to the order shown in Figure 10. For example, after step S911, step S913 is executed first and then step S912 is executed. Alternatively, after step S911 is executed, step S912 and step S913 are executed simultaneously.

In a possible embodiment, the CPU sends the first control instruction to the GPU at the same time, and the GPU sends the first control instruction. In this way, the time when the GPU receives the first control instruction and the time when the NPU receives the second control instruction are almost the same, so that the CPU can control the running state of the control network in the GPU and the NPU at the same time.

S914: In response to receiving the first control instruction, the GPU runs the encoder in the U-type network.

S915. In response to receiving the second control instruction, the NPU runs a control module in the control network.

Specifically, in response to receiving the first control instruction, the GPU establishes a thread to run the encoder in the U-type network, and in response to receiving the second control instruction, the NPU establishes a thread to run the control module in the control network. It can be understood that the GPU and the NPU establish threads respectively, so that the encoder in the U-type network on the GPU and the control module in the control network on the NPU can run in parallel, and the thread of the GPU running the encoder of the U-type network and the thread of the NPU running the control module have overlapping time intervals.

In a possible embodiment, at the same time, the GPU, in response to receiving the first control instruction, establishes a thread to run the encoder in the U-type network, and the NPU, in response to receiving the second control instruction, establishes a thread to run the control module in the control network. In this way, the encoder in the U-type network on the GPU and the control module in the control network on the NPU can start running at the same time.

It is understandable that the execution order of step S914 and step S915 is not limited to the order shown in Fig. 10. For example, after executing step S912 and step S913, step S915 is executed first and then step S914.

S916: The control module operation ends.

Specifically, when the control module on the NPU finishes running, the NPU jumps out of the current thread, and at this time the CPU can determine that the control module has finished running.

S917: The CPU updates the flag parameter in the memory to indicate that the control module operation has ended.

Specifically, after each execution of the image generation process, the electronic device sets the flag parameter stored in the memory to indicate that the Control module has not completed operation. In this way, before each execution of the image generation process, the flag parameter stored in the memory of the electronic device indicates that the Control module has not completed operation. When the control module on the NPU completes operation, the CPU promptly updates the flag parameter in the memory to indicate that the control module has completed operation. For example, before executing the image generation process, the flag parameter stored in the memory of the electronic device is "0", indicating that the Control module has not completed operation. When the control module on the NPU completes operation, the CPU promptly updates the flag parameter in the memory to "1", indicating that the control module has completed operation.

S918. In response to the control module completing its operation, the NPU stores a first operation result of the control module.

Specifically, the running time of the control module is generally shorter than the running time of the encoder in the U-type network. Considering the possibility that the control module may end running first due to some equipment anomalies, the NPU can first store the first running result of the control module (including feature vector 4) in the memory when the control module ends running. When the encoder of the U-type network on the GPU ends running, the first running result of the control module can be read from the memory.

In other embodiments, in response to the completion of the control module operation, the NPU may send the first operation result of the control module to the GPU, so that the GPU runs the decoder in the U-type network and uses the first operation result of the control module (including feature vector 4) as input data for upsampling processing.

It is understandable that both step S917 and step S918 are steps executed after the control module is finished running, and the execution order of the above two steps is not limited to the order shown in Figure 10. For example, after the control module is finished running, step S918 is executed first and then step S917.

S919. In response to the encoder in the U-type network finishing its operation, the GPU reads a flag parameter from the memory, where the flag parameter indicates that the control module has finished its operation.

Specifically, the GPU responds to the end of the operation of the encoder in the U-type network, and the GPU enters an interrupt waiting state, and can periodically read the flag as a parameter from the memory. When the flag as a parameter indicates that the control module has ended, the GPU can run the decoder in the U-type network. Among them, the GPU enters the interrupt waiting state specifically for the framework running the U-type network to suspend execution, that is, the framework running the U-type network does not continue to run the decoder in the U-type network. Exemplarily, the GPU uses the FAITH framework to run the U-type network. The FAITH framework responds to the end of the operation of the encoder in the U-type network. The FAITH framework does not continue to run the decoder in the U-type network, and the FAITH framework can periodically read the flag as a parameter from the memory. The flag bit parameter is used to indicate the running state of the control module. Exemplarily, the flag bit parameter is "0", indicating that the Control module has not ended, and the flag bit parameter is "1" indicating that the control module has ended. When the flag bit parameter is "1" indicating that the control module has ended, the FAITH framework can run the decoder in the U-type network.

Among them, since the image generation takes a short time, the unit of the period of reading the flag bit is small, generally in the ms level. The GPU periodically reads the flag bit parameters to determine the operating status of the control module in a timely manner. The above-mentioned period of reading the flag bit can be specifically set according to actual conditions. For example, the period of reading the flag bit is 2ms, or the period of reading the flag bit is 5ms, which is not specifically limited here.

It should be noted that when the GPU responds to the completion of the encoder operation in the U-type network and reads the flag parameter from the memory, the flag parameter may also indicate that the control module has not completed the operation. At this time, the GPU continues to maintain the interrupt waiting state until the flag parameter read indicates that the control module has completed the operation.

S920: The GPU reads a first operation result of the control module from the memory.

Specifically, when the flag parameter indicates that the control module has finished running, the GPU reads the first running result of the control module from the memory, so as to use the second running result of the encoder in the U-type network and the first running result of the control module as input data of the decoder in the U-type network.

S921, GPU runs the decoder in the U-type network.

Specifically, when the GPU reads the first operation result (including feature vector 4) of the control module, it superimposes the second operation result (including feature vector 5) of the encoder in the U-type network and the above-mentioned first operation result and inputs them into the decoder to run the decoder in the U-type network.

S922, GPU runs the VAE decoder in the control network to obtain image data 3.

Specifically, when the GPU finishes running the decoder in the U-type network, it sequentially runs the VAE decoder in the control network to obtain image data 3.

S923. GPU sends image data 3 to the camera application.

Specifically, the GPU sends the image data 3 output by the control network to the camera application.

S924. The camera application displays a first interface, where the first interface includes image data 3.

Specifically, when the camera application receives the image data 3, it displays a first interface including the image data 3. At this time, the image data 3 can be used as a photo taken and previewed by the camera application.

By adopting the above-mentioned image generation process, the electronic device can use the control network to optimize the initial image data collected by the camera during the camera application shooting process, and run the control module in the control network and the encoder in the U-shaped network in parallel to improve the efficiency of the control network in generating image data 3 based on the initial image data, thereby quickly displaying pictures that meet user needs on the interface and improving the efficiency of the electronic device in taking high-quality photos.

Some other embodiments of the present application provide an electronic device, which may include: a memory and one or more processors. The memory is coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device may perform each step in the above method embodiment. The structure of the electronic device may refer to the structure shown in FIG8.

An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each step in the above-mentioned method embodiment.

The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each step in the above method embodiment.

Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (11)

1. An image generation method, characterized in that the method is applied to an electronic device, a control module and a stable diffusion SD model are configured in the electronic device, the SD model comprises a first encoder and a first decoder, the electronic device comprises a first processor, a second processor and a third processor, the SD model is disposed on the second processor, and the control module is disposed on the third processor, the method comprises:

Before a first time interval, the first processor generates a first control instruction and a second control instruction, wherein the first control instruction comprises a first feature vector of a required text corresponding to image data to be processed, and the second control instruction comprises a second feature vector of the image data to be processed; the first processor sends the first control instruction to the second processor and sends the second control instruction to the third processor;

in a first time interval, the third processor takes a second feature vector of the image data to be processed as input based on the second control instruction, and operates the control module to obtain a first feature map of the image data to be processed;

In a second time interval, the second processor takes a first feature vector of a required text corresponding to image data to be processed as input, and operates the first encoder to obtain a potential code of the required text, wherein the required text is used for describing the content requirement and/or the quality requirement of target image data, and the second time interval and the first time interval have overlapping time intervals;

The second processor obtains the first feature map, operates the first decoder with the potential code and the first feature map as inputs, and generates the target image data.

2. The method of claim 1, wherein the electronic device operates the first encoder with a first feature vector of a desired text corresponding to image data to be processed as an input, and wherein after obtaining the potential encoding of the desired text, the method further comprises:

The electronic equipment acquires a zone bit parameter, wherein the zone bit parameter is used for indicating the running state of the control module, and the running state comprises running end and non-running end;

The electronic device operating the first decoder with the potential encoding and the first feature map as inputs, generating the target image data, comprising:

And in response to the flag bit parameter indicating that the running state of the control module is running end, the electronic device runs the first decoder by taking the potential codes and the first feature map as inputs, and generates the target image data.

3. The method of claim 2, wherein the electronic device obtaining the flag parameter comprises:

The electronic equipment periodically acquires the flag bit parameters;

after the electronic device acquires the flag bit parameter each time, the method further comprises:

responding to the flag bit parameter to indicate that the operation state of the control module is not operation end, and the electronic equipment does not operate the first decoder;

And in response to the flag bit parameter indicating that the running state of the control module is running end, the electronic device runs the first decoder by taking the potential codes and the first feature map as inputs, and generates the target image data.

4. The method of claim 2, wherein the flag parameter indicates that the operating state of the control module is not end of operation before the first time interval;

after the obtaining the first feature map of the image data to be processed, the method further includes:

And the electronic equipment updates the zone bit parameter to indicate that the running state of the control module is running ending.

5. The method of claim 4, wherein after generating the target image data, the method further comprises:

and the electronic equipment updates the zone bit parameter to indicate that the running state of the control module is not running.

6. The method of claim 1, wherein during a first time interval, the electronic device takes as input a second feature vector of image data to be processed, and wherein prior to operating the control module to obtain a first feature map of the image data to be processed, the method further comprises:

the electronic equipment runs a camera application in the electronic equipment at a foreground, and a camera of the electronic equipment acquires the image data to be processed;

after the generating the target image data, the method further comprises:

the electronic equipment displays a first interface, wherein the first interface comprises a shooting preview area, and the shooting preview area comprises the target image data.

7. The method of claim 6, wherein the electronic device comprises a plurality of target texts and a plurality of target feature vectors corresponding to the plurality of target texts one-to-one;

In a second time interval, the electronic device uses a first feature vector of a required text corresponding to image data to be processed as input, and runs the first encoder, so that before potential encoding of the required text is obtained, the method further comprises:

The electronic equipment receives setting operation of a user, wherein the setting operation is used for indicating the content requirement and/or quality requirement of the user on target image data;

the electronic device determines the required text matched with the setting operation and the first feature vector corresponding to the required text from the target texts.

8. The method according to claim 5, wherein after the obtaining the first feature map of the image data to be processed, the method further comprises:

in response to the control module ending in operation, the first processor updates the flag bit parameter to indicate that the operation state of the control module is ending in operation;

after the obtaining the potential encoding of the demand text, the method further comprises:

the second processor acquires a flag bit parameter, wherein the flag bit parameter is used for indicating the running state of the control module, and the running state comprises running end and non-running end;

the second processor obtaining the first feature map, operating the first decoder with the potential encoding and the first feature map as inputs, generating the target image data, comprising:

The second processor acquires the first feature map in response to the flag bit parameter indicating that the running state of the control module is running end, and runs the first decoder with the potential codes and the first feature map as inputs to generate the target image data;

after the generating the target image data, the method further comprises:

and the first processor updates the flag bit parameter to indicate that the running state of the control module is not running end.

9. An electronic device comprising a memory and one or more processors; the memory is coupled to the processor; the memory is for storing computer program code comprising computer instructions which, when executed by the processor, cause the electronic device to perform the method of any of claims 1-8.

10. A computer readable storage medium having instructions stored therein, which when run on an electronic device, cause the electronic device to perform the method of any of claims 1-8.

11. A computer program product comprising computer instructions which, when run on an electronic device, cause the electronic device to perform the method of any of claims 1-8.