CN112070888A - Image generation method, device, equipment and computer readable medium - Google Patents
Image generation method, device, equipment and computer readable medium Download PDFInfo
- Publication number
- CN112070888A CN112070888A CN202010936339.8A CN202010936339A CN112070888A CN 112070888 A CN112070888 A CN 112070888A CN 202010936339 A CN202010936339 A CN 202010936339A CN 112070888 A CN112070888 A CN 112070888A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- feature
- sample
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 239000000463 material Substances 0.000 claims abstract description 47
- 238000009877 rendering Methods 0.000 claims abstract description 25
- 238000000605 extraction Methods 0.000 claims description 100
- 239000000523 sample Substances 0.000 claims description 93
- 238000005286 illumination Methods 0.000 claims description 50
- 238000013075 data extraction Methods 0.000 claims description 27
- 238000012549 training Methods 0.000 claims description 24
- 238000012545 processing Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 11
- 230000004044 response Effects 0.000 claims description 8
- 230000000694 effects Effects 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000002457 bidirectional effect Effects 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000005315 distribution function Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
- G06T15/506—Illumination models
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Graphics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Geometry (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the disclosure discloses an image generation method, an image generation device, an electronic device and a computer readable medium. One embodiment of the method comprises: encoding the acquired image to be processed with the target object displayed to obtain a hidden variable; extracting feature data contained in the hidden variables, wherein the feature data comprise material feature data; rendering is carried out based on the characteristic data to obtain a rendered image. The implementation method can obtain the material characteristic data of the target object displayed in the given image, and can show the three-dimensional shape of the target object displayed in the given image and the material used by the object together, so that the rendered image has a more real effect.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to an image generation method, apparatus, device, and computer-readable medium.
Background
With the development of technology, AR (Augmented Reality) greatly enriches people's lives. The related art can model the three-dimensional shape of an object displayed in an image, and it is difficult to model the material used by the object in one block.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose an image generation method, apparatus, device and computer readable medium to solve the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide an image generation method, the method comprising: encoding the acquired image to be processed with the target object displayed to obtain a hidden variable; extracting feature data contained in the hidden variables, wherein the feature data comprise material feature data; rendering is carried out based on the characteristic data to obtain a rendered image.
In a second aspect, some embodiments of the present disclosure provide an image generation apparatus, the apparatus comprising: the encoding unit is configured to encode the acquired image to be processed, on which the target object is displayed, so as to obtain a hidden variable; an extraction unit configured to extract feature data included in the hidden variable, wherein the feature data includes material feature data; and the rendering unit is configured to render based on the characteristic data to obtain a rendered image.
In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first aspect.
In a fourth aspect, some embodiments of the disclosure provide a computer readable medium having a computer program stored thereon, where the program when executed by a processor implements a method as described in any of the implementations of the first aspect.
One of the above-described various embodiments of the present disclosure has the following advantageous effects: the material characteristic data of the target object displayed in the given image can be obtained, and the three-dimensional shape of the target object displayed in the given image and the material used by the object can be displayed together, so that the rendered image has a more real effect.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.
FIG. 1 is a schematic illustration of one application scenario of the image generation method of some embodiments of the present disclosure;
FIG. 2 is a flow diagram of some embodiments of an image generation method according to the present disclosure;
FIG. 3 is a flow diagram of further embodiments of an image generation method according to the present disclosure;
FIG. 4 is a schematic structural diagram of some embodiments of an image generation apparatus according to the present disclosure;
FIG. 5 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 shows a schematic diagram of one application scenario in which the image generation method of some embodiments of the present disclosure may be applied.
In the application scenario shown in fig. 1, first, the computing device 101 encodes an acquired to-be-processed image 102 showing a target object, and obtains a hidden variable 103 of the to-be-processed image. Then, the computing device 101 may extract feature data 104 included in the hidden variable 103, where the feature data includes texture feature data. Finally, the computing device 101 performs rendering 105 using the feature data to obtain a rendered image 106.
The computing device 101 may be hardware or software. When the computing device is hardware, it may be implemented as a distributed cluster composed of a plurality of servers or electronic devices, or may be implemented as a single server or a single electronic device. When the computing device is embodied as software, it may be implemented as multiple pieces of software or software modules, for example, to provide distributed services, or as a single piece of software or software module. And is not particularly limited herein.
It should be understood that the number of computing devices 101 in FIG. 1 is merely illustrative. There may be any number of computing devices 101, as desired for implementation.
With further reference to fig. 2, a flow 200 of some embodiments of an image generation method is shown. The flow 200 of the image generation method comprises the following steps:
In some embodiments, the execution subject of the image generation method performs image coding on the image to be processed through various image coding algorithms (e.g., huffman coding, predictive coding), so as to obtain the hidden variable. The image coding may be to transform and combine the images to be processed according to a certain rule. Image coding can reduce the relevant redundancy of the image pixels to be processed, and realize that the data as less as possible is used for representing as much information as possible. The hidden variable is data of the image to be processed after being coded.
Optionally, the main body of the image generation method may input the image to be processed into an encoder structure of a convolutional neural network (e.g., an encoder structure of SegNet). After the image to be processed is given, the encoder structure obtains the hidden variable of the image to be processed through the learning of the convolutional neural network.
In some embodiments, the characteristic data may include: depth feature data, illumination feature data, normal vector feature data, color feature data, and material feature data. The feature data can be divided into two types, one type is represented in the form of a feature map (e.g., depth feature data), and the other type is represented in the form of a feature vector (e.g., illumination feature data).
Wherein, the illumination characteristic data can be represented by spherical harmonic illumination or ambient illumination. Ambient lighting is a simplified global illumination model that adds a constant color, which is small in value, to the final color of the point of illumination of the object. In a three-dimensional scene, the lighting situation in various directions is usually represented by an ambient light map. The various directions of the ambient light map need to be sampled to determine the lighting situation of a point. Spherical harmonic illumination is a simplification of illumination. Because the illumination is different in all directions for a point in space, spherical harmonic illumination fully restores the illumination for a point in space by recording the illumination in all directions on the surrounding spherical surface. The material characteristic data is a parameter of a bidirectional reflection distribution function, and the parameter of the function can be obtained by a solution optimization method. The bidirectional reflection function is a generation algorithm of the surface color of the object after receiving illumination, and represents the surface material information of the object.
In some embodiments, the executing entity of the image generation method may extract feature data included in the hidden variable through a convolutional neural network. The convolutional neural network may include, but is not limited to: FCN Networks (full Convolutional Networks), ResNet Networks (Residual Networks), and Dense Convolutional Networks (densneet).
And step 203, rendering based on the characteristic data to obtain a rendered image.
In some embodiments, the execution body may perform rendering by using feature data of depth, illumination, normal vector, color, and texture, to obtain a rendered image representing the texture information.
One of the above-described various embodiments of the present disclosure has the following advantageous effects: firstly, the execution main body of the image generation method obtains hidden variables which represent as much information as possible by using as little data as possible by encoding the image to be processed, and is convenient for extracting subsequent characteristic data. Then, the execution body extracts feature data containing material information and renders by using the feature data containing the material information, so that the three-dimensional shape of the target object displayed in the given image and the material used by the object can be displayed together, and the rendered image has a more real effect.
With further reference to fig. 3, a flow 300 of further embodiments of an image generation method is shown. The flow 300 of the image generation method includes the following steps:
In some embodiments, the executing entity of the image generating method performs a masking process on the image to be processed to obtain an image highlighting the target object. The mask processing is to multiply the pre-made target object area mask with the image to be processed to obtain a target object image. The pixel values of the image in the target object area are kept unchanged, and the pixel values of the image outside the target object area are all 0. Wherein the mask is a template of the image filter. For example, a matrix of equal size to the image, which is a mask, pixel filters the image and then highlights the target object.
Step 301 may highlight the image of the target object, ensure that the extracted material characteristic data only includes the material characteristic data of the target object, and reduce the influence of the scene information on the target object characteristic data.
In an optional implementation manner of some embodiments, the pre-trained feature data extraction model may be obtained by training through the following steps: obtaining a sample set, wherein samples in the sample set comprise sample images displaying target objects and sample characteristic data corresponding to the sample images, and the sample characteristic data comprise sample material characteristic data; selecting samples from the sample set, and performing the following training steps: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering an image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset feature weight and image weight, taking the weighted result of the feature loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be a feature data extraction model.
In an optional implementation manner of some embodiments, the pre-trained feature data extraction model may further include the following steps: and responding to the condition that the initial model is not trained completely, adjusting relevant parameters in the initial model, reselecting samples from the sample set, and continuing to execute the training step by using the adjusted initial model as the initial model.
In some embodiments, the above coding network model may include, but is not limited to: VGG Networks (Visual Geometry Group, Deep convolutional neural Networks), ResNet Networks (Deep Residual Networks).
Step 303 is to input the hidden variables into the at least one feature extraction sub-network, and to output a result of the at least one feature extraction sub-network as the feature data, wherein the feature data includes material feature data.
In an optional implementation of some embodiments, the at least one feature extraction sub-network comprises: the system comprises an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network. Thus, step 303 may proceed as follows:
firstly, inputting the hidden variables into the illumination characteristic extraction sub-network to obtain illumination characteristic data.
Wherein, the illumination characteristic data is 3-order spherical harmonic illumination. Among them, spherical harmonic illumination is a simplification of illumination. Because the illumination in each direction is different for one point in the space, when the rendering is performed by utilizing the spherical harmonic illumination, the illumination in each direction on the surrounding spherical surface needs to be recorded to determine the illumination condition of one point in the space, so that the rendering has certain difficulty. In practice, the illumination in all directions of the surrounding spherical surface can be represented by spherical harmonics. The spherical harmonic function can represent a complex spherical function by using a simple spherical harmonic base and corresponding coefficients. Specifically, a spherical function is restored by multiplying a simple spherical harmonic basis function by a corresponding coefficient, thereby approximately representing the spherical harmonic illumination. As an example, a 3 rd order spherical harmonic illumination approximation is utilized instead of spherical harmonic illumination. Thus, the complexity in the rendering process is reduced.
And secondly, inputting the hidden variables into the color feature extraction sub-network to obtain color feature data.
And thirdly, inputting the hidden variables into the material characteristic extraction sub-network to obtain material characteristic data.
The material characteristic data is a parameter of a bidirectional reflection distribution function. The bidirectional reflection function is a generation algorithm of the surface color of the object after receiving illumination, and represents the surface material information of the object.
The parameters of the bidirectional reflection distribution function are solved through the material characteristic extraction sub-network, so that the characteristic extraction efficiency is improved.
And fourthly, inputting the hidden variables into the normal vector feature extraction sub-network to obtain normal vector feature data.
And fifthly, inputting the hidden variables into the depth feature extraction sub-network to obtain depth feature data.
And step 304, rendering based on the characteristic data to obtain a rendered image.
In some embodiments, the execution body renders the extracted depth, material, normal vector, illumination and color to obtain a rendered image containing material information.
As can be seen from fig. 3, compared with the description of some embodiments corresponding to fig. 2, the flow 300 of the image generation method in some embodiments corresponding to fig. 3 highlights an implementation process of extracting depth, texture, color, normal vector, and illumination in the target object image by using the pre-trained feature data extraction model. The pre-trained feature data extraction model greatly reduces the complexity of extracting the feature data of the target object image. Meanwhile, 3-order spherical harmonic illumination replaces spherical harmonic illumination, so that the image rendering process is easier to realize.
With further reference to fig. 4, as an implementation of the methods illustrated in the above figures, the present disclosure provides some embodiments of an image generation apparatus, which correspond to those illustrated in fig. 2, and which may be particularly applicable in various electronic devices.
As shown in fig. 4, the image generation apparatus 400 of some embodiments includes: the encoding unit 401 is configured to encode the acquired image to be processed, in which the target object is displayed, to obtain a hidden variable; an extracting unit 402 configured to extract feature data included in the hidden variable, wherein the feature data includes material feature data; and a rendering unit 403 configured to perform rendering based on the feature data to obtain a rendered image.
In an optional implementation of some embodiments, the encoding unit of the apparatus 400 may be further configured to: performing mask processing on the image to be processed to obtain a target object image; and inputting the target object image into a coding network in a pre-trained feature data extraction model to obtain the hidden variable, wherein the feature data extraction model comprises the coding network and at least one feature extraction sub-network.
In an optional implementation of some embodiments, the extraction unit of the apparatus 400 may be further configured to: the hidden variables are input to the at least one feature extraction sub-network, and the result output by the at least one feature extraction sub-network is used as the feature data.
In an optional implementation of some embodiments, the at least one feature extraction sub-network of the apparatus 400 comprises: the system comprises an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network; and the above extraction unit is further configured to: inputting the hidden variables into the illumination characteristic extraction sub-network to obtain illumination characteristic data; inputting the hidden variables into the color feature extraction sub-network to obtain color feature data; inputting the hidden variables into the material characteristic extraction sub-network to obtain material characteristic data; inputting the hidden variables into the normal vector feature extraction sub-network to obtain normal vector feature data; and inputting the hidden variables into the depth feature extraction sub-network to obtain depth feature data.
In an alternative implementation of some embodiments, the feature data extraction model of the apparatus 400 is trained by: the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is configured to acquire a sample set, samples in the sample set comprise sample images displaying target objects and sample characteristic data corresponding to the sample images, and the sample characteristic data comprises sample material characteristic data; a training unit configured to select samples from the set of samples and to perform the following training steps: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering an image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset feature weight and image weight, taking the weighted result of the feature loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be a feature data extraction model.
In an optional implementation of some embodiments, the apparatus 400 may further include: and the adjusting unit is configured to respond to the condition that the initial model is not trained completely, adjust relevant parameters in the initial model, reselect samples from the sample set, and continue to execute the training step by using the adjusted initial model as the initial model.
It will be understood that the elements described in the apparatus 400 correspond to various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 400 and the units included therein, and will not be described herein again.
Referring now to fig. 5, a schematic diagram of an electronic device (e.g., the server or terminal device of fig. 1) 500 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device in some embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle-mounted terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 5, electronic device 500 may include a processing means (e.g., central processing unit, graphics processor, etc.) 501 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data necessary for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM 502, and the RAM503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
Generally, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, and the like; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and a communication device 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While fig. 5 illustrates an electronic device 500 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 5 may represent one device or may represent multiple devices as desired.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network via the communication means 509, or installed from the storage means 508, or installed from the ROM 502. The computer program, when executed by the processing device 501, performs the above-described functions defined in the methods of some embodiments of the present disclosure.
It should be noted that the computer readable medium described in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: encoding the acquired image to be processed with the target object displayed to obtain a hidden variable; extracting feature data contained in the hidden variables, wherein the feature data comprise material feature data; rendering is carried out based on the characteristic data to obtain a rendered image.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by software, and may also be implemented by hardware. The described units may also be provided in a processor, and may be described as: a processor includes an encoding unit, an extraction unit, and a rendering unit. The names of the units do not limit the units themselves in some cases, for example, the encoding unit may also be described as a "unit that encodes the acquired image to be processed, in which the target object is displayed, to obtain the hidden variable".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
According to one or more embodiments of the present disclosure, there is provided an image generation method including: encoding the acquired image to be processed with the target object displayed to obtain a hidden variable; extracting feature data contained in the hidden variables, wherein the feature data comprise material feature data; rendering is carried out based on the characteristic data to obtain a rendered image.
According to one or more embodiments of the present disclosure, the encoding the acquired to-be-processed image in which the target object is displayed to obtain the hidden variable includes: performing mask processing on the image to be processed to obtain a target object image; and inputting the target object image into a coding network in a pre-trained feature data extraction model to obtain the hidden variable, wherein the feature data extraction model comprises the coding network and at least one feature extraction sub-network.
According to one or more embodiments of the present disclosure, the extracting feature data included in the hidden variable includes: the hidden variables are input to the at least one feature extraction sub-network, and the result output by the at least one feature extraction sub-network is used as the feature data.
According to one or more embodiments of the present disclosure, the at least one feature extraction subnetwork comprises: the system comprises an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network; and inputting the hidden variables into the at least one feature extraction sub-network, respectively, and using a result output by the at least one feature extraction sub-network as the feature data, the method including: inputting the hidden variables into the illumination characteristic extraction sub-network to obtain illumination characteristic data; inputting the hidden variables into the color feature extraction sub-network to obtain color feature data; inputting the hidden variables into the material characteristic extraction sub-network to obtain material characteristic data; inputting the hidden variables into the normal vector feature extraction sub-network to obtain normal vector feature data; and inputting the hidden variables into the depth feature extraction sub-network to obtain depth feature data.
According to one or more embodiments of the present disclosure, the feature data extraction model is obtained by training: obtaining a sample set, wherein samples in the sample set comprise sample images displaying target objects and sample characteristic data corresponding to the sample images, and the sample characteristic data comprise sample material characteristic data; selecting samples from the sample set, and performing the following training steps: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering an image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset feature weight and image weight, taking the weighted result of the feature loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be a feature data extraction model.
According to one or more embodiments of the present disclosure, the method further includes: and responding to the condition that the initial model is not trained completely, adjusting relevant parameters in the initial model, reselecting samples from the sample set, and continuing to execute the training step by using the adjusted initial model as the initial model.
According to one or more embodiments of the present disclosure, there is provided an image generation apparatus including: the encoding unit is configured to encode the acquired image to be processed, on which the target object is displayed, so as to obtain a hidden variable; an extraction unit configured to extract feature data included in the hidden variable, wherein the feature data includes material feature data; and the rendering unit is configured to render based on the characteristic data to obtain a rendered image.
According to one or more embodiments of the present disclosure, the encoding unit is further configured to: performing mask processing on the image to be processed to obtain a target object image; and inputting the target object image into a coding network in a pre-trained feature data extraction model to obtain the hidden variable, wherein the feature data extraction model comprises the coding network and at least one feature extraction sub-network.
According to one or more embodiments of the present disclosure, the above extraction unit is further configured to: the hidden variables are input to the at least one feature extraction sub-network, and the result output by the at least one feature extraction sub-network is used as the feature data.
According to one or more embodiments of the present disclosure, the at least one feature extraction subnetwork comprises: the system comprises an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network; and the above extraction unit is further configured to: inputting the hidden variables into the illumination characteristic extraction sub-network to obtain illumination characteristic data; inputting the hidden variables into the color feature extraction sub-network to obtain color feature data; inputting the hidden variables into the material characteristic extraction sub-network to obtain material characteristic data; inputting the hidden variables into the normal vector feature extraction sub-network to obtain normal vector feature data; and inputting the hidden variables into the depth feature extraction sub-network to obtain depth feature data.
According to one or more embodiments of the present disclosure, the feature data extraction model is obtained by training: the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is configured to acquire a sample set, samples in the sample set comprise sample images displaying target objects and sample characteristic data corresponding to the sample images, and the sample characteristic data comprises sample material characteristic data; a training unit configured to select samples from the set of samples and to perform the following training steps: inputting a sample image in the sample into an initial model to generate prediction characteristic data of the sample image; rendering an image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset feature weight and image weight, taking the weighted result of the feature loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be a feature data extraction model.
According to one or more embodiments of the present disclosure, the apparatus further includes: and the adjusting unit is configured to respond to the condition that the initial model is not trained completely, adjust relevant parameters in the initial model, reselect samples from the sample set, and continue to execute the training step by using the adjusted initial model as the initial model.
According to one or more embodiments of the present disclosure, there is provided an electronic device including: one or more processors; a storage device having one or more programs stored thereon which, when executed by one or more processors, cause the one or more processors to implement a method as in any above.
According to one or more embodiments of the present disclosure, there is provided a non-transitory computer readable storage medium having a computer program stored thereon, wherein the program when executed by a processor implements the method as any one of the above.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (14)
1. An image generation method, comprising:
encoding the acquired image to be processed with the target object displayed to obtain a hidden variable;
extracting feature data contained in the hidden variables, wherein the feature data comprise material feature data;
rendering is carried out based on the characteristic data to obtain a rendered image.
2. The method according to claim 1, wherein the encoding the acquired to-be-processed image with the target object displayed thereon to obtain the hidden variable includes:
performing mask processing on the image to be processed to obtain a target object image;
and inputting the target object image into a coding network in a pre-trained feature data extraction model to obtain the hidden variable, wherein the feature data extraction model comprises the coding network and at least one feature extraction sub-network.
3. The method of claim 2, wherein said extracting feature data contained in said hidden variables comprises:
and respectively inputting the hidden variables into the at least one feature extraction sub-network, and taking the result output by the at least one feature extraction sub-network as the feature data.
4. The method of claim 3, wherein the at least one feature extraction subnetwork comprises: the system comprises an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network; and
inputting the hidden variables into the at least one feature extraction sub-network respectively, and taking the result output by the at least one feature extraction sub-network as the feature data, including:
inputting the hidden variable into the illumination characteristic extraction sub-network to obtain illumination characteristic data;
inputting the hidden variables into the color feature extraction sub-network to obtain color feature data;
inputting the hidden variables into the material characteristic extraction sub-network to obtain material characteristic data;
inputting the hidden variables into the normal vector feature extraction sub-network to obtain normal vector feature data;
and inputting the hidden variables into the depth feature extraction sub-network to obtain depth feature data.
5. The method of claim 2, wherein the feature data extraction model is trained by:
obtaining a sample set, wherein samples in the sample set comprise sample images displaying target objects and sample characteristic data corresponding to the sample images, and the sample characteristic data comprise sample material characteristic data;
selecting samples from the sample set, and performing the following training steps:
inputting a sample image in the sample into an initial model, and generating prediction characteristic data of the sample image;
rendering an image based on the prediction characteristic data to obtain a predicted image;
analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value;
analyzing the predicted image and the sample image to determine an image loss value;
according to preset feature weight and image weight, taking the weighted result of the feature loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value;
determining whether the initial model is trained according to the comparison result;
in response to determining that the initial model training is complete, the initial model is determined to be a feature data extraction model.
6. The method of claim 5, wherein the method further comprises:
and in response to determining that the initial model is not trained completely, adjusting relevant parameters in the initial model, reselecting samples from the sample set, and continuing to perform the training step by using the adjusted initial model as the initial model.
7. An image generation apparatus comprising:
the encoding unit is configured to encode the acquired image to be processed, on which the target object is displayed, so as to obtain a hidden variable;
an extraction unit configured to extract feature data included in the hidden variable, wherein the feature data includes material feature data;
and the rendering unit is configured to render based on the feature data to obtain a rendered image.
8. The apparatus of claim 7, wherein the encoding unit is further configured to:
performing mask processing on the image to be processed to obtain a target object image;
and inputting the target object image into a coding network in a pre-trained feature data extraction model to obtain the hidden variable, wherein the feature data extraction model comprises the coding network and at least one feature extraction sub-network.
9. The apparatus of claim 8, wherein the extraction unit is further configured to:
and respectively inputting the hidden variables into the at least one feature extraction sub-network, and taking the result output by the at least one feature extraction sub-network as the feature data.
10. The apparatus of claim 9, wherein the at least one feature extraction subnetwork comprises: the system comprises an illumination characteristic extraction sub-network, a color characteristic extraction sub-network, a material characteristic extraction sub-network, a normal vector characteristic extraction sub-network and a depth characteristic extraction sub-network; and
the extraction unit is further configured to:
inputting the hidden variable into the illumination characteristic extraction sub-network to obtain illumination characteristic data;
inputting the hidden variables into the color feature extraction sub-network to obtain color feature data;
inputting the hidden variables into the material characteristic extraction sub-network to obtain material characteristic data;
inputting the hidden variables into the normal vector feature extraction sub-network to obtain normal vector feature data;
and inputting the hidden variables into the depth feature extraction sub-network to obtain depth feature data.
11. The apparatus of claim 8, the feature data extraction model is trained by:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is configured to acquire a sample set, samples in the sample set comprise sample images on which target objects are displayed and sample characteristic data corresponding to the sample images, and the sample characteristic data comprises sample material characteristic data;
a training unit configured to select samples from the set of samples and to perform the following training steps: inputting a sample image in the sample into an initial model, and generating prediction characteristic data of the sample image; rendering an image based on the prediction characteristic data to obtain a predicted image; analyzing the predicted characteristic data and the sample characteristic data to determine a characteristic loss value; analyzing the predicted image and the sample image to determine an image loss value; according to preset feature weight and image weight, taking the weighted result of the feature loss value and the image loss value as a sample total loss value, and comparing the sample total loss value with a target value; determining whether the initial model is trained according to the comparison result; in response to determining that the initial model training is complete, the initial model is determined to be a feature data extraction model.
12. The apparatus of claim 11, wherein the apparatus further comprises:
an adjusting unit configured to adjust relevant parameters in the initial model and reselect samples from the sample set in response to determining that the initial model is not trained, and continue the training step using the adjusted initial model as the initial model.
13. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
14. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010936339.8A CN112070888B (en) | 2020-09-08 | 2020-09-08 | Image generation method, device, equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010936339.8A CN112070888B (en) | 2020-09-08 | 2020-09-08 | Image generation method, device, equipment and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112070888A true CN112070888A (en) | 2020-12-11 |
| CN112070888B CN112070888B (en) | 2024-04-05 |
Family
ID=73664454
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010936339.8A Active CN112070888B (en) | 2020-09-08 | 2020-09-08 | Image generation method, device, equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112070888B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114494494A (en) * | 2022-01-24 | 2022-05-13 | 广州繁星互娱信息科技有限公司 | Method and device for adjusting image illumination parameters, storage medium and electronic device |
| CN114764840A (en) * | 2020-12-31 | 2022-07-19 | 阿里巴巴集团控股有限公司 | Image rendering method, device, equipment and storage medium |
| CN114972112A (en) * | 2022-06-17 | 2022-08-30 | 如你所视(北京)科技有限公司 | Method, apparatus, device and medium for image inverse rendering |
| CN115249215A (en) * | 2021-04-28 | 2022-10-28 | 北京字跳网络技术有限公司 | Image processing method, apparatus, electronic device, and readable storage medium |
| GB2631452A (en) * | 2023-06-30 | 2025-01-08 | Sony Interactive Entertainment Deutschland Gmbh | Neural coding and decoding method and apparatus |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107564049A (en) * | 2017-09-08 | 2018-01-09 | 北京达佳互联信息技术有限公司 | Faceform's method for reconstructing, device and storage medium, computer equipment |
| CN108875935A (en) * | 2018-06-11 | 2018-11-23 | 兰州理工大学 | Based on the natural image target materials visual signature mapping method for generating confrontation network |
| CN109636741A (en) * | 2018-11-14 | 2019-04-16 | 建湖云飞数据科技有限公司 | A kind of image denoising processing method |
| CN109753578A (en) * | 2019-01-25 | 2019-05-14 | 浙江理工大学 | An Image Retrieval Method Based on Mask Selection Convolutional Features |
| WO2019227479A1 (en) * | 2018-06-01 | 2019-12-05 | 华为技术有限公司 | Method and apparatus for generating face rotation image |
-
2020
- 2020-09-08 CN CN202010936339.8A patent/CN112070888B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107564049A (en) * | 2017-09-08 | 2018-01-09 | 北京达佳互联信息技术有限公司 | Faceform's method for reconstructing, device and storage medium, computer equipment |
| WO2019227479A1 (en) * | 2018-06-01 | 2019-12-05 | 华为技术有限公司 | Method and apparatus for generating face rotation image |
| CN108875935A (en) * | 2018-06-11 | 2018-11-23 | 兰州理工大学 | Based on the natural image target materials visual signature mapping method for generating confrontation network |
| CN109636741A (en) * | 2018-11-14 | 2019-04-16 | 建湖云飞数据科技有限公司 | A kind of image denoising processing method |
| CN109753578A (en) * | 2019-01-25 | 2019-05-14 | 浙江理工大学 | An Image Retrieval Method Based on Mask Selection Convolutional Features |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114764840A (en) * | 2020-12-31 | 2022-07-19 | 阿里巴巴集团控股有限公司 | Image rendering method, device, equipment and storage medium |
| CN115249215A (en) * | 2021-04-28 | 2022-10-28 | 北京字跳网络技术有限公司 | Image processing method, apparatus, electronic device, and readable storage medium |
| CN114494494A (en) * | 2022-01-24 | 2022-05-13 | 广州繁星互娱信息科技有限公司 | Method and device for adjusting image illumination parameters, storage medium and electronic device |
| CN114972112A (en) * | 2022-06-17 | 2022-08-30 | 如你所视(北京)科技有限公司 | Method, apparatus, device and medium for image inverse rendering |
| WO2023241065A1 (en) * | 2022-06-17 | 2023-12-21 | 如你所视(北京)科技有限公司 | Method and apparatus for image inverse rendering, and device and medium |
| CN114972112B (en) * | 2022-06-17 | 2024-05-14 | 如你所视(北京)科技有限公司 | Method, apparatus, device and medium for image inverse rendering |
| GB2631452A (en) * | 2023-06-30 | 2025-01-08 | Sony Interactive Entertainment Deutschland Gmbh | Neural coding and decoding method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112070888B (en) | 2024-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112070888B (en) | Image generation method, device, equipment and computer readable medium | |
| CN111784565B (en) | Image processing method, migration model training method, device, medium and equipment | |
| CN113689372B (en) | Image processing method, apparatus, storage medium, and program product | |
| CN110021052B (en) | Method and apparatus for generating fundus image generation model | |
| CN112330788B (en) | Image processing method, device, readable medium and electronic device | |
| CN111784712B (en) | Image processing method, device, equipment and computer readable medium | |
| CN110298851B (en) | Training method and device for human body segmentation neural network | |
| CN113449070A (en) | Multimodal data retrieval method, device, medium and electronic equipment | |
| CN117671254A (en) | Image segmentation method and device | |
| CN112418249A (en) | Mask image generation method, apparatus, electronic device and computer readable medium | |
| CN110310293B (en) | Human body image segmentation method and device | |
| CN114066722B (en) | Method and device for acquiring image and electronic equipment | |
| CN119251373A (en) | Map generation method, device, medium, equipment and computer program product | |
| CN111669476A (en) | Watermark processing method, device, electronic equipment and medium | |
| CN115546766B (en) | Lane line generation method, lane line generation device, electronic device, and computer-readable medium | |
| CN110765304A (en) | Image processing method, image processing device, electronic equipment and computer readable medium | |
| CN113628097A (en) | Image special effect configuration method, image recognition method, image special effect configuration device and electronic equipment | |
| CN113066166B (en) | Image processing method, device and electronic device | |
| CN111726476B (en) | Image processing method, device, equipment and computer readable medium | |
| CN112418233A (en) | Image processing method, device, readable medium and electronic device | |
| CN113706385A (en) | Video super-resolution method and device, electronic equipment and storage medium | |
| CN111738899B (en) | Method, apparatus, device and computer readable medium for generating watermark | |
| CN113378808B (en) | Person image recognition method and device, electronic equipment and computer readable medium | |
| CN112215789B (en) | Image defogging method, device, equipment and computer readable medium | |
| US20250054271A1 (en) | Video generation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Tiktok vision (Beijing) Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: BEIJING BYTEDANCE NETWORK TECHNOLOGY Co.,Ltd. Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Douyin Vision Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: Tiktok vision (Beijing) Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |