KR102655867B1 - Method for image outpainting and apparatus using the same - Google Patents

Abstract

According to the present invention, in the image out painting method, there is a Semantic Regeneration Network (SRN) including a Feature Expansion Network (FEN) and a Context Prediction Network (CPN), and the FEN includes a first encoder and a first decoder. In a state where the CPN includes a second encoder and a second decoder, the computing device inputs an input image to the first encoder of the FEN to generate a first bottleneck, and connects the first bottleneck to the first encoder. Obtaining a FEN result having a size larger than the input image by inputting it to a decoder; The computing device merges the FEN result with an extended input image including the input image but has an extended area, and inputs the result to the second encoder of the CPN to generate a second bottleneck, Inputting feature values obtained using a second bottleneck into the second decoder; And the computing device matches the input image using the second decoder, which has been trained through the first loss function and the second loss function, and obtains an output image including an image whose area is expanded by the expanded size. Presents a method including the following steps:

Description

Image outpainting method and device using the same {METHOD FOR IMAGE OUTPAINTING AND APPARATUS USING THE SAME}

The present invention relates to an image outpainting method, wherein there is a Semantic Regeneration Network (SRN) including a Feature Expansion Network (FEN) and a Context Prediction Network (CPN), and the FEN includes a first encoder and a first decoder, The CPN includes a second encoder and a second decoder, and the computing device inputs an input image to the first encoder of the FEN to generate a first bottleneck, and transmits the first bottleneck to the first decoder. Obtaining a FEN result having a size larger than the input image by inputting it; The computing device merges the FEN result with an extended input image including the input image but has an extended area, inputs the result to the second encoder of the CPN, and generates a second bottleneck. Inputting feature values obtained using a second bottleneck into the second decoder; And the computing device matches the input image using the second decoder, which has been trained through a first loss function, a second loss function, and a Conditional Gan Loss function, and includes an image whose area is expanded by the expanded size. It relates to a method including the step of obtaining an output image.

Image outpainting is a technology that can continuously fill in the outside of a given image by considering the context of the image. At this time, it is important to maintain consistency between the content of the created area and the original input.

In the bio industry, class imbalance problems arise due to data imbalance, and to solve this problem, it is necessary to generate abnormal data, and one of the technologies used in this case is outpainting technology. By removing only the lesion from a normal endoscope and using it as input to a deep learning network model, it will be possible to create various endoscopic pictures that fit the lesion context.

The present inventor would like to propose an image outpainting method and a device using the same.

The present invention aims to solve all of the above-mentioned problems.

Another purpose of the present invention is to generate an expanded image while maintaining the context of a small-sized original image.

Additionally, another purpose of the present invention is to generate various images based on the original image.

In order to achieve the object of the present invention as described above and realize the characteristic effects of the present invention described later, the characteristic configuration of the present invention is as follows.

According to one aspect of the present invention, in the image out painting method, there is a Semantic Regeneration Network (SRN) including a Feature Expansion Network (FEN) and a Context Prediction Network (CPN), and the FEN is a first encoder and a first and a decoder, and the CPN includes a second encoder and a second decoder, and the computing device inputs an input image to the first encoder of the FEN to generate a first bottleneck, and generates a first bottleneck. Obtaining a FEN result having a size larger than the input image by inputting it to the first decoder; The computing device merges the FEN result with an extended input image including the input image but has an extended area, and inputs the result to the second encoder of the CPN to generate a second bottleneck, Inputting feature values obtained using a second bottleneck into the second decoder; And the computing device matches the input image using the second decoder that has been trained through a first loss function, a second loss function, and a Conditional Gan Loss function, and includes an image whose area is expanded by the expanded size. A method is provided including the step of obtaining an output image.

In addition, according to another aspect of the present invention, in an apparatus for performing image outpainting, there is a Semantic Regeneration Network (SRN) including a Feature Expansion Network (FEN) and a Context Prediction Network (CPN), and the FEN is A communication unit including one encoder and a first decoder, and the CPN including a second encoder and a second decoder; database; and inputting the input image to the first encoder of the FEN to generate a first bottleneck, inputting the first bottleneck to the first decoder to obtain a FEN result having a size larger than the input image, and inputting the input image to the first encoder of the FEN. Merge the FEN result with an extended input image that includes an image but has an expanded area, input the result to the second encoder of the CPN to generate a second bottleneck, and obtain it using the second bottleneck. One feature value is input to the second decoder, and the second decoder that has completed learning through the first loss function, second loss function, and Conditional Gan Loss function is used to match the input image to an area equal to the expanded size. A computing device is provided that includes a processor that obtains an output image including this expanded image.

According to the present invention, the following effects are achieved.

The present invention has the effect of generating an expanded image while maintaining the context of the original image in a small size.

Additionally, the present invention has the effect of generating various images based on the original image.

1 is a diagram showing an algorithm flow according to an embodiment of the present invention.
Figure 2 is a diagram showing a sub-pixel convolution process according to an embodiment of the present invention.
Figure 3 is a diagram showing the Context Normalization formula according to an embodiment of the present invention.
Figure 4 is a diagram showing a CPN decoder according to an embodiment of the present invention.
Figure 5 is a diagram showing the formulas of the first loss function and the second loss function according to an embodiment of the present invention.
Figure 6 is a diagram illustrating a visualization of the second loss function according to an embodiment of the present invention.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

The computing device of the present invention can perform image outpainting using a deep learning network. The present invention can generally be used in the bio industry and may include a technology for removing only a lesion (ex. cancer area) from a normal endoscope and generating multiple endoscopic pictures based on this. We will look into this below.

A computing device may include a communication unit, a processor, and a database.

The communication unit of the computing device may be implemented using various communication technologies. That is, WIFI, WCDMA (Wideband CDMA), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), HSPA (High Speed Packet Access), Mobile WiMAX, and WiBro. , LTE (Long Term Evolution), Bluetooth, IrDA (infrared data association), NFC (Near Field Communication), Zigbee, wireless LAN technology, etc. may be applied. Additionally, when providing services by connecting to the Internet, TCP/IP, the standard protocol for information transmission on the Internet, can be followed.

1 is a diagram showing an algorithm flow according to an embodiment of the present invention.

First, the algorithm used in the present invention corresponds to a Semantic Regeneration Network (SRN), and the network may include a Feature Expansion Network (FEN, 100) and a Context Prediction Network (CPN, 200).

As can be seen in FIG. 1, the FEN 100 includes a first encoder 110 and a first decoder 130, and the CPN 200 includes a second encoder 210 and a second decoder 240. It can be included. In other words, both FEN (100) and CPN (200) are using the Encoder and Decoder models.

The processor of the computing device may generate the first bottleneck 120 by inputting the input image 10 to the first encoder 110 of the FEN 100. The first bottleneck 120 may correspond to a result of dilated convolution performed in the first encoder 110.

Next, the computing device may input the first bottleneck into the first decoder 130 to obtain a FEN result 20 having a size larger than the input image 10. The FEN result 20 is a type of feature map and may include a larger number of pixels than the number of pixels included in the input image 10.

Since the number of layers included in the first decoder 130 is greater than the number of layers in the first encoder 110, the FEN result 20 may have a larger size than the input image 10.

As the input image 10 with a size of (h, w, c) passes through the FEN 100, the resulting image corresponding to the FEN output 20 has a size of (r1h, r2w, c'). You can. In other words, while going through FEN (100), the size of the features can be upsampled to the k-nearest neighbor.

Figure 2 is a diagram showing a sub-pixel convolution process according to an embodiment of the present invention.

The computing device may perform sub-pixel convolution in a specific layer 131 included in the first decoder 130.

The sub-pixel convolution is one of the upsampling techniques. Unlike existing upsampling, which requires a large amount of calculation, parallel processing is possible, so calculation speed can be fast. In addition, previous upsampling techniques are performed through one layer, but this sub-pixel convolution can have higher accuracy because it goes through multiple layers.

The FEN 100 of the present invention may correspond to a previous step for the next network, the CPN 200.

The computing device may generate the second bottleneck 220 by inputting the FEN result 20 generated by the FEN 100 into the second encoder 210 of the CPN 200. The second bottleneck 220 may also correspond to the result of dilated convolution performed in the second encoder 210.

However, before inputting the FEN result 20 to the CPN 200, the computing device combines the FEN result 20 with an extended input image 40 that includes the input image 10 but has an expanded area. They can be merged, and the resulting value can be input into the CPN (200).

The extended input image 40 includes the pixels of the input image 10 as is, and expands the area to have the same size as the FEN result 20, but the extended area may have a pixel value of '0'. there is. That is, it consists of an expanded area that includes the pixels of the input image 10 and has a pixel value of '0'.

In the case of the FEN result 20, the pixels of the input image 10 are changed, making it difficult to display the content of the image. Therefore, in the computing device of the present invention, the extended input image 40 and the FEN result 20 are used. The relevance of the input image 10 can be added to the FEN result 20 by merging channel-wise.

Ultimately, the computing device can merge the extended input image 40 and the FEN result 20 channel-wise and input the resulting value into the second encoder 210 of the CPN 200.

Figure 3 is a diagram showing the Context Normalization formula according to an embodiment of the present invention.

The computing device may perform a context normalization process on the second bottleneck 220. The Context Normalization corresponds to a process that supports the area created through the CPN 200 process to maintain consistency with the input image 10.

For reference, the Context Normalization corresponds to the formula shown in Figure 3, and the n(.) operator normalizes the expanded part and converts it back to a normal image, which can be done by borrowing the normalization formula of the input image (10). As shown in Figure 1, this can also be confirmed through the fact that the FEN result 20 is used in the CN (Context Normalization) process.

The computing device may perform a CN (Context Normalization) process on the second bottleneck 220 and input the obtained feature value 230 to the second decoder 240. For reference, learning may be performed in the second decoder 240 of the present invention through the first loss function and the second loss function.

Figure 4 is a diagram showing a second decoder of CPN according to an embodiment of the present invention.

The feature value 230 may be input to the second decoder 240 on which learning has been completed through the first loss function and the second loss function. Therefore, learning of the first loss function and the second loss function included in the second decoder 240 must be preceded, and this will be discussed below.

As can be seen in FIG. 4, the second decoder 240 may include a plurality of upscale (or upsample) modules, and the feature value 230 obtained using the second bottleneck 220 Z0 (Z0 will be described later) and a conditional vector (conditional code) derived from the second encoder 210 can be input.

For reference, the upscale module may include Deconv, Leaky RelU, and AdaIN operations.

In calculating the loss function included in the learning process, a plurality of CN Results (feature values, Z0) are required, and for this purpose, the present invention obtains a plurality of CN Result values (Z0) through a plurality of image patches (input images). can do. The plurality of image patches need not be image patches included in one image, but may be image patches included in each of different images (e.g., a wing part in a bird image, a wheel part in a car image). That is, different CN Result values (feature values, Z0) can be obtained from each of the different image patches (input images, 10).

The plurality of Z0 may correspond to a latent vector, and for the plurality of Z0, the first Z0, the second Z0... It can be set to KZ0. For example, four feature values (Z0, 230) can be derived from four image patches (input images), and each of these can be set as the first Z0, the second Z0, the third Z0, the fourth Z0, etc. There is.

The computing device performs MLPS (Multilayer Perceptrons), that is, multilayer perceptrons, on each of the derived first Z0, second Z0, third Z0, and fourth Z0, and then performs a plurality of Affine transformations (A1, A2, A3, …Ak) to perform Z1, Z2, Z3, Z4, … You can obtain Zk. At this time, the result values of the Affine transformation can be set as latent vectors.

Specifically, since the Z0 is plural (ex. 1st Z0, 2nd Z0, 3rd Z0, 4th Z0...), the plurality of Z0 is a plurality of Affine transformations (A1, A2, A3,...Ak), respectively. and operations are performed, and as a result, a plurality of Z1 (1st Z1, 2nd Z1, 3rd Z1, 4th Z1...), a plurality of Z2 (1st Z2, 2nd Z2, 3rd Z2, 4th Z2...) ), … A plurality of Zk (1st Zk, 2nd Zk, 3rd Zk, 4th Zk...) can be obtained, and the obtained result value can be set as a latent vector.

According to an embodiment of the present invention, the computing device may set the conditional vector (conditional code) as a fixed variable and proceed. In the case of the present invention, after learning about a lesion, outpainting is performed based on the lesion, so it is possible to set it as a fixed variable and proceed.

The computing device may input the latent vector (Z1) and the condition vector (c) into the first upscale module 241 and obtain 4x4 features. Additionally, the computing device may input the 4x4 features and Z2 (latent vector) to the second upscale module 242 and obtain 8x8 features, and input the 8x8 features and Z3 (latent vector) to the third upscale module (242). 243), 16x16 features can be obtained.

That is, the computing device sequentially passes the input value through a plurality of upscale modules and increases the number of pixels of the acquired image (Z1, Z2, Z3, ?? Zk) each time it passes the plurality of upscale modules. You can. The computing device can acquire 128x128 features if the last upscale module has been passed. In some cases, the size of the last feature may vary.

The computing device may perform a convolution operation on each of the plurality of features (4x4 features, 8x8 features, 16x16 features, ?? 128x128 features) and generate an image corresponding to the size of each feature. In other words, a 4x4-sized image 21 is created from 4x4 features, and an 8x8-sized image 22 is created from 8x8 features.

Specifically, the first Z0, second Z0, third Z0, and fourth Z0, which are CN (Context normalization) result values (feature values 230), are obtained from four image patches (input image, 10), respectively, In response to this, as a result of the affine transformation (A1), Z1 (latent vector) is obtained, and the Z1 (latent vector) is divided into the first Z1, the second Z1, the third Z1, and the fourth Z1, and the first upscale for each Each 4x4 feature can be acquired through the module 241, and ultimately four 4x4 sized images 21 can be generated. This also applies to other latent vectors (Z2, Z3, Z4), which will be described later.

For reference, Z1, Z2, Z3, Z4, … obtained by performing the plurality of affine transformations (A1, A2, A3, … Ak). Zk (latent vectors) can focus on different areas. That is, Z1 = A1(Z0), Z2 = A2(Z0), … It may correspond to Zk = Ak(Z0).

For example, assuming that the input image 10 is a human face, Z1 focuses on the human eye, and the 4x4 features corresponding to Z1 can also display an image centered on the human eye. Additionally, Z2 focuses on the human nose, and the 8x8 features corresponding to Z2 can also display images centered on the human nose. The focused area may not be set in advance but may correspond to results determined while repeatedly performing learning. Additionally, in an nxn feature, as n becomes larger, the meaning of Z becomes modifying details, and as it becomes smaller, it may correspond to drawing the overall structure.

As above, Z1, Z2, Z3, Z4, … Zk focuses on different parts, and the present invention can thereby have the effect of displaying more accurate details on the output image.

For convenience of explanation, it may be assumed that each image obtained as a result of a plurality of upscale modules is sequentially a first image and a second image. For example, an image generated from 4x4 features may be referred to as the first image 21, an image generated from 8x8 features may be referred to as the second image 22, and an image generated from 8x8 features may be referred to as the first image 22. The image generated from 16x16 features can also be referred to as the second image 23. That is, it can be assumed that they are the first image and the second image in order of size, and the first loss function will be explained below using this.

Figure 5 is a diagram showing the formulas of the first loss function and the second loss function according to an embodiment of the present invention.

In relation to the first loss function, the computing device can downsample the second image (ex 16x16 features) to be the same as the size of the first image (ex 8x8 features), and ultimately change the sizes of the first image and the second image to You can do the same. For reference, as described above, a latent vector is generated in each layer of the second decoder 240, and a plurality of images corresponding to each of a plurality of latent vectors (Z0) based on a plurality of image patches (input image 10) are generated. It can be.

At this time, among the plurality of images corresponding to 8x8 features, an image corresponding to a specific image patch (ex. the leg part of the new image) (ex. second Z2 basis), and the specific image patch (ex. the new image) among the plurality of images corresponding to 16x16 features. The calculation of the first loss function may be performed between the image (ex. second Z2 basis) corresponding to the leg portion of the image. In other words, the first loss function is calculated between features obtained from the same image patch.

In addition, the computing device downsamples the N x N size image passing through the i-th layer and converts it into an N/2 The size can be made the same as the image.

Next, the computing device can minimize the distance between the downsampled second image and the first image through the Euclidean distance function. The first loss function related equation (L_disent) can be referenced through FIG. 5(a). As can be seen in Figure 5(a), S is downsampling, d can mean Euclidean distance function, and the part modified at each step can be independent.

In the first loss function, L_disent can be calculated through a distance function (Euclidean distance function) and optimization (close to 0) can be performed. In other words, even when the upscale modules 241, 242, 243, 244, ?? are performed and new features (second images) are generated, it is possible to ensure that the difference from previous features (first images) does not exceed a certain range.

This is to maintain the unity of the second image generated through the upscale module with the existing first image, and ultimately to maintain the unity of the image generated as a result of outpainting and the input image.

Additionally, Conditional GAN Loss can be applied in addition to L_disent (first loss function) to each layer of the second decoder 240.

As a result of the second decoder 240, a fake image may be generated. In the learning process, it is necessary to express the loss by comparing the fake image with the real image, and in relation to this, you can refer to the formula (Conditional GAN Loss) below. For reference, in the case of a real image, it may correspond to the actual entire image (x) of the input image 10, and when referring to FIG. 1, the 'new' image will correspond to the real image (x). The Conditional GAN Loss may correspond to the formula below.

When referring to the above formula, it can be seen that a downsampling operation is performed on x, which is the original image of the image patch (input image, 10), and an operation on the nxn feature is performed. At this time, in the above formula, the Conditional GAN Loss can be obtained by calculating n discriminators for each of the downsampling operation and the operation on nxn features.

The above formula is a combination of the formula for the fake image and the formula for the real image. In the case of a fake image (operation on nxn features), the result value is 0, and in the case of a real image (down-sampling operation on x) It can represent an optimization process that ensures that the result is 1.

In addition, in the above formula, i is attached to the discriminator, and since the second decoder 240 of the present invention has a plurality of layers (ex. 6 - 4x4, 8x8, 16x16, 32x32, 64x64, 128x128, see FIG. 4), the i There may also be multiple numbers (ex. 6). Therefore, six discriminators may be used.

For reference, in the case of the above-described first loss function and the second loss function described later, the operation is performed on the loss of the generator, and in the case of the Conditional GAN Loss, the operation is performed on the loss of the discriminator. was carried out.

Additionally, the computing device may connect residual connections for each layer of the second decoder 240 and proceed with the equation of FIG. 5(a). Accordingly, the parts to be modified for each layer (ex. eyes, nose, etc.) change independently, so the details expressed for each layer may vary.

Next, we will look at the second loss function below.

Figure 6 is a diagram illustrating a visualization of the second loss function according to an embodiment of the present invention.

If learning is done only with input images (image patches), image diversity may not be secured. Therefore, during training, the second loss function can be calculated by inputting not only the image patch but also the ground truth image as input to the CPN (200). In other words, the ground truth image can be directly input to the CPN (200) without going through the FEN (100) to generate a representation vector. Afterwards, the second loss function (L_Ndiv) can be derived by putting the ground truth image patch and the ground truth image together in the second decoder 240. The equation expressed in FIG. 5(b) may correspond to the second loss function (L_Ndiv). Referring to FIG. 5(b), k may correspond to a layer, n may correspond to the number of layers, N may correspond to an image size pixel, D may correspond to a discriminator, and z may correspond to a latent vector.

Specifically, the computing device of the present invention acquires a plurality of feature values (Z0, 230) based on a plurality of image patches (input image, 10) during the learning process, and from these, a plurality of latent vectors (Z1, Z2, ? ?), so there are a plurality of images obtained as a result of inputting them to a specific upscale module, and the plurality of images are the 1-1 image (ex. based on the 1st Z2) and the 1-2 image (ex. based on the 2nd Z2) It can be assumed that it contains .

For example, 8x8 features may be obtained as a result of a specific up-scale module 242, and a plurality of corresponding images 22 may be generated based on this. At this time, the plurality of images 22 can be assumed to be the 1-1 image (ex. 1st Z2 basis) and the 1-2 image (ex. 2nd Z2 basis).

Above, as an example, only the up-scale module 242 that obtains 8x8 features is described, but the same can be applied to other up-scale modules (241, 243, etc.). As can be seen in FIG. 4, a plurality of images can be generated by each layer of the second decoder 240.

The second loss function (L_Ndiv) may correspond to a formula that supports generating various images by maximizing the distance between the 1-1 image and the 1-2 image. In other words, a latent vector (Z1, Z2, etc.) is generated from each layer of the second decoder 240, and there are a plurality of images generated from the latent vector (z), including images 1-1 and 1-2. It may correspond to an image.

At this time, the second loss function (L_Ndiv) minimizes the distance between latent vectors (z) and maximizes the pixel distance between images (1st-1st image, 1-2nd image) generated from the latent vector (z). It can be done for the purpose. Additionally, during the learning process, a ground truth image may be input and an image may be generated in each layer of the second decoder 240, which may be considered as the 1-1 image or the 1-2 image.

The diversity of the generated images can be secured by setting the goal of the second loss function to maximize the distance between the ground truth result generated image and the general latent vector (z) (so that the bidirectional points in Figure 6 are as far apart as possible).

In other words, a plurality of images (including images generated as ground truth results) are generated from a specific latent vector (z), and the pixel distance of each of the plurality of images is maximized to generate various images.

For example, when Z1 focuses on the human eye and 4x4 features and a plurality of images (1-1st image, 1-2nd image), etc. are generated through the upscale module 241, the plurality of Among the images, image 1-1 can intensively create round and large blue eyes, and image 1-2 can intensively generate narrow and long brown eyes. Images that focus on different parts may be obtained for other Z2, Z3, etc.

Ultimately, the diversity of the generated output images can be secured while ensuring that the result of the second loss function is optimized (close to 0).

The computing device can balance accuracy and diversity by giving the first loss function (L_discent) and the second loss function (L_Ndiv) a linear weight sum (loss function = a(L_discent) + b(L_Ndiv)), Learning about the second decoder 240 may proceed.

The computing device matches the input image 10 using the second decoder 240, which has been trained through the first loss function and the second loss function, and outputs an output image including an image whose area is expanded by the expanded size. You will be able to obtain (30).

The size of the expanded size can be determined through each layer of the first encoder 110 and first decoder 130 included in the FEN 100. In some cases, the computing device may make the expanded size of the human leg and arm areas larger than the human face area, and accordingly, the output image that is the result of outpainting also includes the human leg and arm areas. The image (generated image) may be larger than the image (generated image) containing the facial area.

Meanwhile, in the present invention, out painting can be performed using an image patch containing a lesion, such as a cancer photo, as the input image 10. At this time, the computing device can learn a separate determination module to determine in advance whether it is a lesion such as cancer. At this time, the decision module used may be various algorithms such as CNN, RNN, and GAN.

As a result, the computing device can perform the FEN (100) and CPN (200) processes only for image patches that are determined to be lesions through the judgment module, and FEN (100) and CPN (200) for image patches that are not determined to be lesions. (200) The process may not be performed.

In addition, the computing device may be set in advance for a purpose (ex. skin cancer, breast cancer, liver cancer, stomach cancer, etc.), and determines whether an image patch (input image) corresponds to the set specific purpose (ex. breast cancer) through a judgment module. The FEN (100) and CPN (200) processes may be performed only when applicable to a specific purpose.

The above processes can be referred to Wide-Context Semantic Image Extraction (2019, Yi Wang…) and Nested Scale-Editing for Conditional Image Synthesis (2020, Lingzhi Zhang…).

Embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person skilled in the art to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

10: Input image
20: FEN result
21, 22, 23, 24: Images generated during the calculation process of the second decoder
30: Output image
40: Extended input image
100: Feature Expansion Network (FEN)
110: first encoder
120: first bottleneck
130: first decoder
200: Context Prediction Network (CPN)
210: first encoder
220: second bottleneck
230: Feature value
240: second decoder
241, 242, 243, 344: Upscale module included in the second decoder

Claims (5)

In the image out painting method,
There is a Semantic Regeneration Network (SRN) including a Feature Expansion Network (FEN) and a Context Prediction Network (CPN), the FEN includes a first encoder and a first decoder, and the CPN includes a second encoder and a second decoder. In a state containing,
(a) A computing device generates a first bottleneck by inputting an input image into the first encoder of the FEN, and inputs the first bottleneck into the first decoder to produce a FEN result having a size larger than the input image. Obtaining;
(b) The computing device merges the FEN result with an extended input image including the input image but has an extended area, and inputs the result to the second encoder of the CPN to generate a second bottleneck. and inputting the feature value obtained using the second bottleneck into the second decoder; and
(c) The computing device matches the input image using the second decoder that has completed learning through the first loss function and the second loss function, and an output image including an image whose area is expanded by the expanded size. Obtaining a;
Includes,
A plurality of different input images are input to the first encoder,
The second decoder includes a plurality of upscale modules,
The first loss function is applied between a plurality of images obtained by sequentially passing one of the plurality of input images through the plurality of upscale modules,
The second loss function is applied between a plurality of result values corresponding to each of the plurality of input images for one of the plurality of upscale modules,
A method in which the second decoder is learned through the sum of linear weights for the first loss function and the second loss function. According to paragraph 1,
The context normalization result value, which is the feature value acquired using the second bottleneck, and the condition vector derived from the second encoder are input to the second decoder and sequentially pass through a plurality of upscale modules included in the second decoder. do,
A method characterized in that the number of pixels of the acquired image increases each time it passes through the plurality of upscale modules. According to paragraph 2,
When each image obtained as a result of the plurality of upscale modules is sequentially referred to as a first image and a second image,
The first loss function downsamples the second image and minimizes the distance between the downsampled second image and the first image, so that even when the upscale module is performed, the difference between the second image and the first image is A method characterized in that it corresponds to a formula that supports ensuring that does not exceed a certain range. According to paragraph 2,
When there are a plurality of images obtained as a result of a specific upscale module, and the plurality of images include a 1-1 image and a 1-2 image,
The second loss function corresponds to a formula that supports generating various images by maximizing the distance between the 1-1 image and the 1-2 image. In a device that performs image out painting,
There is a Semantic Regeneration Network (SRN) including a Feature Expansion Network (FEN) and a Context Prediction Network (CPN), the FEN includes a first encoder and a first decoder, and the CPN includes a second encoder and a second decoder. In a state containing,
Ministry of Communications;
database; and
An input image is input to the first encoder of the FEN to generate a first bottleneck, the first bottleneck is input to the first decoder to obtain a FEN result having an expanded size than the input image, and the input image is input to the first encoder. Merges the FEN result with an extended input image including an extended area, inputs the result to the second encoder of the CPN to generate a second bottleneck, and generates a second bottleneck using the second bottleneck. Feature values are input to the second decoder, and an image whose area is expanded by the expanded size is included while matching the input image using the second decoder that has completed learning through the first loss function and the second loss function. a processor that acquires an output image;
Includes,
A plurality of different input images are input to the first encoder,
The second decoder includes a plurality of upscale modules,
The first loss function is applied between a plurality of images obtained by sequentially passing one of the plurality of input images through the plurality of upscale modules,
The second loss function is applied between a plurality of result values corresponding to each of the plurality of input images for one of the plurality of upscale modules,
The second decoder is a computing device that is learned through the sum of linear weights for the first loss function and the second loss function.