KR102587233B1 - 360 rgbd image synthesis from a sparse set of images with narrow field-of-view - Google Patents

Abstract

A method and device for synthesizing a 360 RGBD image from a small number of narrow-angle-of-view RGBD images are disclosed. An image synthesis method according to an embodiment includes generating a field of view image by estimating a field of view (FoV) relative to a panoramic image using a field of view estimation network, and generating the field of view image using a panorama generation network. It may include generating a panoramic image from.

Description

360 RGBD image synthesis from a small number of narrow field of view RGBD images {360 RGBD IMAGE SYNTHESIS FROM A SPARSE SET OF IMAGES WITH NARROW FIELD-OF-VIEW}

The description below relates to an image synthesis method and device for synthesizing a 360 RGBD image from a small number of narrow field of view RGBD images.

Recently, interest in virtual reality and augmented reality technologies, which are realistic media that provide a sense of immersion, is rapidly increasing. 3D scene understanding is a very important element in these fields, and one of the most basic research areas is depth estimation research. A depth image is an image that expresses depth information in a 3D space on a 2D plane, and contains 3D structural information of the space, so it is used in various 3D vision fields such as view synthesis, 3D modeling, autonomous driving, and robotics.

When acquiring a scene using an existing narrow FoV camera, an image corresponding to a portion of the surrounding scene is obtained with a significant portion of the surrounding scene missing. In order to acquire an omnidirectional scene, multiple cameras must be built and multiple images must be processed. Because of this, a lot of costs are incurred. Alternatively, omnidirectional images can be acquired using a small number of fisheye lens cameras with a wide FoV of 180° or more. Since the fisheye lens is a spherical model and it is difficult to utilize the existing pinhole camera-based flat model as is, a spherical projection image such as an equirectangular projection image can be used. However, these 360° projection images may have distortion problems at the boundaries and poles, and recently, a deep learning network that reflects the characteristics of these 360° projection images has been proposed.

Depth estimation research that has been studied for a long time includes depth estimation using traditional stereo images, depth estimation for monocular images using deep learning networks, and a mixture of these two techniques. Most of these techniques target monocular images with a narrow FoV. In addition, unlike general image datasets, high-quality 360° datasets acquired using sensors based on narrow FoV images are not sufficient. In particular, depth images have high precision in pixels from which depth information is acquired, but are suitable for deep learning network training. There are many cases where sparse depth images that have not been used are acquired. For this reason, research has been conducted to complete sparse depth images into dense depth images, and images are acquired by newly arranging cameras to acquire stereo images, but this also has the disadvantage of being difficult to maintain a consistent baseline and high operation costs. There is.

[Prior document number]

Korean Patent Publication No. 10-2020-0095112

Provided is an image synthesis method and device for synthesizing 360 RGBD images from a small number of narrow-angle-of-view RGBD images.

In the image synthesis method of a computer device including at least one processor, the field of view image is generated by estimating a field of view (FoV) relative to the panoramic image using a field of view estimation network by the at least one processor. generating step; and generating, by the at least one processor, a panoramic image from the generated field of view image using a panorama generation network.

According to one side, the panorama generation network may be characterized as including a U-Net-based adversarial generation neural network network.

According to another aspect, the panorama generation network may be trained using a Least Squares Generative Adversarial Network (LSGAN) as an adversarial loss function.

According to another aspect, the loss function of the panorama generation network includes a first loss function for RGB of the RGBD (Red Green Blue Depth) network and a second loss function for the depth of the RGBD network. It can be characterized.

According to another aspect, the first loss function includes a first adversarial loss function for RGB of the RGBD network, a pixel loss function between an image generated by a generator for RGB of the RGBD network and a corresponding true value image, the RGBD It may be determined using a perceptual loss objective function for RGB of the network and a Frechet distance loss function measured between the generated image and the true value image.

According to another aspect, the second loss function includes a second adversarial loss function for the depth of the RGBD network, a pixel loss function between the image generated by the generator for the depth of the RGBD network and the corresponding true value image, the RGBD It may be characterized by being determined using a perceptual loss objective function for the depth of the network and a Frechett distance loss function measured between the generated image and the true value image.

According to another aspect, the panorama generation network is binary in the remaining portion excluding the true value region of the input image for the image generated by the RGB generator of the RGBD network and the image generated by the depth generator of the RGBD network. It may share the characteristics of the masked image, and the output of the last layer of the panorama generation network may be transmitted to the decoder of the RGBD network by performing channel connection to the last block of the RGBD network.

A computer program stored on a computer-readable recording medium is provided in conjunction with a computer device to execute the method on the computer device.

Provided is a computer-readable recording medium on which a program for executing the above method on a computer device is recorded.

A computer device, comprising: at least one processor configured to execute instructions readable by the computer device, wherein the at least one processor determines a field of view relative to a panoramic image using a field of view estimation network. A computer device is provided, which generates a field of view image by estimating FoV) and generates a panoramic image from the generated field of view image using a panorama generation network.

An image synthesis method and device for synthesizing a 360 RGBD image from a small number of narrow-angle-of-view RGBD images can be provided.

Figure 1 is a diagram showing an example of the overall structure of an RGBD generation network according to an embodiment of the present invention.
Figure 2 is a diagram illustrating an example of a feature sharing module of an RGBD generation network according to an embodiment of the present invention.
Figure 3 is a diagram showing an example of a sample of a refined dataset (3D60 dataset sample).
Figure 4 is a diagram showing an example of qualitative evaluation results of the generated RGBD image.
Figure 5 is a diagram showing an example of a 3D point cloud result of the generated RGBD image.
Figure 6 is a diagram illustrating an example of the results compared by applying the results before and after using the Frechette distance loss function to a network before and after using the fusion module.
Figure 7 is a diagram showing an example of the verification result of the Pritchett distance.
Figure 8 is a diagram showing an example of a qualitative comparison result of RGB image results with existing techniques.
Figure 9 is a diagram showing an example of a qualitative comparison result with existing techniques for depth image results.
Figure 10 is a block diagram showing an example of a computer device according to an embodiment of the present invention.
Figure 11 is a flowchart showing an example of an image synthesis method according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the following description and accompanying drawings, detailed descriptions of well-known functions and configurations that may unnecessarily obscure the gist of the present invention are omitted.

A depth image is an image that displays distance information in 3D space on a 2D plane and is useful in various 3D vision research. Many existing depth estimation studies mainly use narrow Field of View (FoV) images to estimate depth information for images in which a significant portion of the entire scene is missing. Embodiments of the present invention provide a method and device for simultaneously generating 360° omnidirectional RGBD images from a small number of narrow FoV images. For example, it is possible to provide an image generation model based on an adversarial generation neural network that estimates the relative FoV for the entire panoramic image from four non-overlapping minority images and simultaneously generates a 360° color image and a depth image, and provides features of the two modalities. Share to confirm mutually complementary results. In addition, improved performance is achieved by constructing a network that reflects the spherical characteristics of 360° images.

More specifically, embodiments of the present invention can provide a deep learning network based on a convolutional neural network (CNN) that generates an RGBD image in consideration of existing problems, and n non-overlapping sheets (for example, 4 sheets) can be provided. A framework for generating a 360° RGBD panoramic image can be provided for narrow FoV images. In the embodiments of the present invention, it is shown that it is a generalized network for the 3D60 360° indoor dataset, and it shows improved results than the existing single network through the interaction of the two modalities. In addition, the Frechet distance was calculated by extracting features considering 360° image characteristics, and the loss function was used to improve restoration performance.

The main contributions of the embodiments of the present invention are summarized as follows.

·Generalize the existing outdoor video-based network using the 3D60 indoor dataset.

·Simultaneously generate color (RGB) images and depth (D) images using a U-Net-based adversarial generative neural network.

·RGBD shows complementary performance by sharing the characteristics of the two modalities.

·Improved results are shown through the Frechett distance loss function using features that reflect 360° image characteristics.

1. Related research

1-1. Dense depth estimation study for spherical models

As for depth estimation research using 360° images, similar to depth estimation research in existing narrow FoV images, there are traditional stereo image-based depth estimation techniques and deep learning-based depth estimation research techniques for monocular images. 360SD-Net is a deep learning-based depth estimation network using spherical stereo images. It uses stereo images obtained by setting cameras in a top-bottom structure and polar angle images to solve the distortion problem of spherical panoramas, and reduces cost. A learnable shifting filter was proposed for volume construction. OmniDepth is a deep learning-based depth estimation study for monocular images and proposed a network that takes into account the characteristics of 360° images with an autoencoder structure. A spherical panorama has the characteristic of becoming more distorted toward both poles. For this structure, a network was designed using dilated convolution for global context and 1Х1 convolution for spatial correlation. BiFuse is also a depth estimation study for monocular images based on deep learning, and to improve the boundary problem caused by the equirectangular format and cubemap format in spherical panoramas, a module that fuses the two formats was proposed and a network was designed. Additionally, a spherical padding technique was proposed in the cube map format to reduce distortion of spherical panoramic images. In a study on depth completion for dense depth estimation, we proposed a network that generates and assists gray-scale images to generate depth images using a feature sharing module. In embodiments of the present invention, a network was designed based on a method that uses the fused characteristics of the two modalities.

1-2. Mask-based convolutional neural network

The study of applying masks to input images and filling in damaged areas is an old research topic and can be used in tasks such as video editing. In a traditional way, the diffusion-based technique uses a reference image to fill in information from surrounding areas, and the patch-based technique uses pixel information from the undamaged area of the input image. The method is to import and fill in the empty area. These methods are dependent on the input image, have poor consistency in the overall structure, and have limitations that make it difficult to generate semantic information. With the recent development of deep learning networks, the generative model is widely used in mask-based image generation research in that, unlike existing methods, it is possible to generate consistent images including semantic information for the entire structure without relying on the input image. . StructureFlow is a deep learning-based network that randomly generates and applies a binary mask to the image and generates missing pixels. It generates an edge-preserve smooth image for structural information and generates detailed texture information based on this to create the entire image. proposed a technique to restore . MED is an encoder-decoder based image generation network that applies a multi-scale kernel to fuse the texture information of the shallow layer and the structural information of the deep layer, then performs smoothing and adds it to the decoder to restore irregular mask areas. Perform.

Although many depth estimation studies on 360° images are being conducted, research that generates omnidirectional images for input images that do not overlap with existing studies using narrow FoV as input is rare. In addition, most image generation studies use binary mask images that are irregular and sporadically distributed throughout the image, but in embodiments of the present invention, a large block-shaped mask is used to overcome difficult conditions by generating images in an environment without much surrounding information. Shows the network.

2. Embodiments of the present invention

The network according to embodiments of the present invention may be composed of a network that estimates the relative FoV for a 360° image and a panorama generation network that generates a 360° image from the estimated FoV image. The network that estimates the relative FoV and the panorama generation network go through separate training processes, and a mask can be applied to the resulting image generated after the FoV estimation step and a panorama can be generated for the input transformed into an equirectangular projection image format. The panorama creation result is an interactive result that shares the characteristics of RGBD.

Figure 1 is a diagram showing an example of the overall structure of an RGBD generation network according to an embodiment of the present invention, and Figure 2 is a diagram showing an example of a feature sharing module of an RGBD generation network according to an embodiment of the present invention. .

2-1. Relative FoV estimation

The step of estimating the relative FoV for the entire panorama is performed before creating the panorama, and can be modeled as an estimation problem of the size occupied by the entire panoramic image rather than an absolute angle. Four horizontal observation direction images can be used as input, and the bottleneck layer can output 256 classes corresponding to FoV angles and perform classification tasks. The cross-entropy loss function can be used as the objective function for the classification task, and the L1 distance objective function with the ground truth mask can be used by generating a mask image with padding for the FoV angle in the decoder layer. The existing network can be expanded and applied to RGBD images, and in the next step, the panorama generation step, under the assumption that the FoV estimation network estimates the accurate FoV, it is difficult to generate the highest quality image among the mask ratios commonly used in recent related research. An experiment was conducted on input with a FoV of 60° (mask ratio approximately 57%). In mask-based image generation research, masks between 0-60% are generated and evaluation results are shown according to the ratio. The larger the mask ratio, the larger the area to be created, and the lower the performance. In addition, in an experiment conducted by configuring the mask in a random shape, a grid shape, and a block shape similar to the embodiments of the present invention, the best result was obtained in the random mask composition (75% ratio), and the block mask composition (50% ratio) ratio) showed the lowest performance when compared to other mask types.

2-2. Create a panorama

The panorama generation network may be a U-Net-based adversarial generative neural network network. During training, LSGAN (Least Squares GAN (Generative Adversarial Network)) is used as the adversarial loss function, and the adversarial loss functions L _adv1 and L _adv2 for each RGBD network are as shown in Equation 1 and Equation 2 below.

[Equation 1]

[Equation 2]

G _rgb , D _rgb , G _depth , and D _depth may be the generator and identifier of the RGBD network, respectively. Also, I _i , , R _i , may be the input of G _rgb and G _depth and the generated image, respectively, and I _p and R _p may be the true value image for the generated image.

The L1 loss function can be used as the pixel loss function between the generated image and the true image. Next, L _pix1 and L _pix2 are pixel loss functions of the RGBD network and can be expressed as Equations 3 and 4 below.

[Equation 3]

[Equation 4]

Additionally, to generate realistic images, a VGG (Visual Geometry Group) network pre-trained with a perceptual loss objective function can be used. Next, L _vgg1 and L _vgg2 are the perceptual loss objective functions of the RGBD network and can be expressed as Equations 5 and 6 below. VGG (·) may be the ith feature vector extracted from the VGG network.

[Equation 5]

[Equation 6]

In order to extract features reflecting 360° image features, train a model according to embodiments of the present invention and use a pre-trained model to extract the features of the last layer of the module latent space in which RGBD features are shared. can be extracted. Then the longitudinal invariant feature and latitudinal equivariant features can be extracted. The longitudinal invariance feature and the transverse isovariability feature can be expressed as Equation 7 and Equation 8, respectively. c , w , and h can refer to each channel, width, and height. Unlike the VGG loss objective function, which improves similarity by calculating the feature map distance of a single image, it measures the distribution distance between the generated image dataset and the ground truth image dataset, and is fused using the last layer of the Fusion module. The similarity of the RGBD feature vector distribution can be reflected.

[Equation 7]

[Equation 8]

And the Frechette distance can be measured and used as a loss function. Frechet Street is the Pritchett distance between the Mean ( m , C ) Gaussian of the ground truth image and the Mean ( m' , C' ) Gaussian of the generated image, and can be expressed as Equation 9 below. m , C , m' , and C' may be the mean and covariance of the ground truth image and the generated image, respectively. The Frechette distance loss function is the value measured at the 0th iteration during training. It can be divided and normalized as shown in Equation 10 below.

[Equation 9]

[Equation 10]

Therefore, the overall loss functions L _rgb and L _depth of the panorama generation network are as shown in Equation 11 and Equation 12 below, where λ ₁ , λ ₂ , and λ ₃ may be constant factors.

[Equation 11]

[Equation 12]

For the images generated from each RGBD network, they share the features of the image to which the binary mask corresponding to the remaining portion excluding the true value region of the input image is applied, and each feature can be expressed as Equation 13 and Equation 14 below.

[Equation 13]

[Equation 14]

F () can be an encoder, and M can be a binary mask corresponding to the largest FoV class. Each layer of the RGBD network performs pixel summation, which can be expressed as Equation 15 and Equation 16 below. The last layer f _s can be delivered to each decoder by performing channel connection to the last block of each RGBD network. may be a channel-based connection function. The proposed feature sharing module is shown in Figure 2.

[Equation 15]

[Equation 16]

2-3. Experiment result

A total of 21,600 indoor datasets such as Matterport3D, Stanford-3D, and SunCG go through the process of removing 3,446 datasets that are inappropriate for training with damaged areas exceeding a certain value based on the depth images, and a total of 18,154 datasets are trained. It is divided into dataset (80%) and test dataset (20%).

Figure 3 is a diagram showing an example of a sample of a refined dataset (3D60 dataset sample). In Figure 3, rows 1 and 2 are RGBD image samples, and row 3 is a map visualizing the damaged area based on the depth image.

In network training, an adaptive momenteum (ADAM) optimizer is used, learning rate α = 0.0002, β ₁ = 0.5, β ₂ = 0.99, batch size is set to 2, and NVIDIA RTX A6000 GPU is used. trained using it. The Matterport3D, Stanford3D, and SunCG datasets were converted to cube map format to construct a dataset so that four surfaces could be used for training, and each of the three datasets was trained separately.

In quantitative evaluation, PSNR (Peak Signal-to-Noise Ratio) for the similarity of the generated model, SSIM (Structural Similarity Index Map) and absolute difference (Abs Diff), absolute relative error (Abs Rel) for similarity evaluation of depth images, The square relative error (Sq Rel), root mean square error (RMS), log root mean square error (RMS log), and relative accuracy (δ) were measured. The mask ratio used in the evaluation was evaluated for the FoV 60° input, which is the mask ratio for the widest area frequently used in recent related research.

For qualitative evaluation, a network using the proposed fusion block was compared with a network separately constructed with RGBD without using a fusion block, and the results before and after using the Frechette distance loss function were compared. Additionally, a point cloud for the generated RGBD image was created and compared.

Figure 4 is a diagram showing an example of qualitative evaluation results of the generated RGBD image. Figure 4 qualitatively compares the results of a network according to an embodiment of the present invention and a network that does not use a feature sharing module. In Figure 4, rows 1 and 3 represent color images, and rows 2 and 4 represent depth image samples. It can be seen from the results of the network according to the embodiments of the present invention that the overall layout and detailed information are well generated.

Figure 5 is a diagram showing an example of a 3D point cloud result of the generated RGBD image. In Figure 5, the 3D point cloud results are compared and evaluated, and through this, the effects of 3D layout and noise, which are difficult to qualitatively compare in the generated depth images, can be confirmed.

Table 1 shows the results of various quantitative evaluations of the generated depth images, and compared them before and after using the feature sharing module.

[Table 1]

Tables 2 and 3 show the results before and after using the Frechet distance loss function. Table 2 shows the RGBD image evaluation results without using Frechette distance loss, and Table 3 shows the RGBD image evaluation results with Frechette distance loss applied.

[Table 2]

[Table 3]

When comparing the generated RGBD images with the results of a network that did not use the feature sharing module, the images generated by the proposed network show better results, and even better results can be seen when compared to before using the Frechette distance loss function. there is.

Figure 6 is a diagram illustrating an example of the results compared by applying the results before and after using the Frechette distance loss function to a network before and after using the fusion module. In Figure 6, rows 1 and 2 and rows 3 and 4 are RGBD image sample results.

To verify the Pritchett distance, features are extracted from the pre-trained model of the proposed model, and Gaussian blur and salt and pepper noise are generated for the panoramic image in four steps. The distance was measured.

Figure 7 is a diagram showing an example of the verification result of the Pritchett distance. In Figure 7, it can be seen that the distance between the feature distribution of the ground truth image and the feature distribution of the generated image increases as the level of disturbance increases for both RGBD images. Features are extracted using the proposed network and the Frechett distance is calculated. When calculated, it can be confirmed that blur and noise are captured normally. (1)-(4) may refer to each disability level.

In addition, we experimented with the proposed feature sharing module in three additional cases: when no residual blocks were used, when only one residual block was used, and when direct connection was performed. Table 4 shows the quantitative evaluation results for the three configurations for the network without Frechett distance loss.

[Table 4]

Overall, better results can be seen when using the three feature sharing modules than when not using the feature sharing module. For comparison with other models, we compared the inpainting model and the existing model w/o Fusion network as a model for RGB image generation. Both models were retrained under the same random FoV, and quantitative and qualitative evaluation was performed on FoV 60°. And to compare depth images, we compared the trained model of the depth estimation model, the inpainting model trained again under the same conditions, and the existing model.

In Table 5 and Table 6, the average PSNR and SSIM for 1,759 Matterport3D datasets were calculated and compared. Table 5 shows an example of comparing the RGB image generation results with existing techniques, and Table 6 shows an example of comparing the depth image generation results with existing techniques.

[Table 5]

[Table 6]

FIG. 8 is a diagram showing an example of a qualitative comparison result with existing techniques for RGB image results, and FIG. 9 is a diagram showing an example of a qualitative comparison result with existing techniques for depth image results. In the case of depth images, the Top and Bottom parts were excluded in order to compare with other models. In addition, the depth image generated using the true RGB image as input was compared with the depth image generated using the RGB image of the proposed model as input.

Embodiments of the present invention can provide an adversarial generative neural network-based network that simultaneously generates RGBD images from a small number of images. For the generative model that shared the characteristics of the two modalities, improved performance over a single network was quantitatively and qualitatively confirmed, and improved performance was shown by applying the Frechett distance loss function that reflects 360° image features. In addition, through an ablation study on the feature sharing module, higher performance was confirmed than that of the existing single network, and among them, the proposed structure was found to show good performance. Even when compared to the existing RGBD image generation model, it was confirmed that it showed excellent performance qualitatively and quantitatively. Unlike existing panoramic image generation models, it is different in that it simultaneously generates high-quality RGBD images from a small number of non-overlapping images with a high ratio mask applied, reflects 360° features, and has complementary results of RGBD. It can contribute to complex 3D scene reconstruction through high-quality RGBD image generation.

The image synthesis device according to embodiments of the present invention may be implemented by at least one computer device. At this time, the computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the image compositing method according to the embodiment of the present invention under the control of the driven computer program. . The above-described computer program can be combined with a computer device and stored in a computer-readable recording medium to execute the image synthesis method on the computer.

FIG. 10 is a block diagram showing an example of a computer device according to an embodiment of the present invention, and FIG. 11 is a flowchart showing an example of an image synthesis method according to an embodiment of the present invention. As shown in FIG. 10, the computer device (1000) includes a memory (1010), a processor (1020), a communication interface (Communication interface, 1030), and an input/output interface (I/O interface, 1040). may include. The memory 1010 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, non-perishable large-capacity recording devices such as ROM and disk drives may be included in the computer device 1000 as a separate permanent storage device that is distinct from the memory 1010. Additionally, an operating system and at least one program code may be stored in the memory 1010. These software components may be loaded into the memory 1010 from a computer-readable recording medium separate from the memory 1010. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 1010 through the communication interface 1030 rather than a computer-readable recording medium. For example, software components may be loaded into the memory 1010 of the computer device 1000 based on a computer program installed by files received through a network (Network, 1060).

The processor 1020 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 1020 by the memory 1010 or the communication interface 1030. For example, the processor 1020 may be configured to execute received instructions according to program codes stored in a recording device such as memory 1010.

The communication interface 1030 may provide a function for the computer device 1000 to communicate with other devices (eg, the storage devices described above) through the network 1060. For example, requests, commands, data, files, etc. generated by the processor 1020 of the computer device 1000 according to the program code stored in a recording device such as memory 1010 are transmitted to the network ( 1060) and can be transmitted to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 1000 through the communication interface 1030 of the computer device 1000 via the network 1060. Signals, commands, data, etc. received through the communication interface 1030 may be transmitted to the processor 1020 or memory 1010, and files, etc. may be stored in a storage medium (as described above) that the computer device 1000 may further include. It can be stored as a permanent storage device).

The input/output interface 1040 may be a means for interfacing with an input/output device (I/O device, 1050). For example, input devices may include devices such as a microphone, keyboard, or mouse, and output devices may include devices such as displays and speakers. As another example, the input/output interface 1040 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen. At least one of the input/output devices 1050 may be configured as one device with the computer device 1000. For example, like a smart phone, the computer device 1000 may include a touch screen, microphone, speaker, etc.

Additionally, in other embodiments, computer device 1000 may include fewer or more components than those of FIG. 10 . However, there is no need to clearly show most prior art components. For example, the computer device 1000 may be implemented to include at least a portion of the input/output device 1050 described above, or may further include other components such as a transceiver, a database, etc.

The image synthesis method according to this embodiment can be performed by a computer device 1000 that implements an image synthesis device. At this time, the processor 1020 of the computer device 1000 may be implemented to execute control instructions according to the code of an operating system included in the memory 1010 or the code of at least one computer program. Here, the processor 1020 causes the computer device 1000 to perform steps 1110 and 1120 included in the method of FIG. 11 according to control instructions provided by code stored in the computer device 1000. can be controlled.

In step 1110, the computer device 1000 may generate a field of view image by estimating a field of view (FoV) relative to the panoramic image using a field of view estimation network. As already explained, the problem of estimating the field of view is performed before creating a panorama and can be modeled as a problem of estimating the size occupied by the entire panoramic image rather than an absolute angle. n (for example, n=4) horizontal observation direction images can be used as input, and the bottleneck layer of the field of view estimation network can output 256 classes corresponding to the field of view angles and perform a classification task. The cross-entropy loss function can be used as the objective function for the classification task, and the L1 distance objective function with the ground truth mask can be used by generating a mask image with added padding for the viewing angle in the decoder layer. The existing network can be expanded and applied to RGBD images.

In step 1120, the computer device 1000 may generate a panoramic image from a field of view image generated using a panorama generation network. In one embodiment, the panorama generation network may include a U-Net-based adversarial generative neural network, and the panorama generation network uses Least Squares Generative Adversarial Network (LSGAN) as an adversarial loss function. So it can be pre-trained.

Meanwhile, the loss function of the panorama generation network may include a first loss function for RGB of the RGBD (Red Green Blue Depth) network and a second loss function for the depth of the RGBD network. For example, the first loss function may correspond to L _rgb in Equation 11, and the second loss function may correspond to L _depth in Equation 12. As shown in Equation 11, the first loss function is a first adversarial loss function for RGB of the RGBD network (e.g., L _adv1 in Equation 1), an image generated by the generator for RGB of the RGBD network, and A pixel loss function between the corresponding true value images (e.g., L _pix1 in Equation 3), a perceptual loss objective function for RGB of the RGBD network (e.g., L _vgg1 in Equation 5), and the generated image and the true value. The Frechet distance measured between images can be determined using a loss function (for example, L _d1 in Equation 11). In addition, as shown in Equation 12, the second loss function is a second adversarial loss function for the depth of the RGBD network (e.g., L _adv2 in Equation 2), an image generated by the generator for the depth of the RGBD network, and Pixel loss function between the corresponding true images (e.g., L _pix2 in Equation 4), perceptual loss objective function for depth of the RGBD network (e.g., L _vgg2 in Equation 6), and measurement between the generated image and the true image. It can be determined using the Frechett distance loss function (for example, L _d2 in Equation 12).

In addition, as explained through the feature sharing model of FIG. 2 and Equations 13 to 16, the panorama generation network is an image generated by the RGB generator of the RGBD network and the depth generator of the RGBD network. The image shares the characteristics of the image in which a binary mask is applied to the remaining part of the input image except for the true value region, and the output of the last layer of the panorama generation network is transmitted to the decoder of the RGBD network by performing channel connection to the last block of the RGBD network. It can be.

The system or device described above may be implemented with hardware components or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various other software, or servers. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments and equivalents of the claims also fall within the scope of the following claims.

Claims (10)

In an image synthesis method of a computer device including at least one processor,
generating a field of view image by estimating, by the at least one processor, a field of view (FoV) relative to the panoramic image using a field of view estimation network; and
Generating, by the at least one processor, a panoramic image from the generated field of view image using a panorama generation network.
Including,
The loss function of the panorama generation network includes a first loss function for RGB of the RGBD (Red Green Blue Depth) network and a second loss function for the depth of the RGBD network. . According to paragraph 1,
An image synthesis method, wherein the panorama generation network includes a U-Net-based adversarial generation neural network network. According to paragraph 1,
An image synthesis method, characterized in that the panorama generation network is trained using LSGAN (Least Squares GAN (Generative Adversarial Network)) as an adversarial loss function. delete According to paragraph 1,
The first loss function is a first adversarial loss function for RGB of the RGBD network, a pixel loss function between the image generated by the generator for RGB of the RGBD network and the corresponding true value image, and the perception of RGB of the RGBD network. An image synthesis method characterized in that it is determined using a loss objective function and a Frechet distance loss function measured between the generated image and the true value image. According to paragraph 1,
The second loss function is a second adversarial loss function for the depth of the RGBD network, a pixel loss function between the image generated by the generator for the depth of the RGBD network and the corresponding true value image, and the perception of the depth of the RGBD network. An image synthesis method characterized in that it is determined using a loss objective function and a Frechett distance loss function measured between the generated image and the true value image. In an image synthesis method of a computer device including at least one processor,
generating a field of view image by estimating, by the at least one processor, a field of view (FoV) relative to the panoramic image using a field of view estimation network; and
Generating, by the at least one processor, a panoramic image from the generated field of view image using a panorama generation network.
Including,
The panorama generation network is an image generated by the RGB generator of the RGBD network and the image generated by the depth generator of the RGBD network. The characteristics of the image by applying a binary mask to the remaining portion excluding the true value region of the input image. Share it,
An image synthesis method wherein the output of the last layer of the panorama generation network is transmitted to a decoder of the RGBD network by performing channel connection to the last block of the RGBD network. A computer program coupled to a computer device and stored in a computer-readable recording medium to execute the method of any one of claims 1 to 3 or 5 to 7 on the computer device. A computer-readable recording medium recording a computer program for executing the method of any one of claims 1 to 3 or 5 to 7 on a computer device. In computer devices,
At least one processor implemented to execute readable instructions on the computer device
Including,
By the at least one processor,
A field of view image is generated by estimating the relative field of view (FoV) for the panoramic image using a field of view estimation network,
Generating a panoramic image from the generated field of view image using a panorama generation network,
The loss function of the panorama generation network includes a first loss function for RGB of the RGBD (Red Green Blue Depth) network and a second loss function for the depth of the RGBD network.
A computer device characterized by a.