KR102708278B1 - Apparatus and method of video stitching using depth based seam regeneration for seam optimized video stitching - Google Patents

Abstract

The present invention relates to a video stitching device and method utilizing depth-based regeneration of a synthetic boundary line. According to an embodiment of the present invention, the video stitching device includes an image registration unit which receives a plurality of input images for a video frame, processes a common region image for the plurality of input images, and registers a stitching target image; a boundary generation unit which configures a boundary line generation matrix through characteristic information for the registered stitching target images, and generates a synthetic boundary line based on the configured boundary line generation matrix; a boundary line regeneration unit which calculates a depth information matrix according to parallax for the plurality of input images, extracts boundary line position information according to the boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and an image synthesis unit which synthesizes the plurality of input images based on one of the synthetic boundary lines among the generated synthetic boundary line and the synthetic boundary line regenerated according to the regeneration decision. Can be.

Description

{APPARATUS AND METHOD OF VIDEO STITCHING USING DEPTH BASED SEAM REGENERATION FOR SEAM OPTIMIZED VIDEO STITCHING}

The present invention relates to a video stitching device and method utilizing depth-based regeneration of a synthetic boundary line, and more specifically, to a video stitching technique for selectively regenerating a synthetic boundary line for a plurality of video frames by determining whether or not to regenerate a synthetic boundary line using a depth mask based on depth information according to parallax for a plurality of input images and a residual mask using two consecutive images from a plurality of input images.

Image stitching is a technology that synthesizes images taken from multiple cameras into an image of a single point in time, and can be used to create 360-degree and panoramic images.

One of the detailed techniques of the image stitching technology that synthesizes multiple images into one image is the image stitching method that utilizes a synthetic boundary line.

The video stitching technology starts with extracting keypoints that can be easily identified from multiple input images despite changes in size, rotation, camera viewpoint, and illumination, and then proceeds with keypoint matching, common area calculation, homography estimation, and warping.

In addition, for the warped common area image, the path with the minimum accumulated value in the boundary generation matrix composed of the difference between images or the visual perception energy function is set as the synthetic boundary.

The composite boundary line is used as a reference line for synthesizing images, and any distortion in the synthesized images caused by differences in illumination between cameras is removed through the blending process.

In the video stitching process, generating a synthetic boundary along a path with a minimum accumulated value of the boundary generation matrix is intended to minimize parallax distortion. Therefore, in order to stitch a video so that parallax distortion is minimized, it is necessary to generate a new synthetic boundary for every frame of the video.

However, since generating a new synthetic boundary for all video frames is computationally expensive, an efficient video stitching method that can apply video stitching only to specific video frames is needed.

Korean Patent No. 10-2271521, “Method for generating surrounding images and server performing the same” Korean Patent Publication No. 10-2022-0136797, “Priority Object-Based Image Alignment Device and Method Therefor” Korean Patent No. 10-2179153, “Automatic Re-Stitching of 3D Surfaces” Korean Patent Publication No. 10-2021-0111622, "Server, method and computer program for providing realistic content"

The present invention aims to selectively regenerate a synthetic boundary for a plurality of video frames by determining whether or not to regenerate a synthetic boundary using a depth mask based on depth information according to parallax for a plurality of input images and a residual mask using two consecutive images from a plurality of input images.

The present invention aims at stitching videos more quickly by generating synthetic boundaries only for video frames selected by specific rules, rather than generating new synthetic boundaries for all video frames.

The present invention aims to reduce the probability of parallax distortion occurring in a stitched video when synthesizing a video through a single synthesis boundary line by regenerating a synthesis boundary line for a video frame in which parallax distortion may occur, thereby reducing the probability of parallax distortion occurring.

The present invention aims to control the speed and accuracy of image stitching by setting different search levels in a depth mask based on depth information.

According to one embodiment of the present invention, a video stitching device may include an image registration unit which receives a plurality of input images for a video frame, processes a common region image for the plurality of input images, and registers a stitching target image; a boundary generation unit which configures a boundary generation matrix through characteristic information for the registered stitching target images, and generates a synthetic boundary line based on the configured boundary generation matrix; a boundary regeneration unit which calculates a depth information matrix according to parallax for the plurality of input images, extracts boundary line position information according to the boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and an image synthesis unit which synthesizes the plurality of input images based on one of the synthetic boundary lines among the generated synthetic boundary line and the synthetic boundary line regenerated according to the determination of regeneration.

The above boundary line regeneration unit can generate the depth mask by quantizing the depth information matrix into N bits and indicating the position information of the synthetic boundary line generated along the path with the minimum cumulative sum of the boundary line generation matrix as a delimiter for the quantized depth information matrix.

The generated depth mask is distinguished according to the distribution of depth information values adjacent to the generated synthetic boundary line, and is initialized when the generated synthetic boundary line is regenerated, and can be maintained when the generated synthetic boundary line is maintained.

The above boundary line regeneration unit can generate a residual image through the size of the difference value between the N-1th image and the Nth image, which are two consecutive images from the plurality of input images, and remove noise from the generated residual image through erosion and dilation, and generate the residual mask by binarizing the residual image from which the noise has been removed based on a threshold value (θ _th ).

The above boundary line regeneration unit sets a window of a certain size (T×T) centered on the X-coordinate and Y-coordinate of one node of the synthetic boundary line indicated by a delimiter in the depth mask, applies the set window to the residual mask, counts the number of “1”s in the residual mask within the window, and compares a value obtained by dividing the counted number by the square of the size of the window with an error threshold value (θ _err ) to determine whether or not to regenerate the generated synthetic boundary line.

The above error threshold (θ _err ) can be calculated using the error sensitivity (α), the bit size (N) of the quantized depth mask, the left depth mask value (D _L ) of one node of the synthetic boundary line, and the right depth mask value (D _R ).

The above image registration unit can process the common area image by extracting pixels that are the same as feature points from the plurality of input images, even when the images are rotated, changed in brightness, or translated, and matching feature points that represent the same point from the extracted feature points, setting a common area including the matched feature points, estimating a homography through a positional relationship between the matched feature points in the common area, and applying warping to minimize parallax between the input images using the estimated homography.

The above boundary line generation unit calculates an energy function through the pixel change amount sizes for the horizontal and vertical axes for the processed common area image, extracts the calculated energy function as energy information among the characteristic information, extracts depth information among the characteristic information from the processed common area image through the positional difference between the same points in the plurality of input images, and configures the boundary line generation matrix based on the extracted energy information and the extracted depth information.

The above image synthesis unit can check the stitching result for the plurality of input images in the video frame, and if the plurality of input images are not all stitched in the checked stitching result, it can request whether to regenerate the synthesis boundary line, and if the plurality of input images are all stitched, it can request registration of a stitching target image for the next video frame of the video frame.

According to an embodiment of the present invention, a video stitching method may include: a step of registering, in an image registration unit, a stitching target image by receiving a plurality of input images for a video frame and processing a common region image for the plurality of input images; a step of configuring, in a boundary line generation unit, a boundary line generation matrix through characteristic information for the registered stitching target images; and a step of generating a synthetic boundary line based on the configured boundary line generation matrix; a step of calculating, in a boundary line regeneration unit, a depth information matrix according to parallax for the plurality of input images, extracting boundary line position information according to the boundary line generation matrix, generating a depth mask based on the extracted boundary line position information and the depth information matrix, generating a residual mask using two consecutive images from the plurality of input images, and determining whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and a step of synthesizing, in an image synthesis unit, the plurality of input images based on one of the generated synthetic boundary lines and the synthetic boundary lines regenerated according to the determination of regeneration.

The step of determining whether to regenerate the generated synthetic boundary line may include the steps of quantizing the depth information matrix into N bits, displaying position information of the synthetic boundary line generated along a path having a minimum cumulative sum of the boundary line generation matrix as a delimiter for the quantized depth information matrix to generate the depth mask, and the steps of generating a residual image through the size of a difference value between an N-1th image and an Nth image, which are two consecutive images among the plurality of input images, and removing noise from the generated residual image through erosion and dilation, and generating the residual mask by binarizing the residual image from which the noise has been removed based on a threshold value (θ _th ).

The step of determining whether to regenerate the generated synthetic boundary line may include the step of setting a window of a certain size (T×T) centered on the X-coordinate and Y-coordinate of one node of the synthetic boundary line indicated by a delimiter in the depth mask, applying the set window to the residual mask to count the number of “1”s in the residual mask within the window, and comparing a value obtained by dividing the counted number by the square of the size of the window with an error threshold value (θ _err ) to determine whether to regenerate the generated synthetic boundary line.

The present invention determines whether to regenerate a synthetic boundary line by using a depth mask based on depth information according to parallax for a plurality of input images and a residual mask using two consecutive images from a plurality of input images, thereby selectively regenerating a synthetic boundary line for a plurality of video frames.

The present invention can stitch videos more quickly by generating synthetic boundaries only for video frames selected by specific rules, rather than generating new synthetic boundaries for all video frames.

The present invention can reduce the probability of parallax distortion occurring in a stitched video when synthesizing a video through a single synthesis boundary line by regenerating a synthesis boundary line for a video frame in which parallax distortion may occur, thereby reducing the probability of parallax distortion occurring.

The present invention can control the speed and accuracy of image stitching by setting different search levels in a depth mask based on depth information.

FIG. 1 and FIG. 2 are drawings explaining an image stitching method based on a boundary line generation matrix in relation to one embodiment of the present invention.
FIG. 3 is a drawing explaining a video stitching method based on a boundary line generation matrix in relation to one embodiment of the present invention.
FIG. 4 is a drawing for explaining a video stitching device utilizing depth-based synthetic boundary line regeneration according to one embodiment of the present invention.
FIG. 5 is a drawing for explaining an image registration unit of a video stitching device according to an embodiment of the present invention.
FIG. 6 is a drawing for explaining a boundary regeneration unit of a video stitching device according to an embodiment of the present invention.
FIG. 7 is a drawing for explaining depth information calculated in a boundary line regeneration unit according to one embodiment of the present invention.
FIG. 8 is a drawing for explaining a procedure for generating a depth mask in a boundary line regeneration unit according to an embodiment of the present invention.
FIG. 9 is a drawing for explaining the types of depth masks generated in a boundary line regeneration unit according to one embodiment of the present invention.
FIG. 10 is a drawing for explaining a procedure for generating a residual mask in a boundary line regeneration unit according to one embodiment of the present invention.
FIG. 11 is a drawing for explaining an example image generated in a residual mask generation process in a boundary line regeneration unit according to one embodiment of the present invention.
FIG. 12 is a drawing for explaining a boundary inspection procedure of a boundary regeneration unit according to one embodiment of the present invention.
FIGS. 13 and 14 are diagrams for explaining a video stitching method utilizing depth-based synthetic boundary line regeneration according to one embodiment of the present invention.

Below, various embodiments of this document are described with reference to the attached drawings.

It should be understood that the examples and terms used herein are not intended to limit the technology described in this document to a particular embodiment, but rather to encompass various modifications, equivalents, and/or alternatives of the embodiments.

In the following description of various embodiments, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the invention, the detailed description will be omitted.

And the terms described below are terms defined in consideration of functions in various embodiments, and may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

In connection with the description of the drawings, similar reference numerals may be used for similar components.

A singular expression may include a plural expression unless the context clearly indicates otherwise.

In this document, expressions such as "A or B" or "at least one of A and/or B" may include all possible combinations of the items listed together.

Expressions such as "first," "second," "firstly," or "secondly," may be used to describe the components without regard to order or importance, and are only used to distinguish one component from another and do not limit the components.

When it is said that a certain (e.g., a first) component is "(functionally or communicatively) connected" or "connected" to another (e.g., a second) component, said certain component may be directly connected to said other component, or may be connected through another component (e.g., a third component).

In this specification, the phrase “configured to” may be used interchangeably with, for example, “suitable for,” “having the ability to,” “modified to,” “made to,” “capable of,” or “designed to,” in terms of hardware or software, depending on the context.

In some contexts, the expression "a device configured to" may mean that the device is "capable of" doing something in conjunction with other devices or components.

For example, the phrase "a processor configured (or set) to perform A, B, and C" can mean a dedicated processor (e.g., an embedded processor) to perform those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

Also, the term 'or' implies an inclusive or rather than an exclusive or.

That is, unless otherwise stated or clear from the context, the expression 'x utilizes a or b' means any one of the natural inclusive permutations.

The terms '..bu', '..gi', etc. used below mean a unit that processes at least one function or operation, and this can be implemented by hardware, software, or a combination of hardware and software.

FIG. 1 and FIG. 2 are drawings explaining an image stitching method based on a boundary line generation matrix in relation to one embodiment of the present invention.

Figure 1 illustrates the process of an image stitching technique based on a boundary line generation matrix utilized by the present invention.

The present invention is an application technology of image stitching based on a synthetic boundary line, and more specifically, a technology for overcoming the inefficiency of performing the entire image stitching process for all frames when synthesizing a video through the existing image stitching based on a synthetic boundary line.

Referring to Fig. 1, the existing boundary-based image stitching technology extracts feature points in steps (S101) and (S102), matches feature points in step (S103), and sets a common area in step (S104).

In addition, homography is estimated at step (S105), warping is performed at step (S106), a boundary line is generated at step (S107), and an image synthesis process is performed at step (S108).

When compositing a video using this technique, all of the processes described above must be performed for each and every frame of the video.

At this time, the feature point extraction and matching process and the boundary line generation process are processes with high execution complexity, and there is a limitation that the time required for the process increases rapidly when repeatedly executed.

Figure 2 illustrates a procedure for generating a boundary line in an existing boundary line generation matrix-based image stitching method.

Referring to FIG. 2, the image stitching method calculates image characteristic information (200), such as pixel difference values or visual perception energy functions, between multiple input images, such as a left input image and a right input image, in step (S201).

In step (S202), the image stitching method constructs a boundary generation matrix using the corresponding image characteristic information (200). Image (201) exemplifies the boundary generation matrix result.

In step (S203), the image stitching method sets and generates a path with a minimum cumulative sum in a boundary line generation matrix as a synthetic boundary line. Image (202) illustrates the result of generating a synthetic boundary line.

In step (S204), the image stitching method synthesizes input images along a synthesis boundary.

If this process is not performed anew for every frame of the video, a synthetic boundary line may be generated over a moving object in the video, which may cause parallax distortion in the synthesized image. Here, parallax distortion may refer to a phenomenon in which an object appears duplicated or part of an object disappears in the stitched image as the synthetic boundary line is generated in the object area.

FIG. 3 is a drawing explaining a video stitching method based on a boundary line generation matrix in relation to one embodiment of the present invention.

FIG. 3 illustrates a conventional boundary line generation matrix-based video stitching method in relation to one embodiment of the present invention.

Referring to Fig. 3, the video stitching method proposes a video stitching technique that selects a specific frame from a video and applies video stitching only to the corresponding video frame in order to reduce repetitive execution.

Through the object detection procedure (S300), an object is detected in step (S301), the object is tracked in step (S302), whether the object has moved is determined in step (S303), and if the object is determined to have moved in step (S304), a boundary line is regenerated.

The object detection procedure (S300) searches for moving objects within a video, and determines whether to regenerate a synthetic boundary line by comparing the moved position of the object with the position of the synthetic boundary line.

However, the execution complexity of the object detection procedure added to support the synthetic boundary line regeneration judgment function is not low, and the accuracy of selecting a frame requiring boundary line regeneration may vary significantly depending on the performance of object detection.

Although deep learning-based object detection can be used to support fast and accurate object detection, if there are objects in the video that were not defined in the deep learning training stage, they may not be detected, which may lead to incorrect results.

FIG. 4 is a drawing for explaining a video stitching device utilizing depth-based synthetic boundary line regeneration according to one embodiment of the present invention.

FIG. 4 illustrates components of a video stitching device utilizing depth-based synthetic boundary regeneration according to one embodiment of the present invention.

Referring to FIG. 4, a video stitching device (400) according to one embodiment of the present invention includes an image registration unit (410), a boundary line generation unit (420), a boundary line regeneration unit (430), and an image synthesis unit (440).

The video stitching device (400) performs efficient video stitching by regenerating a synthetic boundary line only for video frames selected according to specific rules, rather than performing video stitching for all video frames.

Additionally, the video stitching device (400) controls the speed and accuracy of video stitching by setting the search level differently according to depth information.

According to one embodiment of the present invention, an image registration unit (410) receives a plurality of input images for a video frame and registers a stitching target image by processing a common area image for the plurality of input images.

For example, the image registration unit (410) extracts the same pixels as feature points from multiple input images even when the images are rotated, changed in brightness, or translated, and matches feature points representing the same point among the extracted feature points.

In addition, the image registration unit (410) sets a common area including matched feature points, estimates homography through a positional relationship between matched feature points in the common area, and processes the common area image by applying warping so that the parallax between input images is minimized using the estimated homography.

According to one embodiment of the present invention, a boundary line generation unit (420) can configure a boundary line generation matrix using characteristic information on a stitching target image, and generate a synthetic boundary line based on the configured boundary line generation matrix.

For example, the boundary line generation unit (420) calculates an energy function through the pixel change amount sizes for the horizontal and vertical axes for the processed common area image, and extracts the calculated energy function as energy information among the characteristic information.

According to one embodiment of the present invention, the boundary line generation unit (420) can extract depth information from among characteristic information in a common area image processed through the positional difference between identical points in a plurality of input images, and configure a boundary line generation matrix based on the extracted energy information and the extracted depth information.

For example, the boundary line regeneration unit (430) calculates a depth information matrix according to parallax for multiple input images, extracts boundary line position information according to the boundary line generation matrix, and generates a depth mask based on the extracted boundary line position information and the depth information matrix.

According to one embodiment of the present invention, a boundary line regeneration unit (430) generates a residual mask using two consecutive images from a plurality of input images, and can determine whether to regenerate a generated synthetic boundary line using the generated depth mask and the generated residual mask.

For example, the boundary line regeneration unit (430) quantizes a depth information matrix into N bits and generates a depth mask by indicating the position information of a synthetic boundary line generated along a path with a minimum cumulative sum of the boundary line generation matrix as a delimiter for the quantized depth information matrix.

For example, a depth mask may be distinguished based on the distribution of depth information values adjacent to a synthetic boundary, initialized when the synthetic boundary is regenerated, or maintained when the synthetic boundary is maintained.

According to one embodiment of the present invention, a boundary line regeneration unit (430) generates a residual image through the size of the difference value between an N-1th image and an Nth image, which are two consecutive images from a plurality of input images, and removes noise from the residual image through erosion and dilation.

In addition, the boundary regeneration unit (430) can generate a residual mask by binarizing the residual image from which noise has been removed based on a threshold value (θ _th ). For example, the residual image can be referred to as a motion frame (Mot _N ) image.

For example, the boundary line regeneration unit (430) sets a window of a certain size (T×T) centered on the X-coordinate and Y-coordinate of one node of the synthetic boundary line indicated as a delimiter in the depth mask, applies the set window to the residual mask, and counts the number of “1”s in the residual mask within the window.

For example, the boundary line regeneration unit (430) can determine whether to regenerate the generated synthetic boundary line by comparing the value obtained by dividing the counted number by the square of the size of the window with an error threshold value (θ _err ).

The boundary line regeneration unit (430) determines to regenerate a synthetic boundary line if the value of the counted number divided by the square of the size of the window is greater than the error threshold, and requests the boundary line generation unit (420) to generate a synthetic boundary line.

According to one embodiment of the present invention, the image synthesis unit (440) can synthesize a plurality of input images based on one of the synthesis boundaries among the generated synthesis boundaries and the regenerated synthesis boundaries according to the regeneration decision.

For example, the image synthesis unit (440) checks the stitching results for multiple input images in a video frame, and if none of the multiple input images are stitched in the confirmed stitching results, it requests a determination on whether to regenerate the synthesis boundary line.

In addition, when multiple input images are all stitched, the image synthesis unit (440) requests registration of a stitching target image for the next video frame of the video frame.

Accordingly, the present invention can selectively regenerate a synthetic boundary for a plurality of video frames by determining whether to regenerate a synthetic boundary using a depth mask based on depth information according to parallax for a plurality of input images and a residual mask using two consecutive images from the plurality of input images.

FIG. 5 is a drawing for explaining an image registration unit of a video stitching device according to an embodiment of the present invention.

FIG. 5 illustrates components of an image registration unit of a video stitching device according to an embodiment of the present invention.

Referring to FIG. 5, according to one embodiment of the present invention, an image registration unit (500) includes a feature point extraction and matching unit (510), a common area estimation unit (520), a homography estimation unit (530), and a warping unit (540).

The feature point extraction and matching unit (510) extracts the same pixels as feature points from multiple input images even when the images are rotated, changed in brightness, or translated, and matches feature points representing the same point among the extracted feature points.

The common area estimation unit (520) sets a common area including matched feature points.

The homography estimation unit (530) estimates homography through the positional relationship between matched feature points in a common area.

The warping unit (540) applies warping using homography to minimize parallax between input images.

The image registration unit (500) of the present invention is a procedure for obtaining common areas and homography information between multiple input images, and its utilization is not limited to the procedure mentioned above.

FIG. 6 is a drawing for explaining a boundary regeneration unit of a video stitching device according to an embodiment of the present invention.

FIG. 6 illustrates components of a boundary regeneration unit of a video stitching device according to an embodiment of the present invention.

Referring to FIG. 6, a boundary line regeneration unit (600) of a video stitching device according to an embodiment of the present invention includes a depth information calculation unit (610), a depth mask generation unit (620), a residual mask generation unit (630), a boundary line inspection unit (640), and a boundary line regeneration unit (650).

For example, the depth information calculation unit (610) calculates depth information according to parallax for multiple input images.

Depth information is used to extract boundary location information according to the boundary generation matrix.

The depth mask generation unit (620) generates a depth mask based on boundary position information and a depth information matrix.

The depth mask generation unit (620) quantizes the depth information matrix into N bits and generates a depth mask by indicating the position information of a synthetic boundary line generated along a path with a minimum cumulative sum of the boundary line generation matrix as a delimiter for the quantized depth information matrix.

The depth mask is initialized to a new depth mask only when a new composite boundary needs to be generated, and its value is kept otherwise.

The residual mask generation unit (630) generates a residual image through the size of the difference value between the N-1th image and the Nth image, which are two consecutive images from a plurality of input images, removes noise from the generated residual image through erosion and dilation, and generates a residual mask by binarizing the residual image from which noise has been removed based on a threshold value (θ _th ).

The boundary inspection unit (640) compares the position information and depth information of the boundary in the depth mask and the number of “1”s within the window size applied to the residual mask with an error threshold value to check whether the boundary is regenerated.

The boundary line regeneration unit (650) transmits a boundary line regeneration request to the boundary line generation unit based on the boundary line inspection result so that the boundary line is regenerated.

Accordingly, the present invention can reduce the probability of parallax distortion occurring in a stitched video when synthesizing a video through a single synthesis boundary line by regenerating a synthesis boundary line for a video frame in which parallax distortion may occur, thereby reducing the probability of parallax distortion occurring.

FIG. 7 is a drawing for explaining depth information calculated in a boundary line regeneration unit according to one embodiment of the present invention.

Figure 7 illustrates depth information calculated in a boundary line regeneration unit according to one embodiment of the present invention.

Referring to FIG. 7, image (700) represents an input image and image (710) represents a depth information image.

The depth information image for the image (700) may be the image (710).

The depth information matrix undergoes N-bit quantization, and can be set to a smaller or larger value as the distance gets closer.

The depth information calculation process is not limited by the depth information generation method.

It can be calculated by utilizing the parallax from at least two input images, or extracted from a single image using a deep learning-based depth information predictor.

FIG. 8 is a drawing for explaining a procedure for generating a depth mask in a boundary line regeneration unit according to an embodiment of the present invention.

FIG. 8 illustrates a procedure for generating a depth mask in a boundary regeneration unit according to an embodiment of the present invention.

Referring to FIG. 8, according to one embodiment of the present invention, a boundary line regeneration unit generates a depth mask by indicating position information of a synthetic boundary line generated along a path with a minimum cumulative sum of a boundary line generation matrix as a delimiter for a quantized depth information matrix.

According to one embodiment of the present invention, a boundary line regeneration unit extracts boundary line location information from a boundary line generation matrix (800) and generates boundary line location information (810).

According to one embodiment of the present invention, a boundary line regeneration unit generates a depth mask (830) by combining a previously generated depth information matrix (820) and boundary line position information (810).

FIG. 9 is a drawing for explaining the types of depth masks generated in a boundary line regeneration unit according to one embodiment of the present invention.

FIG. 9 illustrates the types of depth masks generated in a boundary line regeneration unit according to one embodiment of the present invention.

Referring to FIG. 9, the first depth mask (900) may represent a depth mask composed of close distance information.

The second depth mask (910) may represent a depth mask composed of long-distance information.

The third depth mask (920) may represent a depth mask composed of mixed distance information.

The first depth mask (900) consists of a quantized bit size of “12”.

The second depth mask (910) consists of a quantized bit size of “230”.

The third depth mask (920) consists of quantized bit sizes “12” and “230”.

The quantized bit size can be set smaller as the distance information gets closer.

The depth information matrix undergoes N-bit quantization, and can be set to a smaller or larger value as the distance gets closer.

For example, when quantizing depth information, which ranges from "0" to "1", into "8" bits, the value "0" is assigned to the area representing the closest distance, and "255" is assigned to the area representing the farthest distance.

Conversely, when depth information, which ranges from "0" to "1", is quantized into 8 bits, the area representing the closest distance may be assigned a value of "255", and the area representing the farthest distance may be assigned a value of 0.

FIG. 10 is a drawing for explaining a procedure for generating a residual mask in a boundary line regeneration unit according to one embodiment of the present invention.

FIG. 10 illustrates a procedure for generating a residual mask in a boundary line regeneration section in a video stitching method according to an embodiment of the present invention.

Referring to FIG. 10, in step (S1001), the video stitching method generates residual images of the (N-1)th and Nth images from the input image.

The video stitching method requires the (N-1)th frame and the Nth frame from the input video, and generates a motion frame image through the size of the difference value between the (N-1)th frame and the Nth frame. Here, the motion frame image may be a residual image, and the motion frame image may be determined based on mathematical expression 1.

[Mathematical formula 1]

In mathematical expression 1, Mot _N can represent a motion frame image, ?? _N can represent the Nth frame, and ?? _N-1 can represent the (N-1)th frame.

In step (S1002), the video stitching method applies erosion and dilation processing to the residual image.

The video stitching method calculates the movement of objects to determine whether parallax distortion may occur in motion frame images, but noise is included due to the lighting environment of the shooting camera or camera shake, and this is removed.

In step (S1003), the video stitching method applies binarization to the residual image to which erosion and dilation have been applied.

Mathematical expression 2 for forming a residual mask (M _R ) by binarizing a motion frame image (Mot _N ) based on a threshold value (θ _th ) can be organized as follows.

[Mathematical formula 2]

In mathematical expression 2, M _R can represent a residual mask, θ _th can represent a threshold, and Mot _N can represent a motion frame image.

The residual mask consists of “0” and “1” based on mathematical expressions 1 and 2.

FIG. 11 is a drawing for explaining an example image generated in a residual mask generation process in a boundary line regeneration unit according to one embodiment of the present invention.

Referring to FIG. 11, image (1100) may represent the N-1th frame, and image (1101) may represent the Nth frame.

Image (1110) may represent a residual frame image, image (1111) may represent an image to which erosion and dilation have been applied, and image (1112) may represent an image to which binarization has been applied.

FIG. 12 is a drawing for explaining a boundary inspection procedure of a boundary regeneration unit according to one embodiment of the present invention.

FIG. 12 illustrates a boundary inspection procedure of a boundary regeneration unit according to one embodiment of the present invention.

Referring to FIG. 12, the boundary inspection procedure of the boundary regeneration unit according to one embodiment of the present invention is performed using a depth mask (1200) and a binarized residual mask (1210).

The accumulated value (Z) of the residual mask (1210) within a window (1211) of size T x T is calculated, focusing on the X-coordinate and Y-coordinate of one node (1201) among the synthetic boundaries indicated by delimiters in the depth mask (1200). Mathematical expression 3 can be organized as follows.

[Mathematical formula 3]

In mathematical expression 3, Z can represent a cumulative value, and M _R can represent a residual mask.

Here, the size of the window (1211) region is experimentally adjustable, and the boundary checking procedure, as shown in Equation 4, determines whether the synthetic boundary should be regenerated in the corresponding video frame by determining to regenerate the synthetic boundary when the number of "1"s in the residual mask (1210) within the window region divided by the square of the window size (T) is greater than the error threshold (θ _err ). Equation 4 can be summarized as follows.

[Mathematical Formula 4]

In mathematical expression 4, Z can represent an accumulated value, T can represent a window size, and θ _err can represent an error threshold.

The boundary regeneration process regenerates the boundary if the decision value in the corresponding video frame in the previous boundary regeneration process is false, and maintains the previously generated synthetic boundary if it is true.

The error threshold (θ _err ) is calculated using the error sensitivity (α), the bit size (N) of the quantized depth mask, the left depth mask value (D _L ) and the right depth mask value (D _R ) of one node of the synthetic boundary. This can be organized as shown in Equation 5.

[Mathematical Formula 5]

When proceeding sequentially, the judgment of whether to generate the boundary line again is based on the residual mask, and the accumulated value is the number of "1"s in the window size (T), and it can be determined by considering the bit size of N-bit quantization considering the error sensitivity at the right depth and the left depth at the same reference point.

The right depth (1203) and the left depth (1202) of the same reference point (1201) are 6, and the accumulated value within the window (1211) is 6.

FIGS. 13 and 14 are diagrams for explaining a video stitching method utilizing depth-based synthetic boundary line regeneration according to one embodiment of the present invention.

FIG. 13 illustrates a procedure for video stitching by selectively regenerating a synthetic boundary for a plurality of video frames by determining whether to regenerate a synthetic boundary according to a video stitching method utilizing the regeneration of a depth-based synthetic boundary according to one embodiment of the present invention.

Referring to FIG. 13, in step (S1301), a video stitching method according to an embodiment of the present invention registers a stitching target video.

That is, a video stitching method according to one embodiment of the present invention receives a plurality of input images for a video frame and registers a stitching target image by processing a common area image for the plurality of input images.

In step (S1302), a video stitching method according to an embodiment of the present invention generates a synthetic boundary line based on a boundary line generation matrix.

That is, a video stitching method according to an embodiment of the present invention constructs a boundary line generation matrix through characteristic information on a stitching target image registered in step (S1301), and generates a synthetic boundary line based on the constructed boundary line generation matrix.

In step (S1303), a video stitching method according to an embodiment of the present invention generates a depth mask and a residual mask, and determines whether to regenerate a boundary line using the depth mask and the residual mask.

That is, a video stitching method according to an embodiment of the present invention calculates a depth information matrix according to parallax for a plurality of input images, extracts boundary position information according to a boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask.

In step (S1304), a video stitching method according to an embodiment of the present invention synthesizes multiple input images based on a synthetic boundary line.

That is, a video stitching method according to one embodiment of the present invention can synthesize a plurality of input images based on one of a previously generated synthetic boundary line and a synthetic boundary line regenerated according to a regeneration decision.

Therefore, the present invention can stitch videos more quickly by generating synthetic boundaries only for video frames selected by specific rules, rather than generating new synthetic boundaries for all video frames.

FIG. 14 illustrates a procedure for performing video stitching by selectively performing a depth mask generation procedure according to a video stitching method utilizing regeneration of a depth-based synthetic boundary line according to one embodiment of the present invention.

Referring to FIG. 14, in step (S1401), a video stitching method according to an embodiment of the present invention determines whether a synthetic boundary line is newly generated in a previous video frame.

A video stitching method according to one embodiment of the present invention proceeds to step (S1402) if a new synthetic boundary line is created, and proceeds to step (S1403) if a new synthetic boundary line is not created.

In step (S1402), a video stitching method according to an embodiment of the present invention calculates and extracts depth information from a common area image.

In step (S1403), a video stitching method according to an embodiment of the present invention generates a depth mask.

That is, the video stitching method according to one embodiment of the present invention performs depth information calculation and depth mask generation only when a synthetic boundary line is newly generated from a previous video frame.

In step (S1404), a video stitching method according to an embodiment of the present invention generates a residual mask.

In step (S1405), a video stitching method according to an embodiment of the present invention determines whether a boundary line is regenerated based on boundary line inspection.

A video stitching method according to one embodiment of the present invention proceeds to step (S1406) to regenerate the boundary line if it is determined that the boundary line is to be regenerated, and if not, input images are synthesized at step (S1407) based on the previously generated boundary line.

That is, a video stitching method according to one embodiment of the present invention determines whether to regenerate a synthetic boundary line or maintain an existing synthetic boundary line.

In step (S1408), a video stitching method according to an embodiment of the present invention determines whether all video frames have been stitched.

A video stitching method according to one embodiment of the present invention ends the procedure when input images are stitched in all video frames included in a video, and conversely, when there are stitching targets remaining, the procedure returns to step (S1401).

That is, the video stitching method according to one embodiment of the present invention is repeated until stitching is completed for all video frames.

Therefore, the present invention can control the speed and accuracy of image stitching by setting different search levels in a depth mask based on depth information.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone, but those skilled in the art will appreciate that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, the processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, flash memories, etc. Examples of the program commands include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal waves, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

Although the embodiments have been described with limited drawings as described above, those skilled in the art will appreciate that various modifications and variations may be made from the above teachings. For example, appropriate results may be achieved even if the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims are also included in the scope of the claims described below.

400: Video Stitching Device
410: Image registration unit 420: Boundary generation unit
430: Boundary Regeneration Section 440: Image Synthesis Section

Claims (12)

An image registration unit that receives multiple input images for a video frame and registers a stitching target image by processing a common area image for the multiple input images;
A boundary generation unit which constructs a boundary generation matrix using characteristic information on the registered stitching target image, and generates a synthetic boundary line based on the constructed boundary generation matrix;
A boundary line regeneration unit that calculates a depth information matrix according to parallax for the plurality of input images, extracts boundary line position information according to the boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and
An image synthesis unit is included that synthesizes the plurality of input images based on one of the synthetic boundary lines generated above and the synthetic boundary lines re-generated according to the regeneration decision.
The above boundary line regeneration unit is characterized in that it generates a residual image through the size of the difference value between the N-1th image and the Nth image, which are two consecutive images, from the plurality of input images, and removes noise from the generated residual image through erosion and dilation, and generates the residual mask by binarizing the residual image from which the noise has been removed based on a threshold value (θ _th ).
Video stitching device. In the first paragraph,
The above boundary line regeneration unit is characterized in that it quantizes the depth information matrix into N bits and generates the depth mask by indicating the position information of the synthetic boundary line generated along the path with the minimum cumulative sum of the boundary line generation matrix as a delimiter for the quantized depth information matrix.
Video stitching device. An image registration unit that receives multiple input images for a video frame and registers a stitching target image by processing a common area image for the multiple input images;
A boundary generation unit which constructs a boundary generation matrix using characteristic information on the registered stitching target image, and generates a synthetic boundary line based on the constructed boundary generation matrix;
A boundary line regeneration unit that calculates a depth information matrix according to parallax for the plurality of input images, extracts boundary line position information according to the boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and
An image synthesis unit is included that synthesizes the plurality of input images based on one of the synthetic boundary lines generated above and the synthetic boundary lines re-generated according to the regeneration decision.
The above boundary line regeneration unit quantizes the depth information matrix into N bits, and generates the depth mask by indicating the position information of the synthetic boundary line generated along the path with the minimum cumulative sum of the boundary line generation matrix as a delimiter for the quantized depth information matrix.
The generated depth mask is characterized in that it is distinguished according to the distribution of depth information values adjacent to the generated synthetic boundary line, and is initialized when the generated synthetic boundary line is regenerated, and is maintained when the generated synthetic boundary line is maintained.
Video stitching device. delete An image registration unit that receives multiple input images for a video frame and registers a stitching target image by processing a common area image for the multiple input images;
A boundary line generation unit which constructs a boundary line generation matrix using characteristic information on the registered stitching target image, and generates a synthetic boundary line based on the constructed boundary line generation matrix;
A boundary line regeneration unit that calculates a depth information matrix according to parallax for the plurality of input images, extracts boundary line position information according to the boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and
An image synthesis unit is included that synthesizes the plurality of input images based on one of the synthetic boundary lines generated above and the synthetic boundary lines re-generated according to the regeneration decision.
The above boundary line regeneration unit sets a window of a certain size (T×T) centered on the X-coordinate and Y-coordinate of one node of the synthetic boundary line indicated by a delimiter in the depth mask, applies the set window to the residual mask, counts the number of "1"s in the residual mask within the window, and compares the value obtained by dividing the counted number by the square of the size of the window with an error threshold value (θ _err ) to determine whether to regenerate the generated synthetic boundary line.
Video stitching device. In paragraph 5,
The above error threshold value (θ _err ) is characterized in that it is calculated using the error sensitivity (α), the bit size (N) of the quantized depth mask, the left depth mask value (D _L ) of one node of the synthetic boundary line, and the right depth mask value (D _R ).
Video stitching device. An image registration unit that receives multiple input images for a video frame and registers a stitching target image by processing a common area image for the multiple input images;
A boundary generation unit which constructs a boundary generation matrix using characteristic information on the registered stitching target image, and generates a synthetic boundary line based on the constructed boundary generation matrix;
A boundary line regeneration unit that calculates a depth information matrix according to parallax for the plurality of input images, extracts boundary line position information according to the boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and
An image synthesis unit is included that synthesizes the plurality of input images based on one of the synthetic boundary lines generated above and the synthetic boundary lines re-generated according to the regeneration decision.
The image registration unit extracts the same pixels as feature points from the plurality of input images even when the images are rotated, changed in brightness, or translated, matches feature points representing the same point from the extracted feature points, sets a common area including the matched feature points, estimates homography through a positional relationship between the matched feature points in the common area, and processes the common area image by applying warping so that the parallax between the input images is minimized using the estimated homography.
Video stitching device. An image registration unit that receives multiple input images for a video frame and registers a stitching target image by processing a common area image for the multiple input images;
A boundary line generation unit which constructs a boundary line generation matrix using characteristic information on the registered stitching target image, and generates a synthetic boundary line based on the constructed boundary line generation matrix;
A boundary line regeneration unit that calculates a depth information matrix according to parallax for the plurality of input images, extracts boundary line position information according to the boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and
An image synthesis unit is included that synthesizes the plurality of input images based on one of the synthetic boundary lines generated above and the synthetic boundary lines re-generated according to the regeneration decision.
The above boundary line generation unit calculates an energy function through the pixel change amount sizes for the horizontal and vertical axes for the processed common area image, extracts the calculated energy function as energy information among the characteristic information, extracts depth information among the characteristic information from the processed common area image through the positional difference between the same points in the plurality of input images, and configures the boundary line generation matrix based on the extracted energy information and the extracted depth information.
Video stitching device. An image registration unit that receives multiple input images for a video frame and registers a stitching target image by processing a common area image for the multiple input images;
A boundary line generation unit which constructs a boundary line generation matrix using characteristic information on the registered stitching target image, and generates a synthetic boundary line based on the constructed boundary line generation matrix;
A boundary line regeneration unit that calculates a depth information matrix according to parallax for the plurality of input images, extracts boundary line position information according to the boundary line generation matrix, generates a depth mask based on the extracted boundary line position information and the depth information matrix, generates a residual mask using two consecutive images from the plurality of input images, and determines whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and
An image synthesis unit is included that synthesizes the plurality of input images based on one of the synthetic boundary lines generated above and the synthetic boundary lines re-generated according to the regeneration decision.
The above image synthesis unit checks the stitching result for the plurality of input images in the video frame, and if the plurality of input images are not all stitched in the checked stitching result, it requests a determination on whether to regenerate the synthesis boundary line, and if the plurality of input images are all stitched, it requests registration of the stitching target image for the next video frame of the video frame.
Video stitching device. In an image registration unit, a step of receiving a plurality of input images for a video frame and registering a stitching target image by processing a common area image for the plurality of input images;
In the boundary line generation unit, a step of configuring a boundary line generation matrix through characteristic information for the registered stitching target image, and generating a synthetic boundary line based on the configured boundary line generation matrix;
In a boundary line regeneration unit, a step of calculating a depth information matrix according to parallax for the plurality of input images, extracting boundary line position information according to the boundary line generation matrix, generating a depth mask based on the extracted boundary line position information and the depth information matrix, generating a residual mask using two consecutive images from the plurality of input images, and determining whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and
In the image synthesis unit, a step of synthesizing the plurality of input images based on one of the generated synthesis boundary lines and the regenerated synthesis boundary lines according to the regeneration decision is included,
The step of determining whether to regenerate the above-mentioned generated synthetic boundary line is:
It is characterized by including a step of generating a residual image through the size of the difference value between the N-1th image and the Nth image, which are two consecutive images from the plurality of input images, removing noise from the generated residual image through erosion and dilation, and generating the residual mask by binarizing the residual image from which the noise has been removed based on a threshold value (θ _th ).
How to stitch videos. In Article 10,
The step of determining whether to regenerate the above-mentioned generated synthetic boundary line is:
A method characterized by comprising the step of generating the depth mask by quantizing the depth information matrix into N bits and indicating the position information of the synthetic boundary line generated along the path with the minimum cumulative sum of the boundary line generation matrix as a delimiter for the quantized depth information matrix.
How to stitch videos. In an image registration unit, a step of receiving a plurality of input images for a video frame and registering a stitching target image by processing a common area image for the plurality of input images;
In the boundary line generation unit, a step of configuring a boundary line generation matrix through characteristic information on the registered stitching target image, and generating a synthetic boundary line based on the configured boundary line generation matrix;
In a boundary line regeneration unit, a step of calculating a depth information matrix according to parallax for the plurality of input images, extracting boundary line position information according to the boundary line generation matrix, generating a depth mask based on the extracted boundary line position information and the depth information matrix, generating a residual mask using two consecutive images from the plurality of input images, and determining whether to regenerate the generated synthetic boundary line using the generated depth mask and the generated residual mask; and
In the image synthesis unit, a step of synthesizing the plurality of input images based on one of the generated synthesis boundary lines and the regenerated synthesis boundary lines according to the regeneration decision is included,
The step of determining whether to regenerate the above-mentioned generated synthetic boundary line is:
A method for generating a synthetic boundary line, comprising: setting a window of a certain size (T×T) centered on the X-coordinate and Y-coordinate of one node of a synthetic boundary line indicated by a delimiter in the depth mask; applying the set window to the residual mask; counting the number of "1"s in the residual mask within the window; and comparing the value obtained by dividing the counted number by the square of the size of the window with an error threshold value (θ _err ) to determine whether to regenerate the generated synthetic boundary line.
How to stitch videos.