CN110515463B - A 3D model embedding method based on monocular vision in gesture interactive video scene - Google Patents

Abstract

The invention discloses a 3D model embedding method based on monocular vision in a gesture interactive video scene, which belongs to the field of computer graphics and comprises the following steps: the method comprises the steps of monocular scene depth reconstruction, fine gesture extraction, 3D model rendering, shielding judgment and gesture redrawing. The invention provides a 3D model embedding method based on monocular vision in a gesture interactive video scene, which combines the technologies of monocular depth recovery, gesture extraction, 3D model rendering, occlusion judgment, gesture redrawing and the like, can effectively process the occlusion relation between a gesture and a model, and obtains a seamless embedding effect. The invention can effectively solve the problem of shielding generated during gesture interaction, does not depend on expensive depth detection equipment, and meets the requirement of practical application.

Description

A 3D model embedding method based on monocular vision in gesture interactive video scene

technical field

The invention belongs to the field of computer graphics and relates to a method for embedding a 3D model based on monocular vision in a gesture interactive video scene.

Background technique

Augmented reality (AugmentedReality, AR) has the advantages of blending virtual and real, strong expressiveness and good interactivity. By embedding virtual information on images of real scenes, augmented reality technology presents users with an environment with realistic sensory effects. As a natural way of human-computer interaction, gesture interaction is widely used in augmented reality and virtual reality.

Occlusion consistency is the basic principle that the immersive augmented reality environment needs to satisfy, and it requires the correct occlusion relationship between the elements in the augmented reality system. A video image is the projection of a 3D scene on a 2D plane, and restoring the spatial occlusion relationship between the video scene and the embedded model is a prerequisite for achieving occlusion consistency. However, most of the current augmented reality applications ignore the occlusion relationship, and the video is only drawn as the background of the virtual model. However, in gesture-interactive augmented reality applications, gesture interaction is highly dynamic, and the occlusion relationship between gestures and embedded models is not fixed. Drawing the video first and then drawing the embedded model can easily lead to visual confusion. Although the depth information of the video scene can be obtained by using a depth sensor, or the method of stereo vision by using dual cameras, and the occlusion relationship of the scene can be reconstructed, but there are problems in terms of equipment cost or portability.

Contents of the invention

The present invention proposes a 3D model embedding method based on monocular vision in a gesture interactive video scene, which combines technologies such as monocular depth recovery, precise gesture extraction and 3D model embedding, and can effectively handle the occlusion relationship between gestures and models, and obtain Seamless embedding effect.

The technical solution of the present invention is: a 3D model embedding method based on monocular vision in a gesture interactive video scene, comprising the following steps:

Step 1: Monocular scene depth reconstruction:

Use the depth sensor or binocular vision method to obtain the scene depth map, and then register the obtained scene depth map and convert it to the camera coordinate system as the training set of the monocular depth recovery model; use the above training set to recover the monocular depth The network performs migration learning to obtain a monocular depth recovery model suitable for the scene; use the monocular depth recovery model to directly obtain the scene depth map of the current frame.

Step 2: Fine Gesture Extraction:

2.1) Gesture detection and positioning: perform gesture detection and positioning on the current frame to obtain a gesture detection frame.

2.2) Rough gesture mask generation: use the mixed Gaussian model to extract the foreground, then initialize an RGBA image with the same resolution as the current frame as the gesture mask, and set the RGB value of each pixel in the gesture mask to the current frame The RGB value of the element, set the Alpha value of the pixel located in the foreground area and the gesture detection border to 1, and set the Alpha value of other pixels to 0 to obtain a rough gesture mask.

2.3), fine gesture mask generation: use the seed algorithm to detect the pixel connected domains with an Alpha value of 0 within the range of the gesture detection frame in the rough gesture mask, calculate the number of pixels in each connected domain, if the number of pixels is less than the threshold, then The connected domain is noise, and the alpha values of all pixels in the connected domain are set to 1. This method removes small area noise and obtains a fine gesture mask.

Step 3: 3D model rendering:

3.1), adopt the camera tracking method based on the marker template, perform feature extraction and matching on the current video frame and the marker image, and obtain the camera attitude information corresponding to the current frame, that is, the rotation matrix from the current camera coordinate system to the three-dimensional world coordinate system and translation matrix;

3.2), first draw the current frame as the window background, then transform the 3D model into the screen space according to the camera internal reference, and the rotation matrix and translation matrix obtained in 3.1), and draw it on the window background, and obtain and save the depth map as Model depth map.

Step 4: Occlusion judgment and gesture redrawing:

4.1) Occlusion judgment: Sampling the model depth map and scene depth map with pixel coordinates with an Alpha value of 1 in the gesture mask to obtain the model depth and gesture depth respectively. If the gesture depth < the scene depth, it is considered that the gesture at this position is harmful to the virtual 3D If the model is occluded, keep the Alpha value of the corresponding pixel in the gesture mask as 1, otherwise set the Alpha value to 0;

4.2) Gesture redrawing: For pixels with an Alpha value of 1, use their RGB values to cover the window background value, and for pixels with an Alpha value of 0, the window background value is still retained.

The beneficial effects of the present invention are: a 3D model embedding method based on monocular vision in a gesture interactive video scene proposed by the present invention can not only effectively deal with the occlusion problem generated during gesture interaction, but also does not rely on expensive depth detection equipment, meet the needs of practical applications.

Description of drawings

Fig. 1 is the specific technical roadmap described in the present invention;

FIG. 2 is a schematic diagram of depth reconstruction of a monocular scene in the present invention;

Fig. 3 is a schematic diagram of fine gesture extraction in the present invention;

Fig. 4 is a flow chart of occlusion judgment and gesture redrawing in the present invention;

Fig. 5 is a schematic diagram of the embedding effect of the 3D model of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

The present invention: a 3D model embedding method based on monocular vision in a gesture interactive video scene, comprising the following steps:

Step 1: Monocular scene depth reconstruction:

Use the depth sensor or binocular vision method to obtain the scene depth map, and then register the obtained scene depth map and convert it to the camera coordinate system as the training set of the monocular depth recovery model; use the above training set to recover the monocular depth The network performs migration learning to obtain a monocular depth restoration model suitable for the scene; the scene depth map of the current frame is directly obtained by using the monocular depth restoration model.

Step 2: Fine Gesture Extraction:

2.1) Gesture detection and positioning: perform gesture detection and positioning on the current frame to obtain a gesture detection frame.

2.2) Rough gesture mask generation: use the mixed Gaussian model to extract the foreground, then initialize an RGBA image with the same resolution as the current frame as the gesture mask, and set the RGB value of each pixel in the gesture mask to the current frame The RGB value of the element, set the Alpha value of the pixel located in the foreground area and the gesture detection border to 1, and set the Alpha value of other pixels to 0 to obtain a rough gesture mask.

2.3), fine gesture mask generation: use the seed algorithm to detect the pixel connected domains with an Alpha value of 0 within the range of the gesture detection frame in the rough gesture mask, calculate the number of pixels in each connected domain, if the number of pixels is less than the threshold, then The connected domain is noise, and the alpha values of all pixels in the connected domain are set to 1. This method removes small area noise and obtains a fine gesture mask.

Step 3: 3D model rendering:

3.1), adopt the camera tracking method based on the marker template, perform feature extraction and matching on the current video frame and the marker image, and obtain the camera attitude information corresponding to the current frame, that is, the rotation matrix from the current camera coordinate system to the three-dimensional world coordinate system and translation matrix;

3.2), first draw the current frame as the window background, then transform the 3D model into the screen space according to the camera internal reference, and the rotation matrix and translation matrix obtained in 3.1), and draw it on the window background, and obtain and save the depth map as Model depth map.

Step 4: Occlusion judgment and gesture redrawing:

4.1) Occlusion judgment: Sampling the model depth map and scene depth map with pixel coordinates with an Alpha value of 1 in the gesture mask to obtain the model depth and gesture depth respectively. If the gesture depth < the scene depth, it is considered that the gesture at this position is harmful to the virtual 3D If the model is occluded, keep the Alpha value of the corresponding pixel in the gesture mask as 1, otherwise set the Alpha value to 0;

4.2) Gesture redrawing: For pixels with an Alpha value of 1, the RGB value is used to cover the window background value, and for pixels with an Alpha value of 0, the window background value is still retained.

Specifically, the following steps are included:

Step 1: Monocular scene depth reconstruction:

1.1), using depth sensors such as Kinect or double-sided vision to obtain enough samples to form a training set;

1.2) Using deep learning models such as FCRN to perform sample migration using the above training set, and obtain a monocular depth recovery model that satisfies gesture interaction scenarios;

1.3), restore the scene depth under the current frame through the trained model.

Step 2: Fine Gesture Extraction:

2.1) Gesture detection and positioning: Use YOLO and other deep learning models to perform gesture detection and positioning on the current frame to obtain gesture category c and gesture detection frame R.

2.2) Rough gesture mask generation: use the mixed Gaussian model to extract the foreground, then initialize an RGBA image with the same resolution as the current frame as the gesture mask, and set the RGB value of each pixel in the gesture mask to the current frame The RGB value of the element, set the Alpha value of the pixel located in the foreground area and the gesture detection border to 1, and set the Alpha value of other pixels to 0 to obtain a rough gesture mask TD.

2.3), fine gesture mask generation: use the seed algorithm to detect the connected domains of pixels with an Alpha value of 0 within the range of the gesture detection frame in the rough gesture mask TD, calculate the number of pixels in each connected domain, if the number of pixels is less than the threshold, Then the connected domain is noise, and the Alpha value of all pixels in the connected domain is set to 1. This method removes small area noise and obtains a fine gesture mask after denoising.

Step 3: 3D model rendering:

3.1), the video frame captured by the current camera is matched with the pre-set marker template in the system. The marker template can be a natural marker or an artificial marker; the matching process uses SIFT or other feature points and descriptors , get the accurate matching point set after filtering out the wrong matching point set through forward and reverse brute force matching;

3.2), calculate the homography matrix H according to the two pairs of point sets matched in the first step, after obtaining the homography matrix of the template image and the current frame, the four corner points (x, y) of the template image Find the corresponding position (x', y') in the current frame through homography transformation;

3.3), set the three-dimensional world coordinates of the four corner points of the template image as the template image coordinates plus a z coordinate of 0 (x, y, 0), through the three-dimensional coordinates (x, y, 0) and the current frame The coordinates (x', y') of the corresponding four corner points form a typical PnP problem, and the camera attitude information in the current state is obtained, that is, the rotation matrix R and the translation matrix of the current camera coordinate system and the three-dimensional world coordinate system T;

3.4) Obtain the internal parameter matrix K of the camera through the checkerboard calibration method in advance, transform the coordinate system through K, R, and T, transform the three-dimensional coordinates of the virtual three-dimensional object into the screen coordinate system, and use the current frame as the background for overlay display , and at the same time use the glReadPixels function in the 3D graphics engine such as OpenGL to obtain and save the depth map as the model depth map.

Step 4: Occlusion judgment and gesture redrawing:

4.1), the gesture depth D _A is obtained by sampling the scene depth map with pixel coordinates whose Alpha value is 1 in the gesture mask;

4.2), sampling the model depth map with the pixel coordinates of the gesture mask with an Alpha value of 1 to obtain the model depth D _3D ;

4.3), judge for each pixel, if D _A < D _3D , it is considered that the gesture has blocked the virtual 3D model, and keep the Alpha value of the corresponding pixel in the gesture mask as 1, otherwise set the Alpha value as 0;

4.4) For pixels with an Alpha value of 1 in the gesture mask, cover the window background value with RGB values, and for pixels with an Alpha value of 0, still retain the window background value. It is also possible to use a 3D rendering engine such as the glDrawPixels function in OpenGL to directly and transparently overlay and draw the gesture mask on the screen. When this function is drawn, the background color of the gesture mask with an Alpha value of 0 is still maintained.

The invention discloses a 3D model embedding method based on monocular vision in a gesture interactive video scene. The occlusion judgment of the gesture and the model is performed through the depth reconstruction of the monocular scene, and then the 3D model is seamlessly embedded. The specific technical route is shown in Figure 1.

Step 1, monocular scene depth reconstruction:

Perform scene depth reconstruction on the current frame to generate scene depth, as shown in Figure 2, to provide a basis for judging the occlusion relationship;

Step 2, fine gesture extraction:

First, perform gesture detection and positioning on the current frame to obtain the position and range of the gesture detection frame, then extract the foreground, use the common part of the foreground and the gesture detection frame as the gesture part, and then perform denoising processing to obtain a fine gesture mask, as shown in the figure 3 shown;

Step 3, 3D model rendering:

First draw the video as the background, then render the 3D model to the background, and finally redraw the part of the video where the gesture blocks the 3D model;

Step 4, occlusion judgment and gesture redrawing:

For the fine gesture mask extracted in step 2, the occlusion judgment is performed through the scene depth map generated in step 1 and the model depth map obtained in step 3. The specific process is shown in Figure 4, and then the occlusion gesture is redrawn to form a 3D model Seamless embedding effect, as shown in Figure 5.

A 3D model embedding method based on monocular vision in a gesture interactive video scene proposed by the present invention can not only effectively deal with the occlusion problem generated during gesture interaction, but also does not rely on expensive depth detection equipment, which meets the needs of practical applications.

Claims (1)

1. a 3D model embedding method based on monocular vision in a gesture interactive video scene, is characterized in that, comprises the following steps: Step 1: Monocular scene depth reconstruction: Use the depth sensor or binocular vision method to obtain the scene depth map, register the obtained scene depth map, and transform it into the camera coordinate system as the training set of the monocular depth restoration model; use the above training set to restore the monocular depth to the network Perform migration learning to obtain a monocular depth restoration model suitable for the scene; use the monocular depth restoration model to directly obtain the scene depth map of the current frame; Among them, the scene depth map acquisition is to directly recover the scene depth from the image captured by a single camera; Step 2: Fine Gesture Extraction: 2.1), gesture detection and positioning: perform gesture detection and positioning on the current frame to obtain a gesture detection frame; 2.2) Rough gesture mask generation: use the mixed Gaussian model to extract the foreground, then initialize an RGBA image with the same resolution as the current frame as the gesture mask, and set the RGB value of each pixel in the gesture mask to the current frame The RGB value of the element, set the Alpha value of the pixel located in the foreground area and the gesture detection frame range to 1, and set the Alpha value of other pixels to 0 to obtain a rough gesture mask; 2.3), fine gesture mask generation: use the seed algorithm to detect the pixel connected domains with an Alpha value of 0 within the range of the gesture detection frame in the rough gesture mask, calculate the number of pixels in each connected domain, if the number of pixels is less than the threshold, then The connected domain is noise, and the alpha value of all pixels in the connected domain is set to 1; the small area noise is removed by this method, and a fine gesture mask is obtained; Step 3: 3D model rendering: 3.1), adopt the camera tracking method based on the marker template, perform feature extraction and matching on the current video frame and the marker image, and obtain the camera attitude information corresponding to the current frame, that is, the rotation matrix from the current camera coordinate system to the three-dimensional world coordinate system and translation matrix; 3.2), first draw the current frame as the window background, then transform the 3D model into the screen space according to the camera internal reference, and the rotation matrix and translation matrix obtained in 3.1), and draw it on the window background, and obtain and save the depth map as model depth map; Wherein, the model depth map is obtained by reading the depth buffer of the 3D rendering engine; The 3D model embedding scene includes two video drawing processes and one 3D model rendering process. The drawing sequence is: first draw the video as the background, then render the 3D model to the background, and finally redraw the part of the video where the gesture blocks the 3D model ; Step 4: Occlusion judgment and gesture redrawing: 4.1) Occlusion judgment: Sampling the model depth map and scene depth map with pixel coordinates with an Alpha value of 1 in the gesture mask to obtain the model depth and gesture depth respectively. If the gesture depth < the scene depth, it is considered that the gesture at this position is harmful to the virtual 3D If the model is occluded, keep the Alpha value of the corresponding pixel in the gesture mask as 1, otherwise set the Alpha value to 0; 4.2) Gesture redrawing: For pixels with a gesture mask Alpha value of 1, cover the window background value with its RGB value, and for pixels with an Alpha value of 0, the window background value is still retained; Among them, in the gesture mask used, all pixels with an Alpha value of 1 form the part that needs to be redrawn due to the gesture occluding the 3D model.

Images