CN110276831A - Method and device for constructing three-dimensional model, equipment and computer-readable storage medium - Google Patents

Abstract

The application relates to a method and a device for constructing a three-dimensional model, a terminal device and a computer readable storage medium. The method comprises the steps of acquiring a visible light graph, and generating a central weight graph corresponding to the visible light graph, wherein the weight value represented by the central weight graph is gradually reduced from the center to the edge; inputting the visible light image and the central weight image into a main body detection model to obtain a main body region confidence image, wherein the main body detection model is obtained by training in advance according to the visible light image, the central weight image and a corresponding marked main body mask image of the same scene; determining a target subject in the visible light map according to a subject region confidence map; acquiring depth information corresponding to a target main body; and performing three-dimensional reconstruction on the target main body according to the target main body and the depth information corresponding to the target main body, returning to the step of acquiring the visible light image to acquire the visible light images at different acquisition angles until a three-dimensional model corresponding to the target main body is obtained, and improving the accuracy of three-dimensional model construction.

Description

Three-dimensional model construction method and device, device, computer readable storage medium

technical field

The present application relates to the field of computer technology, and in particular, to a method and apparatus for constructing a three-dimensional model, a terminal device, and a computer-readable storage medium.

Background technique

With the development of imaging technology, people are more and more accustomed to shooting images or videos through image acquisition devices such as cameras on electronic devices to record various information. Because of the stronger realism of three-dimensional image processing, it has received widespread attention.

The traditional 3D model is often affected by surrounding people or objects during 3D construction, resulting in a low construction accuracy of the 3D model.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a method, device, terminal device, and computer-readable storage medium for constructing a three-dimensional model, and obtain an accurate confidence map of the subject area according to the center weight map and the subject detection model, so as to accurately identify the target subject in the image , during the construction of the three-dimensional model, through the depth information of the target subject, the accurate construction of the three-dimensional model corresponding to the target subject is realized, and the accuracy of the three-dimensional model construction is improved.

A method for constructing a three-dimensional model, the method comprising:

acquiring a visible light map, and generating a center weight map corresponding to the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge;

Input the visible light map and the center weight map into the subject detection model to obtain the confidence map of the subject area, wherein the subject detection model is based on the visible light map of the same scene, the center weight map and the corresponding marked The model obtained by training the subject mask map;

determining the target subject in the visible light image according to the subject area confidence map;

obtaining depth information corresponding to the target subject;

According to the target subject and the depth information corresponding to the target subject, perform three-dimensional reconstruction on the target subject, and return to the step of obtaining visible light images to obtain visible light images at different acquisition angles until the three-dimensional model corresponding to the target subject is obtained. .

A device for constructing a three-dimensional model, the device comprising:

a processing module, configured to acquire a visible light map and generate a center weight map corresponding to the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge;

The detection module is used to input the visible light map and the center weight map into the subject detection model to obtain the confidence map of the subject area, wherein the subject detection model is based on the visible light map, the center weight map and the center weight map of the same scene in advance. The model obtained by training the corresponding annotated subject mask map;

a target subject determination module, configured to determine the target subject in the visible light map according to the subject area confidence map;

A three-dimensional model building module is used to obtain the depth information corresponding to the target body, perform three-dimensional reconstruction of the target body according to the target body and the depth information corresponding to the target body, and return to the step of obtaining the visible light map to obtain Visible light images at different acquisition angles are obtained until a three-dimensional model corresponding to the target body is obtained.

A terminal device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the following steps:

acquiring a visible light map, and generating a center weight map corresponding to the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge;

Input the visible light map and the center weight map into the subject detection model to obtain the confidence map of the subject area, wherein the subject detection model is based on the visible light map of the same scene, the center weight map and the corresponding marked The model obtained by training the subject mask map;

determining the target subject in the visible light image according to the subject area confidence map;

obtaining depth information corresponding to the target subject;

According to the target subject and the depth information corresponding to the target subject, perform three-dimensional reconstruction on the target subject, and return to the step of obtaining visible light images to obtain visible light images at different acquisition angles until the three-dimensional model corresponding to the target subject is obtained. .

A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

acquiring a visible light map, and generating a center weight map corresponding to the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge;

Input the visible light map and the center weight map into the subject detection model to obtain the confidence map of the subject area, wherein the subject detection model is based on the visible light map of the same scene, the center weight map and the corresponding marked The model obtained by training the subject mask map;

determining the target subject in the visible light image according to the subject area confidence map;

obtaining depth information corresponding to the target subject;

According to the target subject and the depth information corresponding to the target subject, perform three-dimensional reconstruction on the target subject, and return to the step of obtaining visible light images to obtain visible light images at different acquisition angles until the three-dimensional model corresponding to the target subject is obtained. .

The construction method, device, terminal device, and computer-readable storage medium of the above-mentioned three-dimensional model generate a center weight map corresponding to the visible light map by acquiring the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge. ; Input the visible light map and the center weight map into the subject detection model to obtain the confidence map of the subject area, wherein the subject detection model is pre-processed according to the visible light map, the center weight map and the corresponding marked subject mask map of the same scene. The model obtained by training; determine the target subject in the visible light image according to the confidence map of the subject area; obtain the depth information corresponding to the target subject; perform three-dimensional reconstruction of the target subject according to the depth information corresponding to the target subject and the target subject, and return to obtain the visible light image Steps to obtain visible light images at different acquisition angles until the 3D model corresponding to the target subject is obtained. Using the center weight map can make the object in the center of the image easier to detect. Using the trained visible light map, center weight map and subject mask The subject detection model obtained by training such as images can more accurately identify the target subject in the visible light image. During the construction of the 3D model, the depth information of the target subject is used to realize the accurate construction of the 3D model corresponding to the target subject. In the presence of interfering objects It can also accurately identify the target subject in the case of the target subject, thereby improving the accuracy of the construction of the 3D model corresponding to the target subject.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1 is a block diagram of the internal structure of a terminal device in one embodiment;

2 is a flowchart of a method for constructing a three-dimensional model in one embodiment;

3 is a schematic diagram of a three-dimensional model corresponding to a target body in one embodiment;

4 is a flowchart of determining a target subject in the visible light map according to the subject area confidence map in one embodiment;

5 is a schematic diagram of a network structure of a subject detection model in one embodiment;

6 is a schematic diagram of a subject detection effect in one embodiment;

7 is a structural block diagram of an apparatus for constructing a three-dimensional model in one embodiment;

FIG. 8 is an internal structural diagram of a terminal device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The method for constructing a three-dimensional model in the embodiment of the present application can be applied to a terminal device. The terminal device may be a computer device with a camera, a personal digital assistant, a tablet computer, a smart phone, a wearable device, and the like. When the camera in the terminal device captures an image, it will automatically focus to ensure that the captured image is clear.

In one embodiment, the above-mentioned terminal device may include an image processing circuit, which may be implemented by hardware and/or software components, and may include various processing units that define an ISP (Image Signal Processing, image signal processing) pipeline. FIG. 1 is a schematic diagram of an image processing circuit in one embodiment. As shown in FIG. 1 , for the convenience of description, only various aspects of the image processing technology related to the embodiments of the present application are shown.

As shown in FIG. 1 , the image processing circuit includes a first ISP processor 130 , a second ISP processor 140 and a control logic 150 . The first camera 110 includes one or more first lenses 112 and a first image sensor 114 . The first image sensor 114 may include a color filter array (eg, a Bayer filter), the first image sensor 114 may acquire light intensity and wavelength information captured with each imaging pixel of the first image sensor 114, and provide information that can be accessed by the first ISP A set of image data processed by processor 130 . The second camera 120 includes one or more second lenses 122 and a second image sensor 124 . The second image sensor 124 may include a color filter array (eg, a Bayer filter), the second image sensor 124 may obtain light intensity and wavelength information captured with each imaging pixel of the second image sensor 124, and provide information that can be accessed by the second ISP A set of image data processed by processor 140 .

The first image collected by the first camera 110 is transmitted to the first ISP processor 130 for processing. After the first ISP processor 130 processes the first image, the statistical data of the first image (such as the brightness of the image, the contrast value of the image, etc.) , image color, etc.) to the control logic 150, and the control logic 150 can determine the control parameters of the first camera 110 according to the statistical data, so that the first camera 110 can perform auto-focus, auto-exposure and other operations according to the control parameters. After being processed by the first ISP processor 130, the first image may be stored in the image memory 160, and the first ISP processor 130 may also read the image stored in the image memory 160 for processing. In addition, after being processed by the ISP processor 130, the first image can be directly sent to the display 170 for display, and the display 170 can also read the image in the image memory 160 for display.

Among them, the first ISP processor 130 processes the image data pixel by pixel in various formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the first ISP processor 130 may perform one or more image processing operations on the image data, collecting statistical information about the image data. Among them, the image processing operations can be performed with the same or different bit depth precision.

The image memory 160 may be a part of a memory device, a storage device, or an independent dedicated memory in a terminal device, and may include a DMA (Direct Memory Access, direct memory access) feature.

Upon receiving the interface from the first image sensor 114, the first ISP processor 130 may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to image memory 160 for additional processing before being displayed. The first ISP processor 130 receives processing data from the image memory 160 and performs image data processing in RGB and YCbCr color spaces on the processed data. The image data processed by the first ISP processor 130 may be output to the display 170 for viewing by the user and/or further processed by a graphics engine or a GPU (Graphics Processing Unit, graphics processor). In addition, the output of the first ISP processor 130 may also be sent to the image memory 160 , and the display 170 may read image data from the image memory 160 . In one embodiment, image memory 160 may be configured to implement one or more frame buffers.

Statistics determined by the first ISP processor 130 may be sent to the control logic 150 . For example, the statistics may include first image sensor 114 statistics such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, first lens 112 shading correction, and the like. The control logic 150 may include a processor and/or a microcontroller executing one or more routines (eg, firmware) that may determine the control parameters and the first camera 110 based on the received statistics. A control parameter of the ISP processor 130. For example, the control parameters of the first camera 110 may include gain, integration time for exposure control, anti-shake parameters, flash control parameters, first lens 112 control parameters (eg, focal length for focusing or zooming), or a combination of these parameters. ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (eg, during RGB processing), and first lens 112 shading correction parameters.

Similarly, the second image captured by the second camera 120 is transmitted to the second ISP processor 140 for processing. After the second ISP processor 140 processes the first image, the statistical data of the second image (such as the brightness of the image, the The contrast value, the color of the image, etc.) are sent to the control logic 150, and the control logic 150 can determine the control parameters of the second camera 120 according to the statistical data, so that the second camera 120 can perform automatic focusing, automatic exposure and other operations according to the control parameters. . After being processed by the second ISP processor 140, the second image can be stored in the image memory 160, and the second ISP processor 140 can also read the image stored in the image memory 160 for processing. In addition, after being processed by the ISP processor 140, the second image can be directly sent to the display 170 for display, and the display 170 can also read the image in the image memory 160 for display. The second camera 120 and the second ISP processor 140 can also implement the processing procedures as described for the first camera 110 and the first ISP processor 130 .

In one embodiment, the first camera 110 may be a color camera, and the second camera 120 may be a TOF (Time Of Flight) camera or a structured light camera. The TOF camera can obtain the TOF depth map, and the structured light camera can obtain the structured light depth map. The first camera 110 and the second camera 120 may both be color cameras. The binocular depth map is acquired by two color cameras. The first ISP processor 130 and the second ISP processor 140 may be the same ISP processor.

The first camera 110 and the second camera 120 capture the same scene to obtain a visible light image and a depth image, respectively, and send the visible light image and the depth image to the ISP processor. The ISP processor can register the visible light map and the depth map according to the camera calibration parameters to keep the field of view completely consistent; and then generate a center weight map corresponding to the visible light map, wherein the weight value represented by the center weight map is from the center to the edge. Gradually decrease; input the visible light map and the center weight map into the trained subject detection model to obtain the confidence map of the subject area, and then determine the target subject in the visible light map according to the confidence map of the subject area; you can also use the visible light map, depth The image and the center weight map are input into the trained subject detection model, and the confidence map of the subject area is obtained, and then the target subject in the visible light image is determined according to the confidence map of the subject area. Using the center weight map can make objects located in the center of the image easier to detect, and using the depth map can make objects closer to the camera easier to detect, improving the accuracy of subject detection. When constructing a 3D model, through the depth information of the target subject, the accurate construction of the 3D model corresponding to the target subject is realized, and the target subject can also be accurately identified in the presence of interfering objects, thereby improving the accuracy of the 3D model construction corresponding to the target subject. .

FIG. 2 is a flowchart of a method for constructing a three-dimensional model in one embodiment. As shown in FIG. 2, a method for constructing a three-dimensional model, which can be applied to the terminal device in FIG. 1, includes:

Step 202, obtaining a visible light image.

Among them, salient object detection refers to automatically processing the region of interest and selectively ignoring the uninteresting region when facing a scene. The region of interest is called the subject region. Visible light images refer to RGB (Red, Green, Blue) images. Color images, ie RGB images, can be obtained by shooting any scene with a color camera. The visible light image may be stored locally by the terminal device, may also be stored by other devices, may also be stored from the network, or may be captured in real time by the terminal device, which is not limited thereto. .

Specifically, the ISP processor or the central processing unit of the terminal device can obtain the visible light image from a local or other device or network, or obtain the visible light image by photographing a scene with a camera.

Step 204: Generate a center weight map corresponding to the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge.

The central weight map refers to a map used to record the weight values of each pixel in the visible light map. The weight value recorded in the center weight map gradually decreases from the center to the four sides, that is, the center weight is the largest, and the weight gradually decreases toward the four sides. The weight value from the image center pixel point to the image edge pixel point of the visible light map is gradually reduced by the center weight map.

The ISP processor or the central processing unit can generate the corresponding center weight map according to the size of the visible light map. The weight value represented by the center weight map gradually decreases from the center to the four sides. The center weight map can be generated by using a Gaussian function, or using a first-order equation, or a second-order equation. The Gaussian function may be a two-dimensional Gaussian function.

Step 206: Input the visible light map and the center weight map into the subject detection model to obtain a confidence map of the subject area, wherein the subject detection model is based on the visible light map, the center weight map and the corresponding marked subject mask of the same scene in advance. The model obtained by training the membrane map.

Among them, the subject detection model is obtained by collecting a large amount of training data in advance, and inputting the training data into the subject detection model including the initial network weights for training. Each set of training data includes the visible light map, center weight map and annotated subject mask map corresponding to the same scene. Among them, the visible light map and the center weight map are used as the input of the trained subject detection model, and the annotated subject mask map is used as the ground truth that the trained subject detection model expects to output. The subject mask map is an image filter template used to identify the subject in the image, which can block other parts of the image and filter out the subject in the image. The subject detection model can be trained to recognize and detect various subjects, such as people, flowers, cats, dogs, backgrounds, etc.

Specifically, the ISP processor or the central processing unit can input the visible light map and the center weight map into the subject detection model, and the subject area confidence map can be obtained by performing detection. The confidence map of the subject area is used to record the probability of which identifiable subject the subject belongs to. For example, the probability of a pixel belonging to a person is 0.8, the probability of a flower is 0.1, and the probability of the background is 0.1.

Step 208: Determine the target subject in the visible light image according to the subject area confidence map.

Among them, the subject refers to various objects, such as people, flowers, cats, dogs, cows, blue sky, white clouds, background, etc. The target subject refers to the required subject, which can be selected as required.

Specifically, the ISP processor or the central processing unit can select the highest or second highest confidence level as the main body in the visible light map according to the confidence level map of the main body area. If there is one main body, the main body is used as the target main body; One or more subjects can be selected as target subjects according to configuration information or depth information. In one embodiment, the distance between each subject and the shooting terminal is determined according to the depth information corresponding to each subject, and the subject with the smallest distance is used as the target subject.

Step 210: Acquire depth information corresponding to the target body, perform three-dimensional reconstruction of the target body according to the target body and the depth information corresponding to the target body, and return to step 202 to obtain visible light images at different acquisition angles until a three-dimensional model corresponding to the target body is obtained .

Among them, if the target subject has an accurate contour, it is only necessary to obtain the depth information corresponding to each pixel in the contour of the target subject, and the depth information corresponding to the target subject can be obtained from the depth map captured by the camera, wherein the method of obtaining the depth map is not limited.

Specifically, three-dimensional reconstruction is performed on the target subject according to the target subject and depth information corresponding to the target subject, and the manner of performing the three-dimensional reconstruction is not limited. The depth information represents the distance between each pixel in the target subject and the shooting device. According to the depth information, the Z-axis coordinates of each pixel in the target subject in the three-dimensional space can be determined, so as to perform three-dimensional reconstruction of the target subject. When performing 3D reconstruction, the pixels with the same depth information are in the same plane, you can select reference pixels, use the depth value corresponding to the reference pixel as the reference depth value, and compare the depth values corresponding to other pixels with the reference depth value. relationship, so as to determine the position of other pixel points in the three-dimensional space relative to the reference pixel point. For example, the depth value of a pixel on the target subject is 10, and the depth value of the adjacent pixel is 12, then this object uses the point with the depth value of 10 as the reference pixel, and the adjacent pixel will be concave 12-10 = 2, the three-dimensional reconstruction of the target body is to give relative concave-convex information on the body surface corresponding to the target body. Since the current visible light image only captures a part of the target subject, it is necessary to obtain visible light images from other acquisition angles. If the visible light image is captured in real time, the acquisition angle can be changed to shoot each part of the target subject and obtain the corresponding parts of each part. The depth information of the complete target body can be reconstructed in 3D. If the captured visible light image is obtained, the complete target subject can be 3D reconstructed from directly obtained visible light images at other shooting angles until the 3D model corresponding to the target subject is obtained. As shown in FIG. 3 , it is a schematic diagram of the constructed three-dimensional model corresponding to the target body in one embodiment.

The construction method of the three-dimensional model in this embodiment generates a center weight map corresponding to the visible light map by acquiring the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge; The image is input into the subject detection model, and the confidence map of the subject area is obtained. The subject detection model is a model that is pre-trained based on the visible light map, the center weight map and the corresponding labeled subject mask map of the same scene; The regional confidence map determines the target subject in the visible light map; obtains the depth information corresponding to the target subject; performs three-dimensional reconstruction of the target subject according to the depth information corresponding to the target subject and the target subject, and returns to the steps of obtaining the visible light map to obtain different acquisition angles Until the 3D model corresponding to the target subject is obtained, the object in the center of the image can be more easily detected by using the center weight map. The model can more accurately identify the target subject in the visible light image. During the construction of the 3D model, the depth information of the target subject is used to realize the accurate construction of the 3D model corresponding to the target subject, and it can also be accurately identified in the presence of interfering objects. The target subject, thereby improving the accuracy of the construction of the 3D model corresponding to the target subject.

In one embodiment, as shown in FIG. 4, step 208 includes:

Step 208A, processing the confidence map of the subject region to obtain a subject mask map.

Specifically, there are some low-confidence and scattered points in the confidence map of the subject area, and the subject mask map can be obtained by filtering the confidence map of the subject area through the ISP processor or the central processing unit. In the filtering process, a configured confidence threshold can be used to filter the pixels whose confidence value is lower than the confidence threshold in the confidence map of the subject area. The confidence threshold can be an adaptive confidence threshold, a fixed threshold, or a corresponding threshold can be configured by region.

Step 208B: Detect the visible light image, and determine the highlight area in the visible light image.

Among them, the highlight area refers to the area whose brightness value is greater than the brightness threshold value.

Specifically, the ISP processor or the central processing unit performs highlight detection on the visible light image, selects target pixels with a brightness value greater than a brightness threshold, and uses connected domain processing on the target pixels to obtain highlight regions.

Step 208C, according to the highlight area in the visible light image and the subject mask image, determine a target subject for eliminating highlights in the visible light image.

Specifically, the ISP processor or the central processing unit may perform differential calculation or logical AND calculation between the highlight area in the visible light image and the subject mask image to obtain the target subject for eliminating highlights in the visible light image.

In this embodiment, the subject mask map is obtained by filtering the confidence map of the subject area, which improves the reliability of the confidence map of the subject area. The visible light map is detected to obtain the highlight area, and then processed with the subject mask map, which can be A target subject with no highlights is obtained, and a filter is used to process the highlight and highlight areas that affect the accuracy of subject recognition, thereby improving the accuracy and accuracy of subject recognition.

In one embodiment, step 208A includes: performing an adaptive confidence threshold filtering process on the subject area confidence map to obtain a subject mask map.

The adaptive confidence threshold refers to a confidence threshold. The adaptive confidence threshold may be a locally adaptive confidence threshold. The locally adaptive confidence threshold is to determine the binarization confidence threshold at the pixel position according to the pixel value distribution of the domain block of the pixel. The binarization confidence threshold is configured to be higher for image regions with higher brightness, and the binarization threshold confidence for image regions with lower brightness is configured to be lower.

In one embodiment, the configuration process of the adaptive confidence threshold includes: when the brightness value of the pixel point is greater than the first brightness value, configure the first confidence threshold value, and when the brightness value of the pixel point is less than the second brightness value, configure The second confidence threshold, when the brightness value of the pixel point is greater than the second brightness value and less than the first brightness value, a third confidence threshold is configured, where the second brightness value is less than or equal to the first brightness value, and the second confidence threshold The threshold is smaller than the third confidence threshold, and the third confidence threshold is smaller than the first confidence threshold.

In one embodiment, the configuration process of the adaptive confidence threshold includes: when the brightness value of the pixel point is greater than the first brightness value, configuring the first confidence threshold value, and when the brightness value of the pixel point is less than or equal to the first brightness value, Then a second confidence threshold is configured, wherein the second brightness value is less than or equal to the first brightness value, and the second confidence threshold is less than the first confidence threshold.

When performing adaptive confidence threshold filtering on the confidence map of the main area, the confidence value of each pixel in the confidence map of the main area is compared with the corresponding confidence threshold, and if it is greater than or equal to the confidence threshold, the pixel is retained. If it is less than the confidence threshold, the pixel will be removed.

In one embodiment, the subject region confidence map is subjected to adaptive confidence threshold filtering to obtain a subject mask map, including:

Perform adaptive confidence threshold filtering on the confidence map of the subject area to obtain a binarized mask map; perform morphological processing and guided filtering on the binarized mask map to obtain a subject mask map.

Specifically, after the ISP processor or the central processing unit filters the confidence level map of the main area according to the adaptive confidence level threshold, the confidence level value of the retained pixel points is represented by 1, and the confidence level value of the removed pixel points is represented by 0. , to get the binarized mask image.

Morphological treatments can include corrosion and swelling. The binarized mask image can be etched first, and then expanded to remove noise; then the morphologically processed binarized mask image can be subjected to guided filtering to realize the edge filtering operation, and obtain the main mask of edge extraction. Membrane diagram.

Morphological processing and guided filtering can ensure that the obtained subject mask has less or no noise and softer edges.

In one embodiment, determining a target subject to eliminate highlights in the visible light image according to the highlight area in the visible light image and the subject mask image includes: comparing the highlight area in the visible light image with the subject mask image Differential processing to get the target subject to eliminate highlights.

Specifically, the ISP processor or the central processing unit performs differential processing between the highlight area in the visible light image and the main mask image, that is, the corresponding pixel values in the visible light image and the main mask image are subtracted to obtain the visible light image. target subject. The target subject with highlights removed is obtained through differential processing, and the calculation method is simple.

In one embodiment, the subject detection model includes an input layer, an intermediate layer, and an output layer connected in sequence. Inputting the visible light map and the center weight map into the subject detection model includes: applying the visible light map to the input layer of the subject detection model; and applying the center weight map to the output layer of the subject detection model.

The subject detection model can use a deep learning network model. The deep learning network model may include an input layer, an intermediate layer and an output layer which are connected in sequence. The intermediate layer may be a one-layer or at least two-layer network structure. The visible light map is input from the input layer of the subject detection model, which acts on the input layer of the subject detection model. The central weight map is input in the output layer of the subject detection model, that is, it acts on the output layer of the subject detection model. Applying the center weight map to the output layer of the subject detection model can reduce the influence of other layers of the subject detection model on the weight map, making it easier for objects in the center of the screen to be detected as subjects.

In one embodiment, acquiring depth information corresponding to the target subject in step 210 includes: acquiring a depth map corresponding to a visible light map; the depth map includes at least one of a TOF depth map, a binocular depth map, and a structured light depth map. The image and the depth map are registered to obtain the registered visible light map and depth map, and the depth information corresponding to the target subject is determined from the registered depth map according to the region where the target subject is located in the visible light image.

The depth map refers to a map containing depth information. The corresponding depth map is obtained by shooting the same scene with a depth camera or a binocular camera. The depth camera can be a structured light camera or a TOF camera. The depth map may be at least one of a structured light depth map, a TOF depth map, and a binocular depth map.

Specifically, the ISP processor or the central processing unit can capture the same scene with the camera to obtain the visible light image and the corresponding depth map, and then use the camera calibration parameters to register the visible light image and the depth map to obtain the registered visible light image and depth map. . In the registered visible light map and depth map, each pixel in the visible light map has matching pixels in the depth map, so that the corresponding pixels in the depth map of the region where the target subject is located can be obtained according to the matching relationship. , and obtain the depth value corresponding to the target subject according to the pixel value of the pixel point.

In one embodiment, when the depth map cannot be obtained by shooting, a simulated depth map can be automatically generated. The depth value of each pixel in the simulated depth map may be a preset value. In addition, the depth values of each pixel in the simulated depth map may correspond to different preset values.

In one embodiment, it includes: inputting the registered visible light map, the depth map and the center weight map into a subject detection model to obtain a subject area confidence map; wherein the subject detection model is based on the same scene in advance The model obtained by training the visible light map, depth map, center weight map and corresponding annotated subject mask map.

Among them, the subject detection model is obtained by collecting a large amount of training data in advance, and inputting the training data into the subject detection model including the initial network weights for training. Each set of training data includes the visible light map, depth map, center weight map and annotated subject mask map corresponding to the same scene. Among them, the visible light map and the center weight map are used as the input of the trained subject detection model, and the annotated subject mask map is used as the actual value that the trained subject detection model expects to output. The subject mask map is an image filter template used to identify the subject in the image, which can block other parts of the image and filter out the subject in the image. The subject detection model can be trained to recognize and detect various subjects, such as people, flowers, cats, dogs, backgrounds, etc.

In this embodiment, the depth map and the center weight map are used as the input of the subject detection model. The depth information of the depth map can be used to make objects closer to the camera easier to be detected. The center weight map is used to have a large center weight and a small weight on the four sides. The central attention mechanism makes it easier to detect the object in the center of the image. The depth map is introduced to enhance the depth feature of the subject, and the center weight map is introduced to enhance the central attention feature of the subject, which can not only accurately identify the target subject in a simple scene , which greatly improves the accuracy of subject recognition in complex scenes, and the introduction of depth maps can solve the problem of poor robustness of traditional target detection methods to ever-changing targets in natural images. A simple scene refers to a scene with a single subject and low contrast in the background area.

In one embodiment, the training method of the subject detection model includes: acquiring a visible light map, a depth map and an labeled subject mask map of the same scene; generating a center weight map corresponding to the visible light map, wherein the center weight The weight value represented by the figure gradually decreases from the center to the edge; the visible light map is applied to the input layer of the subject detection model containing the initial network weights, and the depth map and the center weight map are applied to the output of the initial subject detection model layer, take the marked subject mask image as the real value output by the subject detection model, train the subject detection model including the initial network weight, and obtain the target network weight of the subject detection model.

The visible light map, depth map and corresponding annotated subject mask map of a scene can be collected. Semantic-level annotation is performed on the visible light map and the depth map, and the subject inside is marked. A large number of visible light images can be collected, and then based on the foreground target image in the COCO dataset and a simple background image, a large number of images with solid color backgrounds or simple backgrounds can be obtained as training visible light images. The COCO dataset contains a large number of foreground objects.

The network structure of the subject detection model adopts the architecture based on mobile-Unet, and the bridge between layers is added in the decoder part, so that the high-level semantic features can be more fully transmitted during up-sampling. The central weight map acts on the output layer of the subject monitoring model, and introduces a central attention mechanism to make it easier for objects in the center of the screen to be detected as subjects.

The network structure of the subject detection model includes an input layer, a convolution layer (conv), a pooling layer (pooling), a bilinear interpolation layer (Bilinear Up sampling), a convolutional feature connection layer (concat+conv), and an output layer. The deconvolution+add (deconvolution feature stacking) operation is used to bridge between the bilinear interpolation layer and the convolution feature connection layer, so that the high-level semantic features are more fully transmitted during upsampling. Convolutional layers, pooling layers, bilinear interpolation layers, convolutional feature connection layers, etc. can be intermediate layers of the subject detection model.

The initial network weights refer to the initial weights of each layer of the initialized deep learning network model. The target network weight refers to the weight of each layer of the deep learning network model trained to detect the subject of the image. The target network weight can be obtained by preset training times, and the loss function of the deep learning network model can also be set. When the loss function value obtained from training is less than the loss threshold, the current network weight of the subject detection model is used as the target network weight.

FIG. 5 is a schematic diagram of a network structure of a subject detection model in one embodiment. As shown in FIG. 5 , the network structure of the subject detection model includes a convolutional layer 402, a pooling layer 404, a convolutional layer 406, a pooling layer 408, a convolutional layer 410, a pooling layer 412, a convolutional layer 414, a pooling layer layer 416, convolutional layer 418, convolutional layer 420, bilinear interpolation layer 422, convolutional layer 424, bilinear interpolation layer 426, convolutional layer 428, convolutional feature connection layer 430, bilinear interpolation layer 432 , convolutional layer 434, convolutional feature connection layer 436, bilinear interpolation layer 438, convolutional layer 440, convolutional feature connection layer 442, etc. The convolutional layer 402 is used as the input layer of the subject detection model, and the convolutional feature connection layer 442 as the output layer of the subject detection model. The network structure of the subject detection model in this embodiment is only an example, and is not intended to limit the present application. It can be understood that, multiple convolution layers, pooling layers, bilinear interpolation layers, convolution feature connection layers, etc. in the network structure of the subject detection model can be set as required.

The encoding part of the subject detection model includes convolutional layer 402, pooling layer 404, convolutional layer 406, pooling layer 408, convolutional layer 410, pooling layer 412, convolutional layer 414, pooling layer 416, convolutional layer 410 Layer 418, the decoding part includes convolutional layer 420, bilinear interpolation layer 422, convolutional layer 424, bilinear interpolation layer 426, convolutional layer 428, convolutional feature connection layer 430, bilinear interpolation layer 432, volume A convolutional layer 434 , a convolutional feature connection layer 436 , a bilinear interpolation layer 438 , a convolutional layer 440 , and a convolutional feature connection layer 442 . The convolutional layer 406 is concatenated with the convolutional layer 434 , the convolutional layer 410 is concatenated with the convolutional layer 428 , and the convolutional layer 414 is concatenated with the convolutional layer 424 . The bilinear interpolation layer 422 and the convolutional feature connection layer 430 are bridged using Deconvolution+add. The bilinear interpolation layer 432 and the convolutional feature connection layer 436 employ deconvolutional feature stacking bridges. The bilinear interpolation layer 438 and the convolutional feature connection layer 442 are bridged using deconvolutional feature stacking.

The original image 450 (such as a visible light image) is input to the convolutional layer 402 of the subject detection model, the depth map 460 acts on the convolutional feature connection layer 442 of the subject detection model, and the center weight map 470 acts on the convolutional feature connection layer of the subject detection model. 442. The depth map 460 and the center weight map 470 are each input to the convolutional feature connection layer 442 as a multiplication factor. The original image 450 , the depth map 460 and the center weight map 470 are input to the subject detection model and output a confidence map 480 containing the subject.

During the training process of the subject detection model, a loss rate of a preset value is used for the depth map. The preset value may be 50%. Probabilistic dropout is introduced in the training process of the depth map, so that the subject detection model can fully mine the information of the depth map. When the subject detection model cannot obtain the depth map, it can still output accurate results. The dropout method is used for the depth map input, which makes the subject detection model more robust to the depth map, and can accurately segment the subject area even without a depth map.

In addition, in the normal shooting process of terminal equipment, the shooting and calculation of the depth map are time-consuming and labor-intensive, and it is difficult to obtain. The depth map is designed with a dropout probability of 50% during training, which can ensure the detection of subjects when there is no depth information. The model can still be detected normally.

The highlight detection layer 444 is used to perform highlight detection on the original image 450 to identify the highlight area in the original image. Perform adaptive threshold filtering on the confidence map of the subject area output by the subject detection model to obtain a binarized mask map, and perform morphological processing and guided filtering on the binarized mask map to obtain the subject mask map. The film image is differentially processed with the original image containing the highlight area, and the highlight area is deleted from the subject mask image to obtain the subject with the highlight removed. The confidence map of the main area is a confidence map distributed from 0 to 1. The confidence map of the main area contains more noise points, and there are many noise points with low confidence, or small high-confidence areas that are aggregated together. The adaptive confidence threshold is filtered to obtain a binarized mask image. Morphological processing of the binarized mask image can further reduce noise, and guided filtering processing can make the edges smoother. It can be understood that the confidence map of the subject region may be a subject mask map containing noise.

In this embodiment, the depth map is used as a feature to enhance the network output results, and the depth map is not directly input into the network of the subject detection model, and a dual deep learning network structure can be designed in addition, one of which is used for deep learning network structure. The image is processed, another deep learning network structure is used to process the RGB image, and then the outputs of the two deep learning network structures are connected by convolutional features, and then output.

In one embodiment, the training method of the subject detection model includes: acquiring a visible light map and a marked subject mask map of the same scene; generating a center weight map corresponding to the visible light map, wherein the center weight map represents The weight value gradually decreases from the center to the edge; the visible light map is applied to the input layer of the subject detection model containing the initial network weights, the central weight map is applied to the output layer of the initial subject detection model, and the labeled The subject mask image is used as the real value output by the subject detection model, and the subject detection model including the initial network weight is trained to obtain the target network weight of the subject detection model.

The training in this embodiment uses the visible light map and the center weight map, that is, the depth map is not introduced in the output layer part of the network structure of the subject detection model in FIG. The convolutional feature connection layer 442 of the subject detection model.

FIG. 6 is a schematic diagram of the effect of target subject recognition in one embodiment. As shown in Fig. 6, there is a butterfly in the RGB image 502. After inputting the RGB image into the subject detection model, the subject area confidence map 504 is obtained, and then the subject area confidence map 604 is filtered and binarized to obtain the binarization mask image 506 , and then perform morphological processing and guided filtering on the binarized mask image 506 to achieve edge enhancement, and obtain a main mask image 508 .

In one embodiment, the step of returning to the step of acquiring visible light images to acquire visible light images of different acquisition angles includes: taking the target subject as the center, continuously changing the acquisition angle around the target subject, and overlapping between adjacent visible light images Under the condition of the area, the visible light image is collected in real time, and the visible light image of different collection angles is obtained.

Specifically, the acquisition angle can be converted by rotating the photographing device, and the magnitude of the conversion can be customized. For example, when photographing a cubic object, each time rotates 45 degrees to acquire visible light images of different angles. Since the collected visible light images need to cover the entire target body, an overlapping area needs to exist between adjacent visible light images to ensure the integrity of information collection, and the overlap ratio can be customized.

In this embodiment, by continuously changing the acquisition angle around the target body and collecting the visible light image in real time, the integrity of the information collection can be ensured, thereby realizing the three-dimensional reconstruction of the target body in real time.

In one embodiment, step 210 includes: acquiring a first depth value corresponding to the first plane pixel on the target body, and obtaining the first plane pixel in the target body according to the position and the first depth value of the first plane pixel For the first three-dimensional pixel point corresponding to the three-dimensional space, obtain the second depth value corresponding to the second plane pixel point on the target body, use the first depth value as the reference depth value, and determine the relative value of the second three-dimensional pixel point according to the second depth value. The relative position of the first three-dimensional pixel point, the second three-dimensional pixel point is the three-dimensional pixel point corresponding to the second plane pixel point in the three-dimensional space, and the second three-dimensional pixel point is determined according to the relative position and the position of the second plane pixel point in the target subject. The corresponding position of the three-dimensional pixel point in the three-dimensional space; connect each three-dimensional pixel point in the three-dimensional space.

Specifically, three-dimensional pixel point matching can be performed on each plane pixel point on the target body, and three-dimensional pixel point matching can also be performed on key plane pixel points on the target body, so as to improve the efficiency of three-dimensional reconstruction and save computer resources. . In one embodiment, when the target subject is a human face, the key plane pixels are the key plane pixels obtained by detecting the feature key points of the human face, such as points on the tip of the nose, eyes, mouth, and eyebrows. The construction of a 3D model requires a relative depth value. Select a point, such as the first plane pixel point, as the basic calibration point, and other points can calculate the depth value relative to this point, so that the corresponding depth values of the target subject in the 3D space can be gradually constructed. Three-dimensional pixel point, connecting each three-dimensional pixel point in the three-dimensional space to obtain the three-dimensional model corresponding to the target subject.

In one embodiment, step 210 includes: determining a target type corresponding to the target body, obtaining an initial three-dimensional model of the same type according to the target type, and obtaining actual depth values corresponding to key plane pixels on the target body; obtaining from the initial three-dimensional model The 3D model pixel points matched with the key plane pixels; adjust the 3D spatial position of the matched 3D model pixels according to the actual depth value ratio between the key plane pixels; obtain the actual corresponding non-key plane pixels on the target subject Depth value, obtain the 3D model pixel points matching the non-critical plane pixels from the initial 3D model, and adjust the 3D model pixels matching the non-critical plane pixels according to the actual depth value ratio between the non-critical plane pixels and the key plane pixels. The three-dimensional spatial position of the model pixel point, until each plane pixel point on the target body has a matching adjusted three-dimensional model pixel point; each adjusted three-dimensional model pixel point forms a three-dimensional model corresponding to the target body.

Specifically, the subject detection model not only outputs the outline of the target subject in the visible light image, but also outputs the target type corresponding to the target subject, and the target type includes people, flowers, cats, dogs, etc. The initial 3D model corresponding to each target type can be established in advance, such as the initial 3D model of the face, the initial 3D model of the human body, the initial 3D model of the flower, the initial 3D model of the cat, the initial 3D model of the dog, etc. The key plane pixels are the key feature points to determine the three-dimensionality of the three-dimensional model. For example, for a human face, the key plane pixels are obtained by detecting the human face. For example, the points on the tip of the nose, eyes, mouth, and eyebrows are the key plane pixels. Adjust the three-dimensional spatial position of the matched three-dimensional model pixels according to the actual depth value ratio between the key plane pixels. Thereby, the initial 3D model is adjusted to obtain the three-dimensional contour of the 3D model that matches the target body. For example, the nose of the initial 3D model is lower, but the nose of the actual target body is higher. The body matches the three-dimensional contour of the 3D model. The initial 3D model is further adjusted according to the actual depth values of other non-critical plane pixels to obtain a 3D model that better matches the target subject.

In this embodiment, since the initial three-dimensional model is a conventional model, the influence of the black hole formed on the depth map on the construction of the three-dimensional model due to the accuracy and inaccuracy of the depth map can be reduced. On the basis of the initial 3D model, a 3D model matching the target body is obtained by correcting step by step.

In one embodiment, the visible light image is collected in real time, and the method further includes: displaying the construction process of the three-dimensional model corresponding to the target body on the preview interface in real time according to the visible light image collected in real time.

Specifically, the visible light image including the target body can be collected in real time through the terminal device, the 3D model corresponding to the target body can be gradually constructed while collecting, and the construction process of the 3D model corresponding to the target body can be displayed in real time on the preview interface, so that the user can Intuitively watch the construction process of the 3D model. Therefore, the acquisition angle of the visible light image can be adjusted according to the displayed constructed three-dimensional model, so that the construction of the three-dimensional model corresponding to the target subject is more accurate and efficient.

In one embodiment, according to the texture and color of the visible light image, the corresponding texture and color are matched to the three-dimensional model corresponding to the target body. It should be understood that although the steps in the flowcharts of FIG. 2 and FIG. 4 are displayed in sequence according to the arrows, these steps are not necessarily executed in the sequence indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIG. 2 and FIG. 4 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. These sub-steps or stages may be executed at different times. The order of execution of the stages is also not necessarily sequential, but may be performed alternately or alternately with other steps or sub-steps of other steps or at least a portion of a stage.

FIG. 7 is a structural block diagram of an apparatus for constructing a three-dimensional model in an embodiment. As shown in FIG. 7 , a three-dimensional model construction device includes a processing module 602 , a detection module 604 , a target subject determination module 606 and a three-dimensional model construction module 608 . in:

The processing module 602 is configured to acquire a visible light map and generate a center weight map corresponding to the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge.

The detection module 604 is used to input the visible light map and the center weight map into the subject detection model to obtain the confidence map of the subject area, wherein the subject detection model is pre-based on the visible light map and the center weight map of the same scene and the corresponding annotated subject mask image for training.

A target subject determination module 606, configured to determine a target subject in the visible light map according to the subject area confidence map.

The three-dimensional model construction module 608 is used to obtain the depth information corresponding to the target body, perform three-dimensional reconstruction of the target body according to the target body and the depth information corresponding to the target body, and return to the processing module to obtain different collections The visible light map of the angle is obtained until the three-dimensional model corresponding to the target body is obtained.

The three-dimensional model construction device in this embodiment uses the center weight map to make the object in the center of the image easier to detect, and uses the trained subject detection model using the visible light map, the center weight map, and the subject mask map. The target subject in the visible light image can be more accurately identified. During the construction of the 3D model, the depth information of the target subject is used to realize the accurate construction of the 3D model corresponding to the target subject, and the target subject can also be accurately identified in the presence of interfering objects. , so as to improve the accuracy of the 3D model construction corresponding to the target subject.

In one embodiment, the target subject determination module 606 is further configured to process the confidence map of the subject region to obtain a subject mask map; detect the visible light map to determine the highlight region in the visible light map; The highlight area and the subject mask map determine the target subject to eliminate highlights in the visible light map.

In one embodiment, the target subject determination module 606 is further configured to perform an adaptive confidence threshold filtering process on the subject area confidence map to obtain a subject mask map.

In one embodiment, the target subject determination module 606 is further configured to perform adaptive confidence threshold filtering processing on the confidence map of the subject region to obtain a binarized mask map; perform morphological processing on the binarized mask map and guided filtering processing to obtain the subject mask map.

In one embodiment, the target subject determination module 606 is further configured to perform differential processing between the highlight area in the visible light image and the subject mask image to obtain the target subject in the visible light image.

In one embodiment, the subject detection model includes an input layer, an intermediate layer, and an output layer connected in sequence;

The detection module 604 is further configured to apply the visible light map to the input layer of the subject detection model; and to apply the center weight map to the output layer of the subject detection model.

In one embodiment, the three-dimensional model building module 608 is further configured to obtain a depth map corresponding to the visible light map; the depth map includes at least one of a TOF depth map, a binocular depth map and a structured light depth map; The visible light map and the depth map are registered to obtain the registered visible light map and the depth map, and the depth information corresponding to the target subject is determined from the registered depth map according to the region where the target subject is located in the visible light map.

In one embodiment, the three-dimensional model building module 608 is further configured to take the target body as the center, and continuously change the acquisition angle around the target body; under the condition that there is an overlapping area between adjacent visible light images, collect the visible light images in real time, and obtain different Visible light map of the acquisition angle.

In one embodiment, the three-dimensional model building module 608 is further configured to obtain the first depth value corresponding to the first plane pixel on the target body; obtain the first depth value according to the position of the first plane pixel on the target body and the first depth value The first three-dimensional pixel point corresponding to the plane pixel point in the three-dimensional space; the second depth value corresponding to the second plane pixel point on the target body is obtained; the first depth value is used as a reference depth value, and the second three-dimensional value is determined according to the second depth value. The relative position of the pixel point relative to the first three-dimensional pixel point, the second three-dimensional pixel point is the three-dimensional pixel point corresponding to the second plane pixel point in the three-dimensional space; determined according to the relative position and the position of the second plane pixel point in the target subject The position corresponding to the second three-dimensional pixel point in the three-dimensional space; connecting each three-dimensional pixel point in the three-dimensional space.

In one embodiment, the three-dimensional model building module 608 is further configured to determine the target type corresponding to the target body, and obtain an initial three-dimensional model of the same type according to the target type; obtain the actual depth value corresponding to the key plane pixels on the target body; Obtaining the three-dimensional model pixel points matching the key plane pixels in the three-dimensional model, and adjusting the three-dimensional space position of the matched three-dimensional model pixel points according to the actual depth value ratio between each key plane pixel point; acquiring the non-key points on the target subject The actual depth value corresponding to the plane pixel point; obtain the three-dimensional model pixel point matching the non-critical plane pixel point from the initial three-dimensional model; according to the actual depth value ratio between the non-critical plane pixel point and the key plane pixel point. The 3D spatial position of the 3D model pixels matched by the non-critical plane pixels, until each plane pixel on the target body has matching adjusted 3D model pixels; the adjusted 3D model pixels form the corresponding 3D model of the target body. Model.

In one embodiment, the apparatus further includes:

The display module is used to display the construction process of the three-dimensional model corresponding to the target body in real time on the preview interface according to the visible light image collected in real time.

In one embodiment, the detection module 604 is further configured to input the registered visible light map, depth map and center weight map into the subject detection model to obtain a subject region confidence map; wherein the subject detection model is based on The model obtained by training the visible light map, depth map, center weight map and corresponding annotated subject mask map of the same scene.

In one embodiment, the above-mentioned apparatus for constructing a three-dimensional model further includes a training image acquisition module, a training weight generation module and a training module.

The training image acquisition module is used to acquire the visible light map, depth map and annotated subject mask map of the same scene.

The training weight generation module is configured to generate a center weight map corresponding to the visible light map, wherein the weight value represented by the center weight map gradually decreases from the center to the edge.

The training module is used to apply the visible light map to the input layer of the subject detection model containing the initial network weights, apply the depth map and the center weight map to the output layer of the initial subject detection model, and apply the labeled The subject mask map of , is used as the real value output by the subject detection model, and the subject detection model including the initial network weight is trained to obtain the target network weight of the subject detection model. When the loss function of the subject detection model is smaller than the loss threshold, or when the number of training times reaches a preset number of times, the network weight of the subject detection model is used as the target network weight of the subject detection model.

FIG. 8 is a schematic diagram of an internal structure of a terminal device in an embodiment. As shown in FIG. 8, the terminal device includes a processor and a memory connected through a system bus. Among them, the processor is used to provide computing and control capabilities to support the operation of the entire terminal equipment. The memory may include non-volatile storage media and internal memory. The nonvolatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement a method for constructing a three-dimensional model provided by various embodiments. Internal memory provides a cached execution environment for operating system computer programs in non-volatile storage media. The terminal device may be a mobile phone, a tablet computer, a personal digital assistant, a wearable device, or the like.

The implementation of each module in the apparatus for constructing a three-dimensional model provided in the embodiments of the present application may be in the form of a computer program. The computer program can be run on a terminal or server. The program modules constituted by the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, the steps of the methods described in the embodiments of the present application are implemented.

The embodiment of the present application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of a method for constructing a three-dimensional model .

A computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of a method of constructing a three-dimensional model.

Any reference to a memory, storage, database, or other medium as used in embodiments of the present application may include non-volatile and/or volatile memory. Suitable nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), Memory Bus (Rambus) Direct RAM (RDRAM), Direct Memory Bus Dynamic RAM (DRDRAM), and Memory Bus Dynamic RAM (RDRAM).

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (10)

1. a kind of constructing method of threedimensional model, which is characterized in that the described method includes:

Visible light figure is obtained, generates center weight figure corresponding with the visible light figure, wherein represented by the center weight figure Weighted value be gradually reduced from center to edge；

The visible light figure and the center weight figure are input in subject detection model, body region confidence level figure is obtained, Wherein, the subject detection model is the visible light figure previously according to Same Scene, center weight figure and corresponding has marked The model that main body exposure mask figure is trained；

The target subject in the visible light figure is determined according to the body region confidence level figure；

Obtain the corresponding depth information of the target subject；

According to the target subject and the corresponding depth information of target subject, three-dimensionalreconstruction is carried out to the target subject, is returned Described the step of obtaining visible light figure, is corresponding until obtaining the target subject to obtain the visible light figure of different acquisition angle Threedimensional model.

2. the method according to claim 1, wherein described according to body region confidence level figure determination Target subject in visible light figure, comprising:

The body region confidence level figure is handled, main body exposure mask figure is obtained；

The visible light figure is detected, determines the highlight area in the visible light figure；

According in the visible light figure highlight area and the main body exposure mask figure, determine and eliminate bloom in the visible light figure Target subject.

3. the method according to claim 1, wherein described obtain the corresponding depth information packet of the target subject It includes:

Obtain depth map corresponding with the visible light figure；The depth map includes TOF depth map, binocular depth figure and structure light At least one of depth map；

Registration process, visible light figure and depth map after being registrated are carried out to the visible light figure and depth map；

Region according to where target subject described in the visible light figure determines the target master from the depth map after registration The corresponding depth information of body.

4. the method according to claim 1, wherein the step of return acquisition visible light figure, is to obtain The step of visible light figure of different acquisition angle includes:

Centered on the target subject, the target subject continuous transformation acquisition angles are surrounded；

There are under conditions of overlapping region between adjacent visible light figure, visible light figure is acquired in real time, the difference is obtained and adopts Collect the visible light figure of angle.

5. the method according to claim 1, wherein described corresponding according to the target subject and target subject Depth information carries out three-dimensionalreconstruction to the target subject, comprising:

Obtain corresponding first depth value of the first image plane vegetarian refreshments on the target subject；

It is flat that in the position of the target subject and first depth value described first is obtained according to the first image plane vegetarian refreshments Image surface vegetarian refreshments is in the corresponding first three-dimensional image vegetarian refreshments of three-dimensional space；

Obtain corresponding second depth value of the second image plane vegetarian refreshments on the target subject；

Using first depth value as reference depth value, determine the second three-dimensional image vegetarian refreshments relative to institute according to second depth value The relative position of the first three-dimensional image vegetarian refreshments is stated, the second three-dimensional image vegetarian refreshments is that the second image plane vegetarian refreshments is corresponding in three-dimensional space Three-dimensional image vegetarian refreshments；

Determine that described second is three-dimensional in the position of the target subject with the second image plane vegetarian refreshments depending on that relative position Pixel is in the corresponding position of the three-dimensional space；

Connect each three-dimensional image vegetarian refreshments in the three-dimensional space.

6. the method according to any one of claims 1 to 5, which is characterized in that described according to the target subject and mesh The step of marking the corresponding depth information of main body, carried out by three-dimensionalreconstruction, returns to the acquisition visible light figure for the target subject with Obtain different acquisition angle visible light figure, include: until obtaining the corresponding threedimensional model of the target subject

It determines the corresponding target type of target subject, the initial threedimensional model of same type is obtained according to the target type；

Obtain the corresponding actual depth value of crucial image plane vegetarian refreshments on the target subject；

Acquisition and the matched threedimensional model pixel of the crucial image plane vegetarian refreshments from the initial threedimensional model, according to each Actual depth value ratio between crucial image plane vegetarian refreshments adjusts the three-dimensional space position of matched threedimensional model pixel；

Obtain the corresponding actual depth value of non-key image plane vegetarian refreshments on the target subject；

It is obtained and the non-key matched threedimensional model pixel of image plane vegetarian refreshments from the initial threedimensional model；

According to the adjustment of actual depth value ratio and non-key plane between non-key image plane vegetarian refreshments and crucial image plane vegetarian refreshments The three-dimensional space position of the matched threedimensional model pixel of pixel, until each image plane vegetarian refreshments presence on target subject The threedimensional model pixel adjusted matched；

Each threedimensional model pixel adjusted forms the corresponding threedimensional model of target subject.

7. the method according to any one of claims 1 to 5, which is characterized in that the visible light figure acquires in real time, The method also includes:

According to the visible light figure acquired in real time, the corresponding threedimensional model of the target subject described in preview interface real-time display Construction process.

8. a kind of constructing devices of threedimensional model, which is characterized in that described device includes:

Processing module generates center weight figure corresponding with the visible light figure, wherein in described for obtaining visible light figure Weighted value represented by heart weight map is gradually reduced from center to edge；

Detection module obtains main body for the visible light figure and the center weight figure to be input in subject detection model Region confidence figure, wherein the subject detection model is the visible light figure previously according to Same Scene, center weight figure and right The model that the main body exposure mask figure marked answered is trained；

Target subject determining module, for determining the target master in the visible light figure according to the body region confidence level figure Body；

Threedimensional model building block, for obtaining the corresponding depth information of the target subject, according to the target subject and mesh The corresponding depth information of main body is marked, three-dimensionalreconstruction is carried out to the target subject, the processing module is returned and is adopted with obtaining difference The visible light figure for collecting angle, until obtaining the corresponding threedimensional model of the target subject.

9. a kind of terminal device, including memory and processor, computer program, the computer are stored in the memory When program is executed by the processor, so that the processor executes the step of the method as described in any one of claims 1 to 7 Suddenly.

10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program The step of method as described in any one of claims 1 to 7 is realized when being executed by processor.