CN107707831A - Image processing method and device, electronic device, and computer-readable storage medium - Google Patents

Abstract

The invention discloses an image processing method, an image processing device, an electronic device and a computer readable storage medium. The method comprises the following steps: acquiring a scene image of a current user; acquiring a depth image of a current user; processing the scene image and the depth image, and extracting a person region of a current user in the scene image to obtain a first person region image; judging whether the resolution of the first person region image is matched with the resolution of the first target background image; if not, compressing the first person region image and/or the first target background image to obtain a second person region image and a second target background image with matched resolutions; and fusing the second person region image and the second target background image to obtain a combined image. Therefore, in the merged image after the merging, the merging position of the person region and the background region is more natural, and the effect of the merged image is better.

Description

Image processing method and device, electronic device, and computer-readable storage medium

technical field

The present invention relates to the technical field of image processing, in particular to an image processing method and device, an electronic device, and a computer-readable storage medium.

Background technique

With the development of technology and the gradual changes in people's lifestyles, the way of group photos is not limited to the real scene where people are. People can fuse the characters with their favorite background images, so as to obtain a group photo image of the characters and the preset background.

Existing techniques for merging a person with a preset background usually use feature points to extract the outline of the person, and then fuse it with a preset background image. However, the accuracy of the character outline extracted using feature points is not high, especially the boundary of the character cannot be accurately marked, which affects the effect of image fusion, and the fused image is unnatural.

Contents of the invention

Embodiments of the present invention provide an image processing method, an image processing device, an electronic device, and a computer-readable storage medium.

The image processing method in the embodiment of the present invention is used in an electronic device, and the image processing method includes:

Get the scene image of the current user;

Obtain the depth image of the current user;

processing the scene image and the depth image, extracting the character area of the current user in the scene image to obtain a first character area image;

judging whether the resolution of the first person area image matches the resolution of the first target background image;

If not, compressing the first person area image and/or the first target background image to obtain a second person area image and a second target background image with matching resolutions;

The second person area image and the second target background image are fused to obtain a merged image.

An image processing device according to an embodiment of the present invention is used in an electronic device. The image processing device includes a visible light camera, a depth image acquisition component and a processor. The visible light camera is used to acquire the scene image of the current user. The depth image acquisition component is used to acquire the depth image of the current user. The processor is configured to process the scene image and the depth image, extract the character area of the current user in the scene image to obtain a first character area image, and determine the resolution of the first character area image and Whether the resolution of the first target background image matches; if not, performing compression processing on the first character area image and/or the first target background image to obtain a second character area image and a second character area image with matching resolutions Two target background images: merging the second person area image and the second target background image to obtain a merged image.

An electronic device according to an embodiment of the present invention includes one or more processors, memory and one or more programs. Wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs include instructions for executing the above image processing method.

The computer-readable storage medium according to the embodiment of the present invention includes a computer program used in combination with an electronic device capable of taking pictures, and the computer program can be executed by a processor to implement the above-mentioned image processing method.

The image processing method, image processing device, electronic device and computer-readable storage medium in the embodiments of the present invention extract the first person area in the scene image by acquiring the depth image of the current user. Since the acquisition of the depth image is not easily affected by factors such as illumination and color distribution in the scene, the first person region extracted through the depth image is more accurate, especially the boundary of the person region can be accurately calibrated. Further, the resolutions of the first target background image and/or the first character region image are respectively processed to obtain the second target background image and the second character region image with matched resolutions, and then fused, so that the fused merged In the image, the fusion of the character area and the background area is more natural, and the effect of merging images is better.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 2 is a block diagram of an image processing device in some embodiments of the present invention.

Fig. 3 is a schematic structural diagram of an electronic device according to some embodiments of the present invention.

Fig. 4 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 5 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 6(a) to Fig. 6(e) are schematic diagrams of scenes of structured light measurement according to an embodiment of the present invention.

Fig. 7(a) and Fig. 7(b) are schematic diagrams of a scene of structured light measurement according to an embodiment of the present invention.

Fig. 8 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 9 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 10 is a schematic flowchart of an image processing method in some embodiments of the present invention.

Fig. 11 is an example diagram of an image processing method in some embodiments of the present invention.

Fig. 12 is a block diagram of an image processing device in some embodiments of the present invention.

Fig. 13 is a block diagram of an electronic device according to some embodiments of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

Please refer to FIGS. 1 to 2 together. The image processing method according to the embodiment of the present invention is used in an electronic device 1000 . Image processing methods include:

01: Obtain the scene image of the current user;

02: Obtain the depth image of the current user;

03: Process the scene image and depth image, extract the character area of the current user in the scene image to obtain the first character area image;

04: Determine whether the resolution of the first person area image matches the resolution of the first target background image;

05: If not, perform compression processing on the first person area image and/or the first target background image to obtain a second person area image and a second target background image with matching resolutions;

06: Merging the second person area image and the second target background image to obtain a merged image.

Referring to FIG. 3 , the image processing method of the embodiment of the present invention can be implemented by the image processing apparatus 100 of the embodiment of the present invention. The image processing device 100 according to the embodiment of the present invention is used in an electronic device 1000 . The image processing device 100 includes a visible light camera 11 , a depth image acquisition component 12 and a processor 20 . Step 01 can be implemented by the visible light camera 11 , step 02 can be implemented by the depth image acquisition component 12 , and steps 03-06 can be implemented by the processor 20 .

That is to say, the visible light camera 11 can be used to acquire the scene image of the current user; the depth image acquisition component 12 can be used to acquire the depth image of the current user; the processor 20 can be used to process the scene image and the depth image, and extract the scene image of the current user character area to obtain the first character area image, determine whether the resolution of the first character area image matches the resolution of the first target background image, and if not, the first character area image and/or the first target background image Compression is performed to obtain the second person area image and the second target background image with matching resolutions, and the second person area image and the second target background image are fused to obtain a merged image.

Wherein, the scene image may be a grayscale image or a color image, and the depth image represents the depth information of each person or object in the current user's scene. The scene range of the scene image is consistent with the scene range of the depth image, and each pixel in the scene image can find depth information corresponding to the pixel in the depth image.

The image processing device 100 according to the embodiment of the present invention can be applied to the electronic device 1000 according to the embodiment of the present invention. That is to say, the electronic device 1000 according to the embodiment of the present invention includes the image processing device 100 according to the embodiment of the present invention.

In some embodiments, the electronic device 1000 includes a mobile phone, a tablet computer, a laptop computer, a smart bracelet, a smart watch, a smart helmet, smart glasses, and the like.

It is understandable that in the existing technologies for merging characters and virtual backgrounds, after fusing the extracted character region image with the preset background image, in the resulting merged image, the fusion of the character region and the background region tends to be stiff and uneven. Naturally, the visual effect of the merged image is poor. However, it is found through research that the above situation is usually caused by the difference in resolution between the image of the person area and the image of the background area.

Therefore, in the embodiment of the present invention, after obtaining the first person region image by extracting the character region of the current user in the scene image according to the acquired scene image and depth image of the current user, in order to improve the visual effect of the merged image, the second Before a person area image is fused with the first target background image, it may be determined whether the resolutions of the two images match.

If the resolutions of the first person area image and the first target background image match, the first person area image and the first target background image may be fused to obtain a merged image.

If the resolution of the first character area image does not match the resolution of the first target background image, the first character area image and/or the first target background image may be compressed to obtain a second character area image and a resolution matching The second target background image, and then fuse the second person area image and the second target background image to obtain a merged image.

Wherein, the first target background image can be selected by the processor 20, or can be selected by the current user according to needs, and the first target background image can be a dynamic image or a static image, which will not be described here. limit.

Specifically, a resolution difference range can be set in advance according to needs, and when the resolution difference between the first person area image and the first target background image is within the preset range, the first person area image can be determined Matches the resolution of the first target background image.

It should be noted that the processor 20 only performs compression processing on the first person area image to obtain the second person area image, at this time, the second target background image is the first target background image without compression processing; 20. Only compress the first target background image to obtain the second target background image. At this time, the second character area image is the uncompressed first character area image; the processor 20 simultaneously compresses the first target background image The image and the first person area image are compressed to obtain the second target background image and the second person area image respectively.

In addition, the merged image after fusion can be displayed on the display screen of the electronic device 1000 , and can also be printed by a printer connected to the electronic device 1000 .

In some application scenarios, for example, if the current user wants to hide the current background during the video process with others, at this time, the image processing method of the embodiment of the present invention can be used to convert the character area image corresponding to the current user and its resolution The matching target background image is fused, and then the fused merged image is displayed to the other party. Since the current user is having a video call with the other party, the visible light camera 11 needs to capture the scene image of the current user in real time, and the depth image collection component 12 also needs to collect the depth image corresponding to the current user in real time, and the processor 20 timely analyzes the scene collected in real time. The image and the depth image are processed so that the other party can see a smooth video picture composed of multiple frame merged images.

The existing methods for segmenting people and background are mainly based on the similarity and discontinuity of adjacent pixels in pixel values, but this segmentation method is easily affected by environmental factors such as external lighting. The image processing method, the image processing device 100 and the electronic device 1000 according to the embodiments of the present invention extract the first person area in the scene image by acquiring the depth image of the current user. Since the acquisition of the depth image is not easily affected by factors such as illumination and color distribution in the scene, the first person region extracted through the depth image is more accurate, especially the boundary of the person region can be accurately calibrated. Further, the resolutions of the first target background image and/or the first character region image are respectively processed to obtain the second target background image and the second character region image with matched resolutions, and then fused, so that the fused merged In the image, the fusion of the character area and the background area is more natural, and the effect of merging images is better.

Please refer to FIG. 4, in some embodiments, the step of obtaining the depth image of the current user in step 02 includes:

021: Project structured light to the current user;

022: Take a structured light image modulated by the current user; and

023: Demodulate the phase information corresponding to each pixel of the structured light image to obtain a depth image.

Please refer to FIG. 3 again. In some embodiments, the depth image acquisition component 12 includes a structured light projector 121 and a structured light camera 122 . Step 021 can be implemented by the structured light projector 121 , and steps 022 and 023 can be implemented by the structured light camera 122 .

That is to say, the structured light projector 121 can be used to transmit structured light to the current user; the structured light camera 122 can be used to capture the structured light image modulated by the current user, and demodulate the phase information corresponding to each pixel of the structured light image to obtain depth image.

Specifically, after the structured light projector 121 projects a certain pattern of structured light onto the current user's face and body, a structured light image modulated by the current user will be formed on the surface of the current user's face and body. The structured light camera 122 captures the modulated structured light image, and then demodulates the structured light image to obtain a depth image. Among them, the pattern of structured light can be laser stripes, gray codes, sinusoidal stripes, non-uniform speckle, etc.

Please refer to FIG. 5. In some implementations, the step 023 of demodulating the phase information corresponding to each pixel of the structured light image to obtain the depth image includes:

0231: Demodulate the phase information corresponding to each pixel in the structured light image;

0231: Convert phase information to depth information; and

0233: Generate a depth image based on depth information.

Please refer to FIG. 2 again. In some implementation manners, Step 0231 , Step 0232 and Step 0233 can all be implemented by the structured light camera 122 .

That is to say, the structured light camera 122 can be further used to demodulate the phase information corresponding to each pixel in the structured light image, convert the phase information into depth information, and generate a depth image according to the depth information.

Specifically, compared with unmodulated structured light, the phase information of the modulated structured light has changed, and the structured light presented in the structured light image is the structured light after distortion, wherein the changed phase information It can represent the depth information of the object. Therefore, the structured light camera 122 first demodulates the phase information corresponding to each pixel in the structured light image, and then calculates the depth information according to the phase information, thereby obtaining the final depth image.

In order to make those skilled in the art more clearly understand the process of collecting the depth image of the current user's face and body according to the structure, the following uses a widely used grating projection technology (stripe projection technology) as an example to illustrate its specific principles. Among them, grating projection technology belongs to surface structured light in a broad sense.

As shown in Figure 6(a), when using surface structured light projection, firstly generate sinusoidal fringes through computer programming, project the sinusoidal fringes to the measured object through the structured light projector 121, and then use the structured light camera 122 to capture the fringes The degree of curvature modulated by the object is then demodulated to obtain the phase of the curved fringe, and then the phase is converted into depth information to obtain a depth image. In order to avoid the problem of error or error coupling, it is necessary to calibrate the parameters of the depth image acquisition component 12 before using structured light for depth information acquisition. The calibration includes geometric parameters (for example, the relative relationship between the structured light camera 122 and the structured light projector 121). position parameters, etc.), the internal parameters of the structured light camera 122 and the internal parameters of the structured light projector 121, etc.

Specifically, in the first step, the computer is programmed to generate sinusoidal fringes. Since the distorted fringes need to be used to obtain the phase later, for example, the four-step phase shift method is used to obtain the phase, so the four phase differences generated here are stripes, then the structured light projector 121 projects the four stripes onto the measured object (the mask shown in FIG. To read the fringes of the reference surface shown on the right side of Fig. 6(b).

The second step is to perform phase recovery. The structured light camera 122 calculates the modulated phase according to the collected four modulated fringe patterns (ie structured light images), and the obtained phase pattern at this time is a truncated phase pattern. Because the result obtained by the four-step phase-shift algorithm is calculated by the arctangent function, the modulated phase of the structured light is limited to [-π, π], that is, whenever the modulated phase exceeds [-π ,π], which will start all over again. The resulting main value of the phase is shown in Fig. 6(c).

Wherein, during the phase recovery process, transition elimination processing is required, that is, the truncated phase is restored to a continuous phase. As shown in Figure 6(d), the modulated continuous phase map is on the left, and the reference continuous phase map is on the right.

The third step is to subtract the modulated continuous phase from the reference continuous phase to obtain the phase difference (that is, phase information), which represents the depth information of the measured object relative to the reference plane, and then substitute the phase difference into the conversion of phase and depth formula (the parameters involved in the formula have been calibrated), the three-dimensional model of the object to be measured can be obtained as shown in Figure 6(e).

It should be understood that, in practical applications, according to different specific application scenarios, the structured light used in the embodiments of the present invention may be other arbitrary patterns besides the above-mentioned grating.

As a possible implementation manner, the present invention may also use speckle structured light to collect depth information of the current user.

Specifically, the method for obtaining depth information by speckle structured light is to use a substantially flat diffraction element, which has a relief diffraction structure with a specific phase distribution, and a stepped relief structure with two or more concavo-convex cross sections. The thickness of the substrate in the diffraction element is approximately 1 micron, and the height of each step is not uniform, and the height can range from 0.7 micron to 0.9 micron. The structure shown in Fig. 7(a) is the partial diffraction structure of the collimating beam splitting element of this embodiment. Fig. 7(b) is a cross-sectional side view along section A-A, and the units of the abscissa and ordinate are both micrometers. The speckle pattern generated by speckle structured light is highly random, and the pattern will change with the distance. Therefore, before using speckle structured light to obtain depth information, it is first necessary to calibrate the speckle pattern in space, for example, within the range of 0 to 4 meters from the structured light camera 122, take a reference plane every 1 cm, After the calibration is completed, 400 speckle images are saved. The smaller the calibration interval, the higher the accuracy of the obtained depth information. Subsequently, the structured light projector 121 projects the speckle structured light onto the object under test (that is, the current user), and the height difference of the surface of the object under test changes the speckle pattern of the speckle structured light projected on the object under test. After the structured light camera 122 shoots the speckle pattern projected on the object to be measured (that is, the structured light image), the cross-correlation operation is performed on the speckle pattern and the 400 speckle images saved after previous calibration, and then 400 correlation images are obtained. degree image. The position of the measured object in the space will show a peak on the correlation image, and the depth information of the measured object can be obtained by superimposing the above peaks and interpolating.

Since the ordinary diffraction element diffracts the light beam to obtain most of the diffracted light, but the intensity of each diffracted light varies greatly, and the risk of damage to human eyes is also large. Even if the diffracted light is diffracted twice, the uniformity of the obtained beam is low. Therefore, the projection effect of the light beam diffracted by the common diffraction element on the measured object is relatively poor. In this embodiment, a collimating beam-splitting element is used, which not only has the function of collimating the uncollimated beam, but also has the function of splitting light, that is, the uncollimated light reflected by the mirror passes through the collimating beam-splitting element and then Multiple collimated beams are emitted from different angles, and the cross-sectional areas of the emitted multiple collimated beams are approximately equal, and the energy flux is approximately equal, so that the projection effect of the scattered light after the diffraction of the beam is better. At the same time, the emitted laser light is dispersed to each beam, which further reduces the risk of damage to human eyes. Compared with other evenly arranged structured light, when the speckle structured light achieves the same collection effect, the speckle structured light consumes less lower power.

Please refer to FIG. 8. In some implementations, step 03 processes the scene image and the depth image, and the step of extracting the character area of the current user in the scene image to obtain the first character area image includes:

031: Identify the face area in the scene image;

032: Obtain the depth information corresponding to the face area from the depth image;

033: Determine the depth range of the character area according to the depth information of the face area; and

034: Determine, according to the depth range of the person region, a person region that is connected to the face region and falls within the depth range to obtain a first person region image.

Please refer to FIG. 2 again, in some implementation manners, step 031 , step 032 , step 033 and step 034 can all be implemented by the processor 20 .

That is to say, the processor 20 can be further used to identify the face area in the scene image, obtain the depth information corresponding to the face area from the depth image, determine the depth range of the person area according to the depth information of the face area, and A character area connected to the face area and falling within the depth range is determined according to the depth range of the character area to obtain a first character area image.

Specifically, firstly, the trained deep learning model can be used to identify the face area in the scene image, and then the depth information of the face area can be determined according to the corresponding relationship between the scene image and the depth image. Since the face area includes features such as nose, eyes, ears, lips, etc., the depth data corresponding to each feature in the face area in the depth image is different, for example, when the face is facing the depth image acquisition component 12 In the depth image captured by the depth image acquisition component 12, the depth data corresponding to the nose may be relatively small, while the depth data corresponding to the ear may be relatively large. Therefore, the above-mentioned depth information of the face area may be a value or a range of values. Wherein, when the depth information of the face area is a value, the value may be obtained by taking an average value of the depth data of the face area; or, may be obtained by taking a median value of the depth data of the face area.

Since the character area includes the face area, that is to say, the character area and the face area are in a certain depth range, therefore, after the processor 20 determines the depth information of the face area, it can A depth range of the person area is set, and a person area falling within the depth range and connected to the face area is extracted according to the depth range of the person area to obtain a first person area image.

In this way, the first person region image can be extracted from the scene image according to the depth information. Since the acquisition of depth information is not affected by factors such as illumination and color temperature in the environment, the extracted image of the first person area is more accurate.

Referring to Fig. 9, in some embodiments, the image processing method also includes the following steps:

07: processing the scene image to obtain a full-field edge image of the scene image; and

08: Correct the image of the first person area according to the edge image of the whole field.

Please refer to FIG. 2 again. In some implementation manners, both step 07 and step 08 may be implemented by the processor 20 .

That is to say, the processor 20 is further configured to process the scene image to obtain a full-field edge image of the scene image, and correct the first person area image according to the full-field edge image.

The processor 20 first performs edge extraction on the scene image to obtain a full-field edge image, wherein the edge lines in the full-field edge image include edge lines of the current user and background objects in the scene where the current user is located. Specifically, edge extraction can be performed on the scene image through the Canny operator. The core of the algorithm for edge extraction by Canny operator mainly includes the following steps: first, the scene image is convolved with a 2D Gaussian filter template to eliminate noise; then, the gradient value of the gray level of each pixel is obtained by using the differential operator, and Calculate the gradient direction of the gray level of each pixel according to the gradient value, through the gradient direction, you can find the adjacent pixels of the corresponding pixel along the gradient direction; then, traverse each pixel, if the gray value of a pixel is the same as the two before and after the gradient direction If the gray value of the adjacent pixel is not the largest, then this pixel is considered not to be an edge point. In this way, the pixel points at the edge position in the scene image can be determined, so as to obtain the edge image of the whole field after edge extraction.

After the processor 20 acquires the edge image of the whole field, it corrects the image of the first person region according to the edge image of the whole field. It can be understood that the image of the first person area is obtained by merging all the pixels connected to the face area in the scene image and falling into the set depth range. In some scenes, there may be some Objects that are connected and fall within the depth range. Therefore, in order to make the extracted first person region image more accurate, the full-field edge map may be used to correct the first person region image.

Further, the processor 20 may also perform secondary correction on the corrected first person region image, for example, may perform expansion processing on the corrected first person region image, and expand the first person region image to retain the first person region The edge detail of the image.

Please refer to FIG. 10 , in some embodiments, step 05 compresses the first person area image and/or the first target background image includes:

051: If the resolution of the first person area image is greater than the resolution of the first target background image, compress the first person area image to obtain a second person area image, wherein the resolution of the second person area image is the same as the first target background image Resolution matching of a target background image;

052: If the resolution of the first person area image is smaller than the resolution of the first target background image, compress the first target background image to obtain a second target background image, wherein the resolution of the second target background image is the same as that of the first target background image A resolution matching of the person area image.

That is to say, when the resolution of the first person area image does not match the resolution of the first target background image, the image with higher resolution can be compressed so as to reduce the resolution of the image with high resolution, so that the processing The resulting person area image has the same resolution as the target background image.

Specifically, if the resolution of the first person region image is relatively high among the first person region image and the first target background image, the first person region image can be compressed to reduce the resolution of the first person region image. , so as to obtain the second character region image matching the resolution of the first target background image; if the resolution of the first target background image is higher among the first character region image and the first target background image, then the first The target background image is compressed to reduce the resolution of the first target background image, so as to obtain a second target background image matching the resolution of the first person area image.

Specifically, compression processing may be performed on the first person area image and/or the first target background image in various ways.

For example, down-sampling can be used to generate a second character area image and/or a second target by sampling every s points in each row and column from the first target background image and/or the first character area image. background image.

For example, assuming that the resolution difference between the person area image and the target background image is preset to be less than 50, it is determined that the person area image matches the target background image. If the resolution of the first person area image is 800*600 pixels per inch (Pixels Per Inch, PPI for short), and the resolution of the first target background area is 400*300PPI, then the image of the first person area can be downsized by s times. Sampling, that is, sampling every s points in each row and column, to obtain a second person area image with a resolution of N=(800/s)*(600/s). Wherein, N is a value greater than 400*300-50 and less than 400*300+50, so that the generated second person area image matches the resolution of the first target background image.

Alternatively, in the first target background image and/or the first person area image, adjacent s*s pixels can be regarded as a pixel, and the value of this pixel is the mean value of the s*s pixels, thereby generating the first A second target background image and/or a second person area image.

It should be noted that, in order to avoid deformation of the processed image, when compressing the first person area image or the first target background image, the same ratio can be compressed in the x direction and the y direction, so that the image can be compressed in the x direction. direction and y direction to achieve proportional reduction. For example, when the resolution of the first target background image is 800×600PPI, if the number of pixels in the x direction of the first target background image is reduced to 1/2 by compression, in order to avoid the appearance of the second target background image Deformation can also reduce the number of pixels in the y direction of the first target background image to 1/2 of the original through proportional compression, that is, the resolution of the obtained second target background image can be 400×300PPI.

In this way, by performing compression processing on the first character region image and/or the first target background image to obtain a second character region image and a second target background image with matching resolution, and then compressing the second character region image with matching resolution By merging with the second target background image, the fusion of the character area and the background area in the fused merged image is more natural, and the visual effect of the merged image is optimized.

As a possible implementation manner, when the resolution of the first person area image does not match the resolution of the first target background image, an interpolation process may be performed on the first person area image and/or the first target background image to obtain Obtain a second person area image and a second target background image with matching resolutions.

Specifically, if the resolution of the first person area image is greater than the resolution of the first target background image, the first target background image is interpolated to obtain the second target background image, wherein the resolution of the second target background image is the same as matching the resolution of the image of the first person area;

If the resolution of the first character area image is smaller than the resolution of the first target background image, interpolation processing is performed on the first character area image to obtain a second character area image, wherein the resolution of the second character area image is the same as that of the first target background image. The resolution of the background image matches.

That is to say, when the resolution of the first person area image does not match the resolution of the first target background image, the image with the lower resolution of the two images can be interpolated to create pixels and use known pixels The gray value of the unknown pixel is generated to increase the resolution of the low-resolution image, so that the processed character area image has the same resolution as the target background image.

Specifically, if the resolution of the first target background image is relatively low among the first person area image and the first target background image, interpolation processing may be performed on the first target background image to increase the resolution of the first target background image , so as to obtain the second target background image matching the resolution of the first person region image. If the resolution of the first person region image is relatively low among the first person region image and the first target background image, the first person region image can be interpolated to increase the resolution of the first person region image, thereby obtaining A second person area image that matches the resolution of the first target background image.

In some implementations, if the resolutions of the first person area image and the first target background image are different but relatively low, in order to improve the clarity of the merged image, the first person area image and the first target background image may also be processed simultaneously. The target background image is interpolated with different proportions to obtain the second person area image and the second target background image respectively. Wherein, the resolution of the second person area image matches the second target background image.

Specifically, interpolation processing may be performed on the first target background image and/or the first person region image by nearest pixel interpolation algorithm, bilinear interpolation algorithm, bicubic interpolation algorithm, fractal algorithm and other interpolation processing methods.

In the following, when the resolution of the first object background image is smaller than the resolution of the first person area image, the bilinear interpolation algorithm is used to perform interpolation processing on the first object background image as an example, and the resolution of the acquisition and the first person area image The process of rate matching the second target background image is described.

Specifically, in the second target background image, the value of the newly created unknown pixel can be obtained from the values of 4 adjacent pixels in the 2*2 area near the unknown pixel in the first target background image through a weighted average algorithm .

Please refer to Figure 11, assuming that the values of the four known pixels in the 2*2 area near the unknown pixel P=(x,y) are respectively calculated by the functions f(Q ₁₂ ), f(Q ₂₂ ), f(Q ₁₁ ), f (Q ₂₁ ) means, wherein, Q ₁₂ =(x ₁ ,y ₂ ), Q ₂₂ =(x ₂ ,y ₂ ), Q ₁₁ =(x ₁ ,y ₁ ), Q ₂₁ =(x ₂ ,y ₁ ) are respectively 4 pixels in the 2*2 area near the unknown pixel P=(x,y), and the value of the unknown function f at the point P=(x,y) should be obtained.

First, R1 and R2 can be obtained by performing linear interpolation in the x direction in the following manner.

Wherein, R ₁ =(x,y ₁ );

Wherein, R ₂ =(x, y ₂ ).

Then, perform linear interpolation in the y direction in the following manner to obtain P.

Through the above method, the value of the unknown pixel created in the second target background image can be obtained according to the value of the known pixel in the first target background image, so as to realize the interpolation process based on the generation of the first target background image and the first target background image. The resolution of the person area image matches the second target background image.

It should be noted that, in order to avoid deformation of the processed image, when performing interpolation processing on the first person area image and/or the first target background image, equal-scale interpolation can be performed in the x direction and the y direction, so that the image Achieving proportional magnification in the x direction and y direction. For example, when the resolution of the first target background image is 400×300PPI, if the number of pixels in the x direction of the first target background image is doubled by interpolation, in order to avoid deformation of the second target background image , the number of pixels in the y-direction of the first target background image can also be doubled by equal proportion interpolation, that is, the resolution of the obtained second target background image can be 800×600PPI.

In this way, by performing interpolation processing on the first character area image and/or the first target background image to obtain the second character area image and the second target background image with matching resolution, and then the second character area image with matching resolution By merging with the second target background image, the fusion of the character area and the background area in the fused merged image is more natural, and the visual effect of the merged image is optimized.

In addition, in the first person area image and the first target background image, the high-resolution image can also be compressed, and the low-resolution image can be interpolated to obtain the second person area image and the second target background image with matching resolutions. Two target background images.

It should be noted that, in the embodiment of the present invention, the boundary of the second person area image can also be feathered and then merged with the second target background image, so that the edge of the person area image is in the second target background image. The transitions in the image are smoother and more natural, and the visual effect of the merged image is better.

In some implementations, when the first target background image is an image selected by the processor 20, in order to avoid the excessive processing load of the processor 20, the resolution of the first person area image can be selected from a plurality of background images. Images with a smaller rate difference, so as to reduce the processing pressure of the processor 20. That is, before step 04, the image processing method may also include the following steps:

09: Acquire a first target background image from multiple background images according to the resolution of the first person region image, wherein the difference between the resolution of the first target background image and the resolution of the first person region image is smaller than a threshold.

Wherein, the threshold can be set as required. For example, it may be determined according to the processing capability or processing speed of the processor 20 . If the processing speed of the processor 20 is faster, the threshold can be set to a larger value; if the processing speed of the processor 20 is slower, the threshold can be set to a smaller value, and so on.

The plurality of background images may be dynamic images or static images, and may be stored in the electronic device 1000 or obtained through network resources, which is not limited here.

Specifically, after the processor 20 obtains the first person area image, it may determine the resolution of the first person area image, then acquire a plurality of background images, determine the resolution of the background images, and then determine the resolution of the plurality of background images respectively. Whether the difference between the rate and the resolution of the first person area image is smaller than the threshold, so that the background image whose resolution difference with the first person area image is smaller than the threshold is selected as the first target background image.

If the background image whose resolution difference with the first character area image is smaller than the threshold is a plurality of images, the image with the smallest resolution difference with the first character area image can be selected from the plurality of images as the first target background image.

By selecting an image whose resolution difference with the first person area image is smaller than a threshold from the plurality of background images as the first target background image, the processing pressure of the processor 20 can be reduced and the image processing speed can be increased.

Please refer to FIG. 3 and FIG. 12 together. Embodiments of the present invention also provide an electronic device 1000 . The electronic device 1000 includes an image processing device 100 . The image processing device 100 can be realized by hardware and/or software. The image processing apparatus 100 includes an imaging device 10 and a processor 20 .

The imaging device 10 includes a visible light camera 11 and a depth image acquisition component 12 .

Specifically, the visible light camera 11 includes an image sensor 111 and a lens 112. The visible light camera 11 can be used to capture the color information of the current user to obtain a scene image, wherein the image sensor 111 includes a color filter array (such as a Bayer filter array), and the lens 112 The number of can be one or more. When the visible light camera 11 acquires the scene image, each imaging pixel in the image sensor 111 senses the light intensity and wavelength information from the shooting scene to generate a set of raw image data; the image sensor 111 sends the set of raw image data to the processing In the processor 20, the processor 20 performs operations such as denoising and interpolation on the original image data to obtain a color scene image. The processor 20 can process each image pixel in the raw image data one by one in a variety of formats, for example, each image pixel can have a bit depth of 8, 10, 12 or 14 bits, and the processor 20 can use the same or different Bit depth is processed for each image pixel.

The depth image acquisition component 12 includes a structured light projector 121 and a structured light camera 122 , and the depth image acquisition component 12 can be used to capture depth information of a current user to obtain a depth image. The structured light projector 121 is used to project the structured light to the current user, wherein the structured light pattern may be laser stripes, gray codes, sinusoidal stripes or randomly arranged speckle patterns and the like. The structured light camera 122 includes an image sensor 1221 and a lens 1222, and the number of the lens 1222 may be one or more. The image sensor 1221 is used to capture the structured light image projected by the structured light projector 121 onto the current user. The structured light image can be sent by the depth acquisition component 12 to the processor 20 for processing such as demodulation, phase recovery, and phase information calculation to obtain the depth information of the current user.

In some embodiments, the functions of the visible light camera 11 and the structured light camera 122 can be implemented by one camera, that is to say, the imaging device 10 only includes one camera and one structured light projector 121, and the above camera can not only capture scene images, Structured light images can also be taken.

In addition to using structured light to obtain depth images, depth images of the current user can also be obtained through binocular vision methods and depth image acquisition methods based on Time of Flight (TOF).

The processor 20 is further configured to judge whether the resolutions of the first character area image extracted from the scene image and the depth image match the resolution of the first object background image, and if they do not match, make a corresponding adjustment to the first character area image and/or the first object Compression processing is performed on the background image, so as to fuse the acquired second person region image with matching resolution and the second target background image. When extracting the image of the first person area, the processor 20 may combine the depth information in the depth image to extract a two-dimensional image of the first person area from the scene image, or establish a three-dimensional image of the person area according to the depth information in the depth image. , combined with the color information in the scene image to perform color filling on the three-dimensional character area to obtain a three-dimensional color image of the first character area. Further, the obtained second person area image may be two-dimensional or three-dimensional in color. Therefore, when fusing the second person area image and the second target background image, it may be to fuse the two-dimensional second person area image and the second target background image to obtain a merged image, or to combine the three-dimensional color second The person area image is fused with the second target background image to obtain a merged image.

Furthermore, the image processing device 100 also includes an image memory 30 . The image memory 30 may be embedded in the electronic device 1000, or may be a memory independent of the electronic device 1000, and may include a direct memory access (DirectMemory Access, DMA) feature. The original image data collected by the visible light camera 11 or the data related to the structured light image collected by the depth image collection component 12 can be transmitted to the image memory 30 for storage or buffering. The processor 20 can read the original image data from the image memory 30 for processing to obtain a scene image, and can also read data related to the structured light image from the image memory 30 for processing to obtain a depth image. In addition, the scene image and the depth image can also be stored in the image memory 30 for processing by the processor 20 at any time. The person area image and/or the first object background image are processed to obtain a second person area image and a second object background image with matching resolutions, and then fusion processing is performed to obtain a merged image. Wherein, the first object background image, the second object background image and the merged image can also be stored in the image memory 30 .

The image processing device 100 may further include a display 50 . The display 50 can directly obtain the merged image from the processor 20 , and can also obtain the merged image from the image memory 30 . The display 50 displays the merged image for the user to watch, or is further processed by a graphics engine or a graphics processing unit (Graphics Processing Unit, GPU). The image processing device 100 also includes an encoder/decoder 60, the encoder/decoder 60 can encode and decode image data such as scene images, depth images, and merged images, and the encoded image data can be stored in the image memory 30, and can be The image is decompressed by the decoder for display before it is displayed on the display 50 . The encoder/decoder 60 may be implemented by a central processing unit (Central Processing Unit, CPU), a GPU or a coprocessor. In other words, the encoder/decoder 60 may be any one or more of a central processing unit (Central Processing Unit, CPU), a GPU, and a coprocessor.

The image processing device 100 also includes a control logic 40 . When the imaging device 10 is imaging, the processor 20 analyzes the data acquired by the imaging device to determine image statistical information of one or more control parameters (eg, exposure time, etc.) of the imaging device 10 . The processor 20 sends the image statistical information to the control logic 40, and the control logic 40 controls the imaging device 10 to perform imaging with the determined control parameters. Control logic 40 may include a processor and/or a microcontroller executing one or more routines (eg, firmware). One or more routines may determine control parameters of imaging device 10 based on received image statistics.

Referring to FIG. 13 , an electronic device 1000 according to an embodiment of the present invention includes one or more processors 200 , a memory 300 and one or more programs 310 . One or more programs 310 are stored in the memory 300 and configured to be executed by the one or more processors 200 . The program 310 includes instructions for executing the image processing method of any one of the above-mentioned embodiments.

For example, the program 310 includes instructions for performing the image processing method described in the following steps:

01: Obtain the scene image of the current user;

02: Obtain the depth image of the current user;

03: Process the scene image and depth image, extract the character area of the current user in the scene image to obtain the first character area image;

04: Determine whether the resolution of the first person area image matches the resolution of the first target background image;

05: If not, perform compression processing on the first person area image and/or the first target background image to obtain a second person area image and a second target background image with matching resolutions;

06: Merging the second person area image and the second target background image to obtain a merged image.

For another example, the program 310 also includes instructions for executing the image processing method described in the following steps:

0231: Demodulate the phase information corresponding to each pixel in the structured light image;

0231: Convert phase information to depth information; and

0233: Generate a depth image based on depth information.

A computer-readable storage medium according to an embodiment of the present invention includes a computer program used in conjunction with the electronic device 1000 capable of imaging. The computer program can be executed by the processor 200 to complete the image processing method in any one of the above implementation manners.

For example, the computer program can be executed by the processor 200 to complete the image processing method described in the following steps:

01: Obtain the scene image of the current user;

02: Obtain the depth image of the current user;

03: Process the scene image and depth image, extract the character area of the current user in the scene image to obtain the first character area image;

04: Determine whether the resolution of the first person area image matches the resolution of the first target background image;

05: If not, perform compression processing on the first person area image and/or the first target background image to obtain a second person area image and a second target background image with matching resolutions;

06: Merging the second person area image and the second target background image to obtain a merged image.

For another example, the computer program can also be executed by the processor 200 to complete the image processing method described in the following steps:

0231: Demodulate the phase information corresponding to each pixel in the structured light image;

0231: Convert phase information to depth information; and

0233: Generate a depth image based on depth information.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless specifically defined otherwise.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which shall It is understood by those skilled in the art to which the embodiments of the present invention pertain.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment used. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. The program is processed electronically and stored in computer memory.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, Programmable Gate Arrays (PGAs), Field Programmable Gate Arrays (FPGAs), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. During execution, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (10)

1. An image processing method, characterized in that, comprising: Get the scene image of the current user; Obtain the depth image of the current user; processing the scene image and the depth image, extracting the character area of the current user in the scene image to obtain a first character area image; judging whether the resolution of the first person area image matches the resolution of the first target background image; If not, compressing the first person area image and/or the first target background image to obtain a second person area image and a second target background image with matching resolutions; The second person area image and the second target background image are fused to obtain a merged image. 2. The method according to claim 1, wherein the acquiring the depth image of the current user comprises: projecting structured light to the current user; taking a structured light image modulated by the current user; and The phase information corresponding to each pixel of the structured light image is demodulated to obtain the depth image. 3. The method according to claim 2, wherein said demodulating phase information corresponding to each pixel of said structured light image to obtain said depth image comprises: Demodulating phase information corresponding to each pixel in the structured light image; converting said phase information into depth information; and The depth image is generated according to the depth information. 4. The method according to claim 1, wherein the extracting the character area of the current user in the scene image to obtain the first character area image comprises: identifying a face area in the scene image; Obtaining depth information corresponding to the face area from the depth image; determining the depth range of the person area according to the depth information of the face area; and Determining a character area connected to the face area and falling within the depth range according to the depth range of the character area to obtain the first character area image. 5. image processing method according to claim 4, is characterized in that, described image processing method also comprises: processing the scene image to obtain a full-field edge image of the scene image; and Correcting the first person area image according to the full-field edge image. 6. The method according to any one of claims 1-5, wherein before the judging whether the resolution of the first person area image matches the resolution of the first target background image, further comprising: Acquire the first target background image from a plurality of background images according to the resolution of the first person area image, wherein the resolution of the first target background image is different from the resolution of the first person area image The difference is less than the threshold. 7. The method according to any one of claims 1-5, wherein the compressing the first person area image and/or the first target background image comprises: If the resolution of the first person area image is greater than the resolution of the first target background image, compress the first person area image to obtain a second person area image, wherein the second person area image a resolution matching that of the first target background image; If the resolution of the first person area image is smaller than the resolution of the first target background image, compress the first target background image to obtain a second target background image, wherein the second target background image The resolution matches the resolution of the first person area image. 8. An image processing device, comprising: Visible light camera, the visible light camera is used to obtain the scene image of the current user; A depth image acquisition component, the depth image acquisition component is used to acquire the depth image of the current user; and a processor for: processing the scene image and the depth image, extracting the character area of the current user in the scene image to obtain a first character area image; judging whether the resolution of the first person area image matches the resolution of the first target background image; If not, compressing the first person area image and/or the first target background image to obtain a second person area image and a second target background image with matching resolutions; The second person area image and the second target background image are fused to obtain a merged image. 9. An electronic device, characterized in that the electronic device comprises: one or more processors; memory; and One or more programs, wherein said one or more programs are stored in said memory and configured to be executed by said one or more processors, said programs comprising means for performing the An instruction of any one of the image processing methods. 10. A computer-readable storage medium, characterized in that it comprises a computer program used in combination with an electronic device capable of taking pictures, and the computer program can be executed by a processor to complete the method described in any one of claims 1-7. image processing method.