KR20170092533A - A face pose rectification method and apparatus - Google Patents

Abstract

A pause correction method for correcting a pose in data representing face 100 images includes: obtaining at least one test frame comprising A-2D near-infrared image data, 2D visible light image data, and a depth map step; C-estimating a pose of a face in the test frame by aligning the depth map according to a 3D model of the head of a known direction; Mapping at least one of the 2D images onto a depth map to generate D-textured image data; And projecting the textured image data in 2D to produce data representing an E-Pose calibrated 2D projected image.

Description

Technical Field [0001] The present invention relates to a facial pose correcting method and apparatus,

The present invention relates to a face pose correction method and apparatus.

Face recognition involves the analysis of data representing a test image of a human face (or a set of test images) and its comparison with a database of reference images. The test image is typically a 2D image captured with a common 2D camera, while the reference image is, for example, a 2D image, or sometimes a 3D image, captured with a depth camera.

Previous approaches for face recognition generally assume that the test image and the reference images are all captured in a complete front view. Accordingly, various performance algorithms have been developed to classify front test images and match them with corresponding front reference images.

The credibility of recognition, however, decreases rapidly when the test image is captured from a non-frontal viewpoint. To solve this problem, it has already been proposed in the prior art to correct the pose, i.e. warp the test image to produce a composite frontal test image from the non-frontal test image.

Thus, robust methods for estimating head pose and then correcting are essential in many applications such as face recognition, or more generally face classification and face processing.

US8055028 relates to a method of normalizing a 2D facial image to a frontal facial image rather than a frontal view. The method includes: determining a pose of an image that is not a front of the object; Performing a smoothing transformation on the non-frontal image of the object, thereby creating a smoothed object image; And synthesizing the frontal image of the object by using the pose determination result and the smoothed object image. The pose is determined by obtaining a first average distance between object feature points that are present on both sides of the center line of the image rather than the front.

US 8199979 discloses a method for classifying and storing images comprising face regions obtained with a 2D image acquisition device. The face detection module identifies a group of pixels corresponding to the face area. The face orientation and pose are then determined. In the case of a half-profile face, the normalization module converts the candidate face area to 3D space, and then rotates it until an appropriate pose correction is made. Alternatively, the texture, color, and feature areas of the 2D face region are then mapped onto the rotated 3D model to correct the pose. This, however, causes a serious deformation when 2D images are mapped onto 3D models of different faces.

US7929775 discloses a method for matching a 2D image with a 3D class model. The method includes identifying image features in a 2D image; Computing an alignment transformation between the class model and the image; And comparing the class portions of the class model with image features under a sorting variant.

US 7289648 discloses a method for automatically modeling a three-dimensional object, such as a face, from a single image. The system and method according to the present invention construct one or more three-dimensional (3D) face models using a single image. It can also be used as a tool to create a database of faces with various poses needed to train most face recognition systems.

US2013156262 proposes an estimate of the pose of an object by defining a set of pairs of features as pairs of geometric primitives, and the geometric components include directed surface points, directed boundary points, and boundary line segments do. The model pair features are determined based on the set of pairs of features for the model of the object. The scene pair features are determined based on a set of pairs of features from the data obtained by the 3D sensor, and then the model pair features are matched with the scene pair features to estimate the object's pose.

Pose correction from a 2D image is, however, a difficult task and the quality of the correction is generally not robust because it depends on the amplitude correction to be achieved with respect to occlusions, illumination,

It has already been proposed to use additional depth information provided by depth sensors, which are becoming increasingly available and available, to overcome some of the inherent problems of head estimation poses based on 2D data.

US 8660306 discloses a method for correcting human pauses determined from depth data. The method includes receiving depth image data, obtaining an initial estimated skeleton of the associated object from the depth image data, using a random forest subspace regression function (utilizing a plurality of random segmentation / To the initially estimated skeleton, and determining a representation of the pose based on the result of applying the random forest subspace regression to the initially estimated skeleton. The method is particularly adapted to the measurement of the pose of the entire body.

US 6 383 464 discloses a system for generating face images, indexing the images by composite codes, and searching similar two-dimensional face images. 3D images of human faces are generated from a data store of 3D face feature surface features. These shapes are organized by facial feature portions. By assembling the shapes for each face portion, a 3D face image is formed.

US 8406484 relates to a face recognition apparatus, which comprises: a two-dimensional information obtaining unit for obtaining two-dimensional image information of an object; A three-dimensional information obtaining unit for obtaining three-dimensional image information of an object; A user information database for storing the three-dimensional face information of the user and the elliptic model corresponding to the two-dimensional face information of the user; Dimensional image information of the object, determines whether or not the recognized face is the face of the user, matches the elliptic model of the user with the three-dimensional image information, and recognizes the face of the user , And includes a control unit for determining whether or not the face of the user is improperly used based on the error.

US 7756325 discloses an algorithm for estimating the 3D shape of a three-dimensional object, such as a human face, based on information retrieved from a single photograph. In addition to pixel intensity, the present invention utilizes a variety of image features in a multi-features fitting algorithm (MFF).

EP1039417 relates to a method of processing an image of a three-dimensional object, the method comprising the steps of providing a deformable object model obtained from a plurality of three-dimensional images, matching the deformable object model with at least one 2D object image And providing the matched deformable object model as a 3D representation of the object.

US8553973 relates to other methods and systems for modeling 3D objects (e.g., human faces). More and more images of faces will include a depth map, due to the appearance of various consumer devices such as game consoles, laptops, tablets, smart phones, and depth sensors in, for example, automobiles. A popular example of a depth sensor that generates RGB-D data sets is the Kinect input device proposed by Microsoft for the Xbox 360, Xbox One and Windows PCs (all trademarks of Microsoft Corporation). The depth sensors include an indication of the depth (or distance to the light source) for each pixel of the image, but a 2.5D image that does not contain any indication of hidden or occluded elements, such as, for example, Data sets are generated.

It would be desirable to provide new face recognition methods in which test images are captured with depth sensors and then compared to existing 2D images. This method will have the advantage of using modern depth sensors for acquiring RGB-D image data and comparing them to widely available 2D reference images.

It would also be desirable to provide new methods using the capabilities of depth sensors and RGB-D data sets to improve the task of pose calibration.

It would also be desirable to provide a new method for evaluating head pose, which is faster than conventional methods.

It would also be desirable to provide a new method for evaluating head pose, which is more accurate than existing methods.

It would also be desirable to provide a new method for evaluating head pose, which can handle wide pose deformations.

It would also be desirable to provide a new method for evaluating head pose, which can handle wide pose deformations.

It is therefore an object of the present invention to propose a new method for correcting pose in 2.5D data sets representing face images.

According to the present invention, these objects are achieved by a pose correction method for correcting a pose in data representing face images, the method comprising:

Obtaining at least one test frame including A-2D near-infrared image data, 2D visible light image data, and a depth map;

C-estimating a pose of a face in the test frame by aligning the depth map according to a 3D model of the head of a known direction;

Mapping at least one of the 2D images onto a depth map to generate D-textured image data;

And projecting the textured image data in 2D to produce data representing an E-Pose calibrated 2D projected image.

The use of a depth map (i.e., a 2.5D data set) is advantageous because it improves the accuracy and robustness of the pose estimation step.

The use of visible light data is advantageous because it allows detection of features that depend on, for example, skin color or texture.

The use of near infrared (NIR) image data is advantageous because it is less dependent on illumination conditions than visible light data.

The head pose is estimated by fitting the depth map according to the existing 3D model in order to estimate the direction of the depth map. This pose estimation is particularly robust.

The 3D model may be a general model, that is, a user-independent model.

The 3D model may be a user-dependent model, e.g., a 3D model of the user's head whose identity needs to be verified.

The 3D model may be a sex-specific model, a race-specific model, or an age-specific model and may be selected based on a priori knowledge of gender, race and / or age.

The existing 3D model is used to estimate the pose of the face. However, the 2D image data is not mapped onto this 3D model, but is mapped onto the depth map.

The method may include, for example, another step of classifying the image, classifying the 2D projected image.

Classification may include face authentication, face identification, gender estimation, age estimation, and / or detection of other facial features.

The method may also include processing the 2D projected image.

The acquiring step may include temporal and / or spatial smoothing of the points in the depth map to remove noise generated by the depth sensor.

Estimating the pose may include, for example, a first step of performing a rough pose estimation based on the random regression forest method.

The step of estimating the pose may include the second step of estimating the fine pose. The fine pose estimation may be based on the outline of the pose estimation.

The fine pose estimation can be based on the alignment of depth maps according to the 3D model, for example, using strict iterative close point (ICP) methods.

The method may further comprise a basic face detection step prior to the pose estimation, to remove at least some portions of a 2D near-infrared image that does not belong to a face, and / or of the 2D visible light image, and / or the depth map .

The method may further comprise a background extracting step to remove portions of the 2D near-infrared image that do not belong to the background, and / or portions of the 2D visible light image, and / or the 2D near-infrared light.

The step of aligning the depth map according to the existing 3D model of the head may include scaling the depth map such that some of its area (e.g., maximum height) And the area to be matched.

Matching the depth map according to the existing 3D model of the head may include warping the depth map and / or the 3D model.

The method may include another step of correcting the illumination of the 2D visible light image data set based on the 2D near infrared image data set. Shadows or light zones in the 2D visible light image data set that do not appear in the 2D near infrared image data set can therefore be corrected.

The method may include another step of flagging portions of the pose corrected 2D projected image data corresponding to portions that are not visible on the depth map and / or on the 2D image.

The method may include another step of reconstructing portions of the pose corrected 2D projected image data corresponding to the unknown portions of the depth map.

The invention will be better understood with the aid of the description of an embodiment given by way of example and illustrated by the figures.

1 is a flow chart of a method of the present invention;
2 is a schematic view of an apparatus according to the invention;
3 shows an example of a 2D visible light image of a face;
Fig. 4 shows an example of a 2D infrared image of the face of Fig. 3; Fig.
5 is a view showing an example of a representation of a depth map of the face of FIG. 3;
Figure 6 shows an example of a representation of a textured depth map of the face of Figure 3;
7 shows an example of a general head model used for head pose estimation;
Fig. 8 is a diagram showing a fine pose estimation step by aligning a depth map in a 3D model; Fig.
9 is a diagram showing a pose corrected 2D projection of a depth map, with missing portions corresponding to parts of the head that do not exist in the depth map.
Figure 10 shows a pose corrected 2D projection of a 2.5D data set, with missing portions corresponding to portions of the head not present in the depth map reconstructed.
Figure 11 shows a pose corrected 2D projection of a 2.5D data set, with missing portions corresponding to parts of the head not present at 2.5D being reconstructed and displayed.

The following detailed description, taken in conjunction with the accompanying drawings, is intended as a description of various embodiments of the invention and is not intended to represent the only aspects in which the disclosure may be practiced. Each aspect described in this disclosure is only provided as an example or illustration of the present invention and is not necessarily to be construed as preferred or essential.

1 is a flow chart schematically illustrating key steps of an example of a pause correcting method according to the present invention. This method can be implemented as an apparatus or system as schematically shown in block diagram on Fig. In this example, the device includes a camera 101 for capturing an image of the user 100. [ The camera 101 may be a depth camera, such as a Kinect camera (trademark of Microsoft), a time-of-flight camera, or any other camera capable of generating an RGB-D data stream. have.

The camera 101 is connected to the processor 102 that accesses the memory 104 and is connected to the network via the network interface 103. [ The memory 104 may include a permanent memory portion for storing the computer code that causes the processor to execute at least some of the steps of the method of FIG. Memory 104, or portions of memory 104, may be removable.

As used herein, device 101 + 102 + 104 may be a mobile phone, a personal navigation device, a personal information manager (PIM), a car device, a gaming device, a personal digital assistant (PDA), a laptop, And / or a handheld computer, a smart glass, a smart clock, a smart TV, other wearable devices, and the like.

In the acquisition step (A) of FIG. 1, the test video stream is generated by the depth camera 101 to capture test images of the user 100 to be identified or otherwise classified.

In the present description, the expressions ("test images") specify that the pose typically includes images of the face that need to be calibrated during registration, as well as during testing (for identification or authentication).

Each frame of the test video stream preferably contains three temporally and spatially aligned data sets:

i) a first (optional) data set corresponding to a two-dimensional (2D) visible light image of a face of a user 100 (e.g., a grayscale or RGB image) One example is shown in Figure 3.

ii) a second (optional) data set representing a 2D Near Infrared (NIR) image of the face of the user 100; One example is shown in Fig.

iii) a depth map (i.e., a 2.5D data set) in which the value associated with each pixel is dependent on the depth of the source of light, i.e., its distance from the camera 101 to the depth sensor. This representation of the depth map is shown in Fig.

Figure 6 is another representation (201) of a frame, wherein the first set of RGB data is projected onto a depth map.

As can be seen in Figure 5, many low cost depth sensors produce noise depth maps, i.e., data sets with depth values assigned to each point, including noise. The deleterious effects of noise can be reduced by smoothing the depth map. Smoothing can include low-pass filtering of the depth map in space and / or time (across consecutive frames) domain.

In the basic face detection step (B) of Figure 1, a portion of each data set representing the user's face is detected and separated from the background and other elements of the image. This detection may be performed on any or all of the three data sets and may be applied to any or all of the data sets of those data sets.

In one embodiment, the basic face detection is based at least in part on thresholding of the depth map to exclude pixels that are not between a predefined depth range, e.g., 20 cm and 100 cm. Other known algorithms may be used to extract the background representing the user's face and to exclude the backgrounds, which include algorithms based on color detection, for example.

In the hair pose estimating step (C) of Fig. 1, the user's pose in the frame is estimated. In one example, this estimation is performed in two consecutive steps:

During a first portion of the head pose estimation step, a rough estimate of the head pose is determined. In one example, the calculation of this approximate estimate uses a random forest algorithm to quickly determine head pose with some accuracy. A rough estimation method of head pose with random regressive forest &Quot; Real Time Head Pose Estimation with Random Regression Forests ", Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference, 617-624, by G. Fanelli, Preferably, the position of the other key features of the nose and / or face is also determined during this step.

During the second phase of head pose estimation, a finer estimate of the head pose is determined. This fine estimate can start from a previously determined approximate estimate of the position of one point, such as direction and nose position, to improve speed and robustness. The fine estimate can be computed by determining the orientation of the 3D model of the head (Figure 7) corresponding best to the depth map. In one example, the matching of the 3D model to the test depth map is performed, for example, by minimizing the distance function between the points of the depth map and corresponding points of the 3D model using an iterative close contact (ICP) method . FIG. 8 schematically illustrates this step of fitting a test depth map 201 (textured in this example) into a 3D model 200 (represented in this example as a mesh).

This alignment step preferably includes at least a portion of its area corresponding to the test depth map, including adjustment of the 3D model.

The 3D model 200 may be generic, i.e., user independent. Alternatively, the 3D model is user-dependent and may be retrieved from a database of user-dependent 3D models based on, for example, the assumed identity of the user 100. The plurality of user-independent 3D models may also be stored and selected according to, for example, the assumed gender, age, or ethnicity of the user.

In one example, a private 3D model is created using a non-stringent iterative close-contact (ICP) method. The 3D model then includes a mesh with some constraints on positions and / or relationships between the nodes, to allow for deformations of the head, e.g., some real and limited deformation at the bottom of the face . In this case, the ICP method can try some deformations or morphs of the model in order to find the direction with the greatest conjugation given all possible deformations.

The output of the head pose estimating step may comprise a set of angles (phi, theta, po (psi)) describing three rotations for a given coordinate system.

In step (D) of Figure 1, 2D textures (in visible and NIR ranges) of the frame are mapped onto the depth map (UV mapping). It is possible to use only the gray scale value of the visible data set. This mapping can be performed before and after the pose correction, producing a textured depth map with known directions. Hidden parts For example, portions of a user's face that are hidden or obscured in the depth map are displayed or reconstructed as invalid by assuming, for example, the symmetry of the user's face.

This step may also include correction of illumination in the visible light and / or NIR image data sets. Correction of illumination may include correction of brightness, contrast, and / or white balance in the case of color images. In one preferred embodiment, the NIR data set is used to remove or attenuate shadows by compensating for changes in brightness that are present in portions of the visible light data set but not in corresponding portions of the NIR data sets, and / Reflect in the data set.

In step (E) of Figure 1, the textured 3D image is projected in 2D to produce at least one data set representing the pose corrected 2D projected image in the visible and / or NIR range. Various projections can be considered. First, variants caused by cameras and / or by perspective are preferably considered. Then, in one embodiment, the projection produces a frontal face image, i.e., a 2D image as seen from the viewer at the front of the user 100. It is also possible to generate a plurality of 2D projections, such as facial images that are not frontal, or, for example, one frontal face projection and one other profile projection. Other projections, including projections, or projections that reduce the size of the more deformable parts of the face, such as the mouth, may be taken into consideration, especially when introducing deformations to magnify the upper half of the face, . To facilitate comparison, it is also possible to transform the head into a generic model before comparison. Purely mathematical projections, such as projections onto space that are not readily representable, can also be considered.

FIG. 9 illustrates an example of a 2D textured projection 202 generated during step (E). Parts of the projection 203 that are not available in the depth map, for example, the hidden or occluded portions are thus marked.

FIG. 10 shows another example of a 2D textured projection 202 generated during step (E). In this example, portions of the projection 204 that are not available in the depth map, for example, the hidden or occluded portions, are reconstructed as non-textured images. The reconstruction may be based on available parts of the image, for example, by assuming that the user's face is symmetrical. Alternatively, or additionally, reconstruction can utilize a generic model of the head. Alternatively, or additionally, reconstruction may utilize available image portion data from other frames in the same video sequence.

FIG. 10 shows another example of a 2D textured projection 202 generated during step (E). In this example, portions of the projection 205 that are not available in the depth map, for example, the hidden or occluded portions, are reconstructed as a textured image. The reconstruction may be based on available parts of the image, for example, by assuming that the user's face is symmetrical. Alternatively, or additionally, reconstruction can utilize a generic model of the head. Alternatively, or additionally, reconstruction may utilize available image portion data from other frames in the same video sequence.

The method described above thus generates a user-pose corrected 2D test image data set based on the 2.5D test view obtained with the depth camera. During the face processing phase (F), this data set can then be used by a classification module such as a user identification or authentication module, or a gender estimation module, an age estimation module, The classification may be based on a single frame, for example, in a frame that can be classified according to highest reliability, or in a first frame that can be classified according to a reliability higher than a given threshold, or on a plurality of successive frames of the same video stream . Additionally, or alternatively, the classification may also be based on a directed textured 3D image. Other face processing may be applied during step F. [

The methods disclosed herein include one or more steps or operations for achieving the described method. The order and / or use of certain steps and / or actions may be altered without departing from the scope of the claims, unless a specific order of steps or acts is specified.

Depending on the embodiment, certain acts or events of any of the methods described herein may be performed in different sequences, or all may be added together, merged, or excluded (E.g., not all actions described are necessary for the performance of the method). In addition, in certain embodiments, operations or events may be performed concurrently, rather than sequentially, for example, through multi-threaded processing, interrupt processing, or multiple processors.

The various operations of the above-described methods may be performed by any suitable means capable of performing corresponding functions. Means may comprise circuitry, an application specific integrated circuit (ASIC), a processor, a field programmable gate array signal (FPGA) or other programmable logic device (PLD) May include various hardware and / or software component (s) and / or module (s), including, but not limited to, gate or transistor logic, discrete hardware components or any combination thereof.

As used herein, the terms ("determining" and "estimating") include broad operations. For example, "determining" and "estimating" may include computing, computing, obtaining, searching (e.g., searching in a table, database, or another data structure) . ≪ / RTI > In addition, "determining" and "estimating" may include receiving, accessing (e.g., accessing data in memory), and the like.

The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may reside in any form of storage medium known in the art. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, . A software module may contain a single instruction, or many instructions, and may be distributed among different programs, through several different code segments, and across multiple storage media. The storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral with the processor.

Accordingly, certain aspects may include a computer program product for performing the operations set forth herein. For example, such a computer program product may include a computer-readable medium having stored (and / or encoded) instructions and instructions may be executable by one or more processors to perform the operations described herein .

Various modifications and variations of the described embodiments of the invention will be apparent to those skilled in the art without departing from the scope of the invention as defined in the appended claims. Although the invention has been described in connection with certain preferred embodiments, it is to be understood that the invention is not to be unduly limited to such specific embodiments as claimed.

100: User 101: Camera
102: Processor
103: Network interface 104: Memory
200: 3D Model 201: Test Depth Map
202: 2D Textured Projection

Claims (16)

CLAIMS What is claimed is: 1. A pose correction method for correcting a pose in data representing face (100) images, comprising:
Obtaining at least one test frame including A-2D near-infrared image data, 2D visible light image data, and a depth map;
C-estimating a pose of a face in the test frame by aligning the depth map according to a 3D model of the head of a known direction;
Mapping at least one of the 2D images onto the depth map to generate D-textured image data;
And projecting the textured image data in 2D to produce data representing an E-Pose calibrated 2D projected image. The method according to claim 1,
And temporal and / or spatial smoothing of the points in the depth map. 3. The method according to claim 1 or 2,
The step (C) of estimating the pose includes, for example, a first step of performing a rough pose estimation based on a random forest, and another step of determining a more accurate estimation of the pose A method of correcting a pose. 4. The method according to any one of claims 1 to 3,
Wherein said step (C) of aligning said depth map according to a 3D model of a head of a known orientation uses an iterative close points (ICP) method. 5. The method according to any one of claims 1 to 4,
(C) prior to the estimation (C) of the pose to remove at least some portions of the 2D near infrared image data that do not belong to the face, and / or the 2D visible light image data, and / Further comprising step (B). 6. The method according to any one of claims 1 to 5,
Wherein the 3D model is user independent. 6. The method according to any one of claims 1 to 5,
Wherein the 3D model is user-dependent. 8. The method according to any one of claims 1 to 7,
Wherein the 3D model warps to adapt the 3D model to the user. 9. The method according to any one of claims 1 to 8,
Wherein said step (C) of aligning said depth map according to an existing 3D model of the head comprises bending said 3D model. 10. The method according to any one of claims 1 to 9,
Further comprising correcting illumination of portions of the 2D visible light image data based on the 2D near infrared image data. 11. The method according to any one of claims 1 to 10,
Further comprising: flagging portions of the pose corrected 2D projected image data corresponding to portions not visible on the depth map. 12. The method according to any one of claims 1 to 11,
Further comprising reconstructing portions of the pose corrected 2D projected image data corresponding to unknown portions of the depth map. 13. The method according to any one of claims 1 to 12,
Further comprising classifying the 2D projected image (F). An apparatus comprising: a depth map camera (101) arranged to obtain at least one test frame comprising 2D near infrared image data, 2D visible light image data, and a depth map, as well as a depth map camera A processor having a memory for storing the program that causes the computer to execute the steps of:
The steps,
C-estimating a pose of a face in the test frame by aligning the depth map according to a 3D model of the head of a known direction;
Mapping at least one of the 2D images onto the depth map to generate D-textured image data;
And projecting the textured image data in 2D to produce data representing an E-Pose calibrated 2D projected image, the apparatus comprising: An apparatus comprising means for performing the method of any one of claims 1 to 14. A computer program product comprising:
Obtaining at least one test frame including A-2D near-infrared image data, 2D visible light image data, and a depth map;
C-estimating a pose of a face in said test frame by aligning said depth map according to a 3D model of a head of known direction;
Map at least one of the 2D images onto the depth map to generate D-textured image data;
And computer-readable media comprising executable instructions for projecting the textured image data in 2D to produce data representing an E-Pose calibrated 2D projected image.

Images