JP4807167B2 - Impersonation detection device - Google Patents

Description

  The present invention relates to a technique for preventing impersonation using a photograph or the like.

  In recent years, mobile devices (cell phones, PDAs (Personal Digital Assistants), mobile computers, etc.) have become more sophisticated and multifunctional. Accordingly, security measures for preventing unauthorized use of portable devices have become important.

  As a security measure, facial recognition technology using a camera that is becoming standard equipment in portable devices has attracted attention. Personal authentication (face authentication) using face recognition technology is effective as a highly convenient security because it can capture a face from a natural motion and check the owner.

  On the other hand, there is a countermeasure against “spoofing” using a photograph or the like as a face recognition technique. Spoofing is an act in which another person tries to authenticate using his / her face photo or the like. In particular, recent mobile devices are equipped with megapixel-class cameras, and are equipped with functions such as autofocus and close-up functions, so it is possible to impersonate even small-sized photos such as snapshots and licenses. There is a risk of being able to do it.

  As an idea for detecting impersonation, a method is known in which a user is requested to perform a predetermined action and whether the user has performed a movement as requested (see Patent Document 1). However, this method imposes an operation burden on the execution of the predetermined operation on the user, so that convenience that is an advantage of the face recognition technology is lost.

  As another idea, there are a method of taking multiple shots while changing the illumination conditions and confirming how the face is shaded (see Patent Document 2), and a method of examining whether it is a solid by shooting from various angles. is there. However, these methods require a plurality of illumination devices and a plurality of cameras or a camera moving mechanism, which makes the apparatus configuration large. In addition, since it is necessary to shoot many times by changing the illumination conditions and the shooting angle, it leads to a loss of convenience.

Although it is not a face recognition technology, as an impersonation detection method in iris authentication, it is possible to take a picture while randomly lighting a plurality of LEDs, and whether or not the identified person's line of sight follows the illuminated LED. A method for determining whether or not a living body is a living body is known (see Patent Document 3). However, this method also has a problem in terms of convenience because the burden on the user is large, and there is also a problem that it is difficult to reduce the size because the apparatus configuration is complicated.
JP 2004-259255 A Japanese Patent Laid-Open No. 11-339048 JP 2000-83930 A JP 2004-274669 A

  The present invention has been made in view of the above circumstances, and the object of the present invention is to provide a technique capable of detecting impersonation by a photograph or the like with simple processing without causing an increase in the size of the apparatus or a decrease in convenience. It is to provide.

The apparatus according to the present invention includes an imaging unit and a display unit as a basic configuration. Then, the image acquisition means captures one second image used for impersonation determination from the imaging means while displaying the first image for attracting the user's attention on the display means.

  As the first image, for example, an image (moving image) sequentially taken from the imaging unit can be used. According to this configuration, the user adjusts the appearance of the face position, size, orientation, and the like while viewing the confirmation image displayed on the display means. Further, as the first image, an image (including moving images and animations) prepared in advance in the apparatus may be used instead of the image captured from the imaging unit. Alternatively, an image obtained by combining an image captured from the imaging unit and an image prepared in advance in the apparatus may be used.

  According to the above configuration, since the first image for drawing the user's attention is displayed on the display unit, the second image can be captured while the user is gazing at the display unit for normal use. Is done. That is, when the second image is not a spoofed image, there is a high possibility that the line of sight of the face in the image is directed to the display means instead of the imaging means.

  Here, “spoofed image” refers to an image obtained by taking a photograph or the like (including not only a hard print but also an image displayed on a display).

  On the other hand, in a photograph or the like, the line of sight is usually facing the front (that is, the direction of the camera that took the photograph or the like). A person who performs impersonation will face a photograph or the like directly on the imaging means, and therefore, even in an “impersonated image” obtained by further photographing the photograph or the like, the line of sight faces the front. That is, when the second image is a spoofed image, there is a high possibility that the line of sight of the face in the image is directed to the imaging means instead of the display means.

  The present invention detects impersonation by utilizing the above-mentioned characteristics found in an impersonated image. Specifically, the face detection unit detects a face from the second image, the gaze estimation unit estimates the gaze of the face detected by the face detection unit, and the impersonation determination unit is the gaze estimation unit. Based on the positional relationship between the estimated line of sight and the display unit or the imaging unit, it is determined whether or not the second image is a spoofed image.

  According to this configuration, impersonation by a photograph or the like can be detected by a very simple process such as analyzing the positional relationship between the line of sight of the face in the second image and the display unit or the imaging unit. In addition, a special illumination device or imaging device is not required for taking the second image, and further, impersonation can be determined from a single image, which may lead to an increase in the size and convenience of the device when the present invention is implemented. There is no.

  Various algorithms can be considered for the determination algorithm by the impersonation determination means. For example, the spoofing determination unit may determine that the second image is a spoofed image when it is determined from the positional relationship that the face does not look at the display unit.

  Alternatively, the impersonation determination unit may consider that the second image is not an impersonation image when it is determined from the positional relationship that the face is looking at the display unit.

  Alternatively, the spoofing determination unit may regard the second image as a spoofed image when it is determined from the positional relationship that the face is looking at the imaging unit.

  Alternatively, the spoofing determination unit may consider that the second image is not a spoofing image when it is determined from the positional relationship that the face does not look at the imaging unit.

  The present invention may be regarded as an impersonation detection device having at least a part of the above means. The present invention can also be understood as an impersonation detection method including at least a part of the above processing, a program for realizing the method, or a recording medium on which the program is recorded. Each of the above means and processes can be combined with each other as much as possible to constitute the present invention.

For example, the detection method spoofing according to the present invention includes an imaging device, the device including a display unit, while displaying the first image to the user's attention to the display means, one used for the determination impersonation second a process of capturing an image from the imaging unit, wherein the processing for detecting a face from the second image, a process of estimating the line of sight of the detected face, the estimated line-of-sight to the display unit or the imaging And a process of determining whether or not the second image is a spoofed image based on a positional relationship with the means.

In addition, the impersonation detection program according to the present invention is a piece used for impersonation determination while displaying a first image for attracting the user's attention on the display means on an apparatus including an imaging means and a display means. second a process of capturing an image from the imaging unit, wherein the processing for detecting a face from the second image, a process of estimating the line of sight of the detected face, the estimated line-of-sight to the display unit or the imaging And a process for determining whether the second image is a spoofed image based on a positional relationship with the means.

  According to the present invention, it is possible to detect impersonation by a photograph or the like with a simple process without causing an increase in the size of the apparatus or a decrease in convenience.

  Exemplary embodiments of the present invention will be described in detail below with reference to the drawings.

<Device configuration>
FIG. 1 is a diagram showing an appearance of a mobile phone in which an impersonation detection function is implemented, and FIG. 2 is a block diagram showing a functional configuration of the mobile phone. In this embodiment, a mobile phone is taken as an example, but the impersonation detection according to the present invention can be applied to various types of devices having an imaging function and a display function. Examples of such a device include a PDA, a personal computer, and a security device that performs entry / exit management and lock control.

  The mobile phone according to the present embodiment includes an imaging unit 10, a display unit 11, an image storage unit 12, an image acquisition unit 13, a face detection unit 14, a gaze estimation unit 15, and an impersonation determination unit 16. These functional elements are realized by an arithmetic processing unit executing software (program) and controlling hardware resources such as a camera, a memory, and a display as necessary. However, part or all of these functional elements may be realized by a dedicated chip.

  The imaging unit 10 includes a camera 10a and a function for controlling the camera 10a. As the camera 10a, a digital camera including an optical system and an image sensor (CCD, CMOS sensor, etc.) can be preferably used. The display unit 11 includes a display 11a and a function for controlling the display 11a. As the display 11a, a liquid crystal display, a plasma display, or the like can be preferably used.

This camera 10a is an inner camera provided on the same side as the display 11a. The inner camera was originally provided for shooting a speaker in a videophone, but it is suitable for self-portrait and is also used for face authentication on a mobile phone. Images captured and captured by the camera 10a are displayed on the display 11a almost in real time. The user can easily confirm his / her own appearance by looking at the display image.

  Note that the camera 10a (lens) must be arranged around the display 11a due to physical restrictions of the hardware. In the example of FIG. 1, a camera is arranged at the upper left of the display. Depending on the model, a camera may be placed directly above or below the display. As described above, since the positions of the camera 10a and the display 11a are slightly shifted, the camera 10a captures a person facing the display 11a from an oblique direction.

  The image storage unit 12 is a storage device that temporarily stores an image captured by the imaging unit 10. As the storage device, any specific technique such as a volatile memory or a non-volatile memory may be applied.

<Authentication process>
The mobile phone of this embodiment has an authentication function using a face image. When the user tries to use a function with security, the authentication function is automatically activated, and the process shown in the flowchart of FIG. 3 is executed.

  When the authentication function is activated, the imaging unit 10 captures images at regular time intervals (step S10), and the display unit 11 sequentially displays the captured images on the display 11a (step S11). The user can adjust the position, size, orientation, and the like of the face while viewing the confirmation image displayed on the display 11a. That is, this confirmation image (first image) has a role of attracting the user's attention to the display 11a.

  When the user presses the shutter button (step S12), the imaging unit 10 performs shooting (step S13), and the image acquisition unit 13 captures the image as a target image (second image) used for authentication and impersonation determination. (Step S14).

  Next, the face detection unit 14 detects a human face from the target image and specifies the position of a facial organ such as an eye (step S15).

  Any existing technique may be applied to the face detection processing by the face detection unit 14. For example, (1) a method for detecting a face by template matching using a reference template corresponding to the contour of the entire face, (2) a face by template matching based on a facial organ (eg, eyes, nose, ears). (3) A technique for detecting a vertex such as a head by chroma key processing and detecting a face based on the vertex, and (4) detecting an area close to the skin color and detecting the area as a face. And (5) learning using a teacher signal using a neural network and detecting a face-like area as a face. Instead of automatically detecting a face, a person may specify the position and size of the face. When a plurality of human faces are detected from the image, the face to be processed may be determined based on predetermined criteria such as the size, orientation, and position of the face. The processing may be interrupted assuming that

  If a face cannot be detected in step S15 (step S16; NO), the process ends as it is. In this case, since the security is not released, the user cannot use the same function.

  If a face can be detected in step S15 (step S16; YES), the line-of-sight estimation unit 15 analyzes the target image to estimate the line of sight of the face detected from the target image (step S17).

  The reason for paying attention to the line of sight is as follows. In photographs, etc. (including not only hard prints such as snapshots, licenses, passports, but also images displayed on other displays), the line of sight is facing the front (so-called “camera line of sight”). It is normal. And when impersonating using such a photograph etc., as shown to Fig.4 (a), since a photograph etc. will be directly opposed to the camera 10a, it was obtained by further photographing the photograph etc. Even the “spoofed image” has a camera line of sight as shown in FIG.

  On the other hand, in the case of normal usage, as shown in FIG. 5A, the target image is taken while the user is gazing at his / her appearance on the display 11a. Therefore, when the camera 10a is arranged above the display 11a as in the present embodiment, the line of sight of the face in the target image is slightly downward as shown in FIG.

  That is, when the line of sight of the face in the target image faces the direction of the camera 10a (front direction), the target image is likely to be a spoofed image, and the line of sight is in the direction of the display 11a (direction other than the front). ), It is highly possible that the target image is not a spoofed image. The impersonation determination of the present embodiment uses such features that appear in the target image.

  Any existing technique may be applied to the line-of-sight estimation process by the line-of-sight estimation unit 15. For example, the techniques described in the following documents 1 and 2 can be applied. Note that the line-of-sight estimation here does not require strict accuracy, and may be of an accuracy that can determine whether the line of sight is facing the camera 10a or the display 11a.

[Reference 1] Yasuhiro Ono, Takahiro Okabe, Yoichi Sato, "Gaze direction estimation from low resolution images considering indeterminacy of eye region clipping", Image Recognition / Understanding Symposium (MIRU2005) Proceedings, pp. 96-103, July 2005 [Reference 2] Mayumi Yuasa, “Real-time high-precision pupil detection technology”, Toshiba Review Vol. 60 No. 11, 2005

  When the line of sight is estimated, the impersonation determination unit 16 analyzes the positional relationship between the line of sight and the display 11a (step S18), and determines whether or not the face in the target image is looking at the display 11a (step S19). If it is determined that the user is looking at the display 11a, the image is regarded as not a spoofed image (that is, authentication by a legitimate user), and the security is released (step S19; YES, step S20). In other cases, the image is regarded as a spoofed image, and the process is terminated without releasing the security (step S19; NO).

  As a method for analyzing the positional relationship between the line of sight and the display, for example, (1) a method for determining that the display is viewed when the angle of the line of sight in the camera coordinate system (image coordinate system) is within a predetermined range; A method to determine the intersection of the line of sight (straight line) and the display (plane) in the camera coordinate system, (3) Whether the focus exists on the display (plane) by estimating the focus (gaze point) from the position of the eyes of both eyes A method of determining, etc. can be considered. It is assumed that the optical axis and angle of view of the camera 10a, the relative positional relationship between the camera 10a and the display 11a, and the like are set in the impersonation determination unit 16 in advance.

  According to the above process, it is possible to detect impersonation by a photograph or the like only with a very simple process such as analyzing the positional relationship between the line of sight of the face in the target image and the display 11a.

In addition, no special illumination device or imaging device is required to capture the target image, and impersonation can be determined from a single image, which may lead to an increase in the size and convenience of the device when implementing impersonation detection. Absent.

  In the impersonation detection method of this embodiment, it is impossible to detect impersonation because a photograph of a scolding face is used, and if the user faces the camera at the time of shooting the target image, a legitimate user In spite of this, there is a disadvantage that it is determined to be impersonation. However, the above method has a great merit in mounting in that no additional hardware is required and the processing is simple. Therefore, the above method can be preferably employed when good operability and low cost are more important than high security. Note that false detection can be reduced in combination with other processing.

<Modification 1>
FIG. 6 is a flowchart showing a modification of the authentication process. The same processes as those in the flowchart of FIG. 3 are given the same step numbers. In the example of FIG. 6, after face detection, face matching is performed using the face image (step S <b> 21), and the process proceeds to impersonation determination based on gaze estimation only when the face is determined to be the person's face (step S <b> 21). S22). Of course, face matching may be performed after impersonation determination. In this way, by combining with personal authentication by face matching, it is possible to reduce false detection and improve security.

<Modification 2>
In the impersonation determination, when it is determined that the face is looking at the display 11a, it is determined that the target image is not an impersonation image. However, the determination algorithm is not limited to this, and other algorithms can be employed. For example, when it is determined that the face is not looking at the display 11a, the target image may be regarded as a spoofed image. Alternatively, when it is determined that the face is looking at the camera 10a, the target image may be regarded as a spoofed image. Conversely, when it is determined that the face is not looking at the camera 10a, the target image may be regarded as not a spoofed image. Since the line of sight of the face in the image faces the front when looking at the camera, it is easier to determine whether or not you are looking at the camera than when you are looking at the display.

<Modification 3>
When the authentication process is started, a guide may be displayed on the display 11a. Examples of the guide include a frame for matching the position and size of the face, and an operation guide using text. By displaying such a guide, the quality of the target image can be improved, and the reliability of the authentication process is improved. Moreover, since the possibility that the user gazes at the display 11a is increased due to the guide display, it is possible to expect false detection reduction of impersonation.

<Modification 4>
In the above embodiment, the target image is shot by the user pressing the shutter button. However, it is also preferable that the apparatus automatically captures the target image. For example, it is preferable to use a control method in which a target image is automatically captured when a certain period of time elapses after the authentication process is started, or a target image is automatically captured when a subject's movement becomes small by analyzing sequentially captured images. When the target image is automatically shot, the user does not know the shooting timing. Therefore, the possibility that the target image is shot while the user is gazing at the display 11a is increased, and it can be expected that false detection of impersonation is reduced.

<Modification 5>
In the above embodiment, confirmation images sequentially taken from the imaging unit 10 are displayed on the display 11a in order to attract the user's attention. However, for the purpose of attracting the user's attention, an image prepared in advance in the memory may be displayed on the display 11a instead of an image captured from the imaging unit 10. For example, a character image, a pattern image (an image made up of a plane figure such as ◯ or Δ, a solid figure such as a rectangular parallelepiped, a geometric pattern such as a wavy line), a photographed image of a person or a landscape, and the like can be used. In particular, a moving animation or moving image can effectively attract the user's attention. Of course, an image prepared in advance in the memory may be combined with the image captured from the imaging unit 10.

FIG. 1 is a diagram illustrating an appearance of a mobile phone in which an impersonation detection function is implemented. FIG. 2 is a block diagram showing a functional configuration of the mobile phone of FIG. FIG. 3 is a flowchart showing the authentication process. FIG. 4A is a diagram illustrating spoofing using a photograph, and FIG. 4B is a diagram illustrating a spoofed image obtained by taking a photograph. FIG. 5A is a diagram illustrating normal user authentication, and FIG. 5B is a diagram illustrating an image obtained by photographing the user. FIG. 6 is a flowchart showing a modification of the authentication process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Imaging part 10a Camera 11 Display part 11a Display 12 Image memory | storage part 13 Image acquisition part 14 Face detection part 15 Gaze estimation part 16 Impersonation determination part

Claims (7)

Imaging means;
Display means;
An image acquisition unit that captures from the imaging unit one second image for use in impersonation determination while displaying a first image for attracting the user's attention on the display unit;
Face detection means for detecting a face from the second image;
Eye gaze estimation means for estimating the gaze of the face detected by the face detection means;
Impersonation determination means for determining whether the second image is a spoofed image based on the positional relationship between the line of sight estimated by the line-of-sight estimation means and the display means or the imaging means;
Impersonation detection device comprising The impersonation determination means is
The impersonation detection device according to claim 1, wherein the second image is regarded as an impersonation image when it is determined from the positional relationship that the face does not look at the display unit. The impersonation determination means is
The impersonation detection device according to claim 1, wherein when it is determined from the positional relationship that the face is looking at the display unit, the second image is regarded as not an impersonation image. The impersonation determination means is
The impersonation detection device according to claim 1, wherein the second image is regarded as an impersonation image when it is determined from the positional relationship that the face is looking at the imaging unit. The impersonation determination means is
The impersonation detection device according to claim 1, wherein, when it is determined from the positional relationship that the face does not look at the imaging unit, the second image is regarded as not an impersonation image. An apparatus comprising an imaging means and a display means,
A process of capturing one second image used for impersonation determination from the imaging unit while displaying a first image for drawing the user's attention on the display unit;
Detecting a face from the second image;
A process of estimating the gaze of the detected face;
A process of determining whether the second image is a spoofed image based on a positional relationship between the estimated line of sight and the display unit or the imaging unit;
Impersonation detection method to execute. In an apparatus provided with an imaging means and a display means,
A process of capturing one second image used for impersonation determination from the imaging unit while displaying a first image for drawing the user's attention on the display unit;
Detecting a face from the second image;
A process of estimating the gaze of the detected face;
A process of determining whether the second image is a spoofed image based on a positional relationship between the estimated line of sight and the display unit or the imaging unit;
Impersonation detection program that executes

Images