JP6993124B2 - Face matching device and face matching method - Google Patents

Description

The present invention relates to a face matching device and a face matching method for performing face authentication to verify the identity of a person.

Conventionally, various devices using face recognition processing have been used. For example, Patent Document 1 describes a gate system including a verification device that confirms an identity by face authentication processing and a gate device that opens and closes a gate according to the result of face authentication processing. At the time of immigration inspection conducted at the airport, the collation device confirms that the passport called the IC passport incorporating the IC chip belongs to the traveler himself / herself by face recognition processing. Specifically, the face image of the traveler who has the passport is imaged, and the face image of the passport holder registered in the IC chip of the passport is read out. Then, it is confirmed by the face recognition process that the face image acquired by imaging the traveler matches the face image read from the IC chip. The gate device controls the opening and closing of the gate to allow the pass of the traveler who can confirm the identity.

Japanese Unexamined Patent Publication No. 2016-53896

However, further functional improvement of the above-mentioned conventional technique is required. For example, when performing immigration at an airport, if the time required for face recognition processing or the time required to pass through the gate after identity verification is long, a long line will be formed at the immigration office. For this reason, it is required to shorten the processing time for a traveler who has been admitted to Japan after obtaining the result of identity verification by face recognition processing to pass through the gate.

This is not limited to immigration. For example, when face recognition is performed at the entrance of an event or when face recognition is performed when entering an office, when a large number of face recognition targets visit during the same time period, it takes time for a series of processes including face recognition. If it takes, a long line will be created.

The present invention has been made to solve the above-mentioned problems caused by the prior art, and an object of the present invention is to provide a face matching device and a face matching method capable of performing a series of processes including face recognition in a short time. And.

In order to solve the above-mentioned problems and achieve the object, the present invention presents the face image of the person to be authenticated or the captured face data which is the feature amount of the face image, and the face image registered in advance or the features of the face image. It is a face matching device that performs face matching processing for collating with registered face data, which is an amount, and confirms whether or not the registered face data is the registered person, and is the subject to be authenticated. A camera that captures a person, a storage unit that detects a face from the image captured by the camera and stores the captured face data, an acquisition unit that acquires the registered face data, and the registered face data are acquired. A face matching process for collating the captured face data previously stored in the storage unit with the registered face data acquired by the acquisition unit is executed to confirm whether or not the person to be authenticated is the person himself / herself. The control unit includes a control unit that performs a predetermined process based on the result of the face matching process , and an input operation unit that accepts an input operation of the person to be authenticated instructing the start of the face matching process by the control unit . The face collation process was started before the input operation was performed by the person to be authenticated in the input operation unit, and the identity of the person to be authenticated could be confirmed by the face collation process and the input operation unit. It is characterized in that the predetermined process is performed on condition that the input operation is performed by the person to be authenticated .

Further, the present invention is characterized in that, in the above invention, the object of interest that the person to be authenticated pays attention to is further provided, and the camera is installed in the vicinity of the object of interest.

Further, in the present invention, the present invention further includes an object of interest that the person to be authenticated pays attention to, and the camera acquires a frontal face image of the person to be authenticated when the person to be authenticated pays attention to the object of interest. It is characterized by being installed in a position where it can be.

Further, in the present invention, the present invention further includes the object of interest that the person to be authenticated pays attention to, and the camera is the attention that the person to be authenticated pays attention to before the acquisition unit acquires the registered face data. It is characterized by being installed near the target.

Further, in the present invention, the object of interest is the display unit, and the display unit displays the registered face data toward the person to be authenticated before the acquisition unit acquires the registered face data. It is a feature.

Further, the present invention is characterized in that, in the above invention, the object of interest is a mirror surface, and the mirror surface reflects the face of the person to be authenticated before the acquisition unit acquires the registered face data. ..

Further, the present invention is characterized in that, in the above invention, the object of interest to be focused on by the person to be authenticated is further provided with the plurality of cameras, and the plurality of cameras are installed in the vicinity of the object of interest.

Further, the present invention is characterized in that, in the above invention, a plurality of objects of interest to be focused on by the person to be authenticated and a plurality of the cameras are further provided, and the cameras are installed in the vicinity of each of the objects of interest. do.

Further, in the present invention, in the above invention, a display unit is further provided at a position lower than the face of the person to be authenticated as an object of interest to be noticed by the person to be authenticated, and the camera is a camera arranged in the vicinity of the display part. And a camera arranged at a position higher than the camera, the subject is characterized in that the person to be authenticated is imaged by a plurality of cameras and an image is acquired.

Further, in the present invention, the present invention further includes a gate portion that opens the gate based on the result of identity verification to allow passage, and the predetermined process performed by the control unit is opening / closing control of the gate portion. It is characterized by that.

Further, in the present invention, in the above invention, the control unit has the imaged face data stored in the storage unit before the acquisition unit acquires the registered face data, and the registered face data acquired by the acquisition unit. After executing the face matching process for collating, the camera captures the imaged person to create new captured face data, and each time new captured face data is acquired, the captured face data and the registered face are captured. It is characterized by performing face matching processing with data.

Further, the present invention is a face matching device that collates a captured face image or captured face data which is a feature amount of the face image with a pre-registered face image or registered face data which is a feature amount of the face image. The camera, the storage unit for storing the captured face data created by detecting the face from the image captured by the camera, the acquisition unit for acquiring the registered face data, and the registered face data are acquired. A control unit that performs face matching processing for collating the captured face data previously stored in the storage unit with the registered face data acquired by the acquisition unit, and an input instructing the start of the face matching process by the control unit. The control unit includes an input operation unit that accepts operations, and the control unit starts the face matching process before the input operation is performed in the input operation unit, and the face matching process is successful and the input operation unit is used. It is characterized in that a predetermined process is executed on condition that the input operation is performed .

Further, the present invention is a face that collates the face image of the person to be authenticated or the captured face data which is the feature amount of the face image with the face image registered in advance or the registered face data which is the feature amount of the face image. A face matching method for confirming whether or not the registered face data is the registered person by performing a matching process, the imaging step in which the camera captures the authenticated person, and the camera. A storage step of detecting a face from an image captured by the user and storing the captured face data created in the storage unit, an acquisition step of acquiring the registered face data by the acquisition unit, and the storage before the registered face data is acquired. A confirmation step of executing a face matching process for collating the captured face data stored in the unit with the registered face data to confirm whether or not the person to be authenticated is the person himself / herself, and an instruction to start the confirmation step. The reception step of accepting the input operation of the person to be authenticated and the predetermined processing step of executing the predetermined process based on the result of the face matching process are included, and the confirmed face matching process includes the input by the person to be authenticated. The predetermined process is executed on condition that the identity of the person to be authenticated can be confirmed by the face matching process and the input operation by the person to be authenticated is accepted, which is started before the operation is accepted. It is a feature.

According to the present invention, a series of processes including face recognition can be performed in a short time.

FIG. 1 is a schematic view showing a top view of the immigration office where the face recognition gate device is installed. FIG. 2 is an external view of the face recognition gate device as seen from the direction facing the operation display unit. FIG. 3 is a perspective view showing the appearance of the face recognition gate device. FIG. 4 is a diagram for explaining the size of the face recognition gate device. FIG. 5 is a block diagram showing a configuration example of an immigration system including a face recognition gate device. FIG. 6 is a diagram showing an example of a screen displayed on the management terminal. FIG. 7 is a diagram showing an example of a screen displayed when the face recognition gate device to be managed is changed. FIG. 8 is a diagram showing an example of a screen of a management terminal in which information on a traveler for whom a confirmation item has occurred is enlarged and displayed. FIG. 9 is a diagram showing an example of a screen displayed on a management terminal for performing a manned examination. FIG. 10 is a diagram showing an example of a screen displayed on the monitoring terminal. FIG. 11 is a perspective view for explaining the configuration of the passport reader. FIG. 12 is a diagram showing the appearance of the passport mounting table. FIG. 13 is a diagram showing a method of placing a passport on a passport mounting table. FIG. 14 is a schematic diagram illustrating a method of detecting a passport by a passport reader. FIG. 15 is a perspective view showing the appearance of the indicator light unit as viewed from the rear side. FIG. 16 is a diagram for explaining an example of state display by the state display unit. FIG. 17 is a flowchart showing the flow of processing by the face recognition gate device. FIG. 18 is an example for explaining an example in which the control unit determines that guidance to a traveler is necessary. FIG. 19 is a diagram showing a setting example in which the guidance method is set according to the proficiency level of the traveler. FIG. 20 is a flowchart showing the flow of operation guidance. FIG. 21 is a diagram showing an example of a layout in which a plurality of face recognition gate devices are arranged. FIG. 22 is a diagram showing an example of a layout in which a plurality of face recognition gate devices are arranged. FIG. 23 is a diagram showing an example of a layout in which a plurality of face recognition gate devices are arranged. FIG. 24 is a diagram showing an example of a layout in which a plurality of face recognition gate devices are arranged.

Hereinafter, the face matching device and the face matching method according to the present invention will be described with reference to the accompanying drawings. The face matching device combines the face image of the subject to be face-authenticated or the captured face data which is the feature amount of the face image and the pre-registered face image or the registered face data which is the feature amount of the face image. Perform face matching processing for matching. By the face matching process, the person to be authenticated performs the process of identity verification to determine whether or not the registered face data is the registered person.

In the present embodiment, a case will be described as an example in which a face authentication gate device that allows the passage of the gate only to the person to be authenticated whose identity has been confirmed is configured by the face verification device. This face recognition gate device is composed of a face recognition device (face matching device) and a gate device that permits the passage of the gate based on the result of face recognition (face matching). At various places such as immigration at airports, admission to venues where concerts and sports competitions are held, admission to buildings such as museums, etc., the person subject to identity verification is examined to determine whether the gate can pass or not. A face recognition gate device can be used for determination. In this embodiment, the details of the face recognition gate device will be described by taking the immigration office of the airport as an example.

In addition, face authentication is to confirm the identity based on the result of the face matching process, but confirming the match between the captured face data and the registered face data in the face matching process and the matching result in the face matching process. There is no big difference in a series of processes from confirming that the person to be authenticated is the person himself / herself and authenticating the person. The difference is that the result of matching the captured face data and the registered face data means that the person is the person himself / herself. In a series of processes, the process for the person and the process for other than the person may be switched depending on the collation result between the captured face data and the registered face data. In the present embodiment, face recognition includes face matching, for example, a face recognition device includes a face matching device.

Further, the person to be authenticated to be face-authenticated is also the person to be verified to be face-matched, and is also the operator of the device when face recognition is performed by the device. Furthermore, the person to be authenticated has an aspect depending on the situation. For example, in immigration and immigration, the certified person is also a traveler, an immigrant, and a returnee. In the scene of events such as concerts and sports competitions, the certified person is also a visitor or an spectator. Employees and visitors are certified at the entrance of the office, and guests are certified at hotel check-in. In the present embodiment, an example relating to the traveler will be described, but the traveler is a certified person, a collated person, and an operator when viewed from the device.

First, the outline of the face recognition gate device used at the immigration office will be described. The face recognition gate device (face matching device) acquires information such as a face image of the holder registered in the IC chip embedded in the passport called an IC passport. The face recognition gate device uses face recognition processing technology to capture the face image (registered face data) of the holder acquired from the IC chip and the traveler (certified person) who has a passport with a camera. Identity verification is performed by face matching processing that collates the acquired face image (captured face data). The face recognition gate device determines whether or not the traveler may be permitted to enter the country based on the result of identity verification or the like. When permitting a traveler to enter the country, the facial recognition gate device opens a normally closed gate to allow the traveler to pass. On the other hand, if you do not allow entry, you will be instructed to the traveler to be examined by an examiner.

Next, the layout of the immigration office where a plurality of face recognition gate devices are installed and the immigration inspection performed by the layout will be described. FIG. 1 is a schematic view showing a top view of the immigration office where the face recognition gate device is installed. Six examiners 610 (610a to 610f) in charge of immigration are waiting for each of the three counters 600 (600a to 600c). Two management terminals 30 (30a to 30f) are arranged at each counter 600. One management terminal 30 is assigned to each examiner 610. In addition to the face recognition gate device 10 and the management terminal 30 shown in FIG. 1, the management server 20 that manages information related to immigration and the monitoring terminal 40 that monitors the immigration office using the surveillance camera provided in the face recognition gate device 10. The immigration system (face recognition system) has been constructed by (see Fig. 5), and the details will be described later.

As shown in FIG. 1, a partition 700 is placed at the immigration office to regulate the flow of travelers 620 (620a to 620k). As the partition, for example, a belt pole type partition that regulates the flow of people by stretching a belt, a panel type partition used as a partition, and the like are used. The traveler 620 enters the unmanned or manned examination space of the immigration office from the lower side of FIG. Then, after undergoing immigration, proceed to the upper side of FIG. 1, pass through the immigration office from between the counter 600, and enter the country. The unmanned examination space is a space where the immigration examination is performed by the face recognition gate device 10 (10a to 10l) without the examiner 610 in charge of immigration examination. In the unmanned examination space, a number of airport staff 611 assisting travelers 620 as a concierge are stationed. The manned examination space is a space where immigration examination is conducted by the examiner 610 at the counter 600.

Only a traveler 620 of a predetermined age or older can undergo immigration inspection using the face recognition gate device 10. Therefore, a child whose age does not meet the conditions, a traveler 620a such as a companion of the child, enters the manned examination space on the right side of FIG. 1, proceeds through the passage 702, and is examined by the examiner 610e. Travelers 620 who have traveled along the passage 702 are lined up in front of the stop line 710 (lower side of the drawing). The examiner 610e calls the travelers 620 waiting for the examination at the stop line 710 to the counter 600c in order to request the presentation of the passport, and performs the immigration examination as before. A traveler 620 who has passed the examination and is admitted to the country enters the country through the immigration checkpoint from the left side of the counter 600c.

At the immigration office, a plurality of substantially square booths 701 (701a to 701l) shown by a broken line in FIG. 1 are provided, and one face recognition gate device 10 is installed in each booth 701. The face recognition gate device 10 includes a gate device 11 that opens and closes the gate 12. The face authentication gate device 10 controls the opening and closing of the gate 12 according to the result of identity verification by the face authentication process. In FIG. 1, the symbols of the gate device 11 and the gate 12 are shown only in the booths 701a and 701g of each row, but all the face recognition gate devices 10 include the gate device 11 that opens and closes the gate 12. .. Further, FIG. 1 shows an example in which six substantially square booths 701 are provided in one row, and a total of 12 booths 701 are provided in two rows. However, the number of booths 701 is not particularly limited, and the immigration office is not particularly limited. It is determined according to the area of.

The face recognition gate devices 10 in each row are installed in the same orientation. As shown in FIG. 1, the face recognition gate devices 10a to 10f in the left column are installed with the gate 12 on the left outer side. The face recognition gate devices 10g to 10l in the right column are installed with the gate 12 on the right outside. In both the left and right rows, all face recognition gate devices 10 are provided with a gate 12 far from the counter 600. A passage 703 in the unmanned examination space is formed between the rows of the face recognition gate devices 10a to 10f on the left side and the rows of the face recognition gate devices 10g to 10l on the right side. This passage 703 is connected to the passage 704 of the manned examination space in front of the counter 600b.

The traveler 620b shown at the bottom of FIG. 1 enters the unmanned examination space from the entrance between the partitions 700. The traveler 620c who has advanced through the aisle 703 enters the booth 701 where the traveler 620 is absent. The traveler 620d who entered the booth 701 undergoes immigration inspection using the face recognition gate device 10. Details of immigration by the face recognition gate device 10 will be described later.

The traveler 620e, who was not admitted by the immigration examination by the face recognition gate device 10 and was determined to need confirmation by the examiner 610c, leaves the booth 701g and returns to the aisle 703. The traveler 620e proceeds along the passage 703 toward the counter 600b, enters the passage 704 of the manned examination space, and lines up in front of the stop line 710. When the traveler 620f lined up at the stop line 710 is called by the examiner 610c, he proceeds to the counter 600b and presents his passport to the examiner 610c. The examiner 610c conducts immigration while confirming the information obtained by the face recognition gate device 10, the passport of the traveler 620 g, and the like. The traveler 620h who is admitted to the country by the examiner 610c passes through the immigration checkpoint from the left side of the counter 600b and enters the country.

On the other hand, when the immigration examination by the face recognition gate device 10 confirms that the traveler 620 is the holder of the passport and is admitted to the country, the gate 12 of the gate device 11 opens. The admitted traveler 620i passes through the gate 12 and exits the examination passer passage 705 outside the booth 701a. After the traveler 620i exits the booth 701 to the aisle 705, the gate 12 closes automatically.

On both the left and right sides of the booths 701 arranged in two rows, a passage 705 for passers of examination is formed by the face recognition gate device 10 and the partition 700 in each row. A traveler 620 who has been admitted by immigration by the face recognition gate device 10 enters the country through this passage 705. Specifically, the traveler 620j who has exited from the booths 701a to 701f in the left column to the aisle 705 on the left side enters the country through the immigration checkpoint between the counter 600a and the counter 600b. A traveler 620k who has left the booths 701g to 701l in the right column to the aisle 705 on the right side enters the country through the immigration checkpoint between the counter 600b and the counter 600c. If the traveler 620 who is permitted to enter the country wishes to stamp the entry stamp on the passport, he / she will have the examiners 610b and 610d stamp the entry stamp at the counters 600a and 600b adjacent to the passage 705.

In this way, in the immigration office using the immigration system, an unmanned examination space is formed by a plurality of face recognition gate devices 10. The unmanned examination space is provided with an entrance for travelers to be examined by the face recognition gate device 10 and an exit for guiding the traveler to the manned examination space. In the unmanned examination space, the gate 12 of each face recognition gate device 10 prevents the traveler from leaving the unmanned examination space except for the entrance from the entrance and the exit from the exit to the manned examination space. Traffic is restricted. A traveler 620 who has passed the immigration examination by the face recognition gate device 10 can pass through the gate 12 and proceed to enter the country through the passage 705 for those who have passed the examination. The traveler 620 who is not admitted to the country by the face recognition gate device 10 cannot pass through the gate 12, moves to the manned space, and is examined by the examiner 610c.

Next, the face authentication gate device 10 will be described with reference to FIGS. 2 to 4. In the following explanation, the right side is the right side, the left side is the left side, the side closer to the traveler is the front side, and the side far from the traveler is the back side when viewed from the traveler who uses the device facing the face recognition gate device 10. .. In the explanation of each part of the device, the side where the traveler is present is the front side (front side), and the opposite side is the back side.

FIG. 2 is an external view of the face recognition gate device 10 as viewed from the direction facing the operation display unit 110. FIG. 3 is a perspective view showing the appearance of the face recognition gate device 10. FIG. 4 is a diagram for explaining the size of the face recognition gate device 10. FIG. 4A shows a view of the face recognition gate device 10 as viewed from above. FIG. 4B shows a face recognition gate device seen from the passage 703 side of the unmanned examination space shown in FIG.

FIG. 2 shows a state in which the gate 12 formed of the three sets of panels 12a to 12c is housed inside the main body of the gate device 11. The gate 12 is open when the panels 12a to 12c are folded and stored inside the main body of the gate device 11 so that the traveler can pass through the gate 12. On the other hand, FIGS. 3 and 4 show a state in which the three sets of panels 12a to 12c are deployed outside the main body of the gate device 11. The state in which the three sets of panels 12a to 12c are expanded and the traveler cannot pass through the gate 12 is the state in which the gate 12 is closed.

As shown in FIG. 2, the face recognition gate device 10 includes a main body portion 100 including the gate device 11, a pillar portion 200 extending upward from the main body portion 100, and an indicator light portion 300 provided on the pillar portion 200. including.

In addition to the gate device 11, the main body 100 includes a partition panel 101, an operation display unit 110, a first face authentication camera 71, a second face authentication camera 72, a face authentication button 111, a call unit 130, a passport reader 400, and a storage. Includes a platform 140 and a motion sensor 120.

The partition panel 101 is provided on one side of a substantially square booth 701 as shown in FIGS. 3 and 4 (a) in order to partition the partition panel 101 from the adjacent booth 701. The upper part of the partition panel 101 is formed of a translucent resin. The semi-transparent portion has a substantially L-shaped shape when viewed from the upper surface, extending from one side partitioning the adjacent booth 701 to the upper surface of the gate device 11. The partition panel 101 is provided up to a height exceeding the upper end of the operation display unit 110.

The operation display unit 110 is arranged on the back side of the upper surface of the main body housing 102 having a substantially hexagonal column shape. The operation display unit 110 is composed of a touch panel type liquid crystal display device, and functions as a display unit that displays operation guidance and the like to the traveler and also functions as an operation unit that accepts the input operation of the traveler. The operation display unit 110 is arranged in an inclined state with the touch panel type liquid crystal screen facing diagonally upward so that the traveler can easily perform the display confirmation and the input operation.

The first face recognition camera 71 and the second face recognition camera 72 have a function of photographing a traveler in order to acquire a face image of the traveler used for face recognition processing. The first face recognition camera 71 and the second face recognition camera 72 can acquire a color visible light image of a traveler's face. The first face recognition camera 71 and the second face recognition camera are arranged in the vicinity of the operation display unit 110 that the traveler pays attention to when confirming the display content or performing an input operation. Specifically, the first face recognition camera 71 is arranged above the operation display unit 110, and the second face recognition camera 72 is arranged below the operation display unit 110. Similar to the operation display unit 110, the first face recognition camera 71 and the second face recognition camera 72 are arranged in an inclined state with the imaging range directed diagonally upward. As a result, the face of the traveler who confirms the display and performs the input operation on the operation display unit 110, which is the object of interest, can be imaged from substantially the front.

The face authentication button 111 functions as an operation unit that accepts an input operation instructing the start of the identity verification process by the traveler. The face authentication button 111 seems to indicate the start of the identity verification process to the traveler, but in the face verification process described later, a trigger (trigger) for proceeding with the process based on the result of the identity verification process. Is to be. The face recognition button 111 is made of, for example, a capacitive touch panel, and is provided on the lower side of the first face recognition camera 71. The face recognition button 111 has a long band shape in the lateral direction and can be easily operated by both right-handed and left-handed travelers.

The call unit 130 is used to have a conversation between a traveler who uses the face recognition gate device 10 and an examiner 610 of the counter 600. The call unit 130 of the face recognition gate device 10 is connected to the call unit 31 of the management terminal 30 provided on the counter 600 (see FIG. 5). The examiner 610 can use the telephone unit 31 to give instructions to the traveler and provide operation guidance. However, the method in which the examiner 610 talks with the traveler is not limited to the method using the management terminal 30. For example, an intercom may be installed at the counter 600 separately from the management terminal 30 and connected to the talking unit 130 of the face authentication gate device 10 by wire or wirelessly to be used as the talking unit 31.

The passport reading device 400 has a function of reading information from a passport in which an IC chip is embedded. Specifically, it reads the text information (MRZ information) and face image (passport photo) on the passport ticket, and also reads the face image, nationality, name, date of birth, passport number, etc. stored in the IC chip. read. The passport reading device 400 functions as an acquisition unit for acquiring information from a storage medium such as an IC chip, and the details will be described later.

The storage table 140 is used for placing a small bag or the like. The storage table 140 having a plate shape is provided along the front surface of the main body housing 102 in a state of protruding from the front surface. A motion sensor 120 is provided at a position on the front surface of the main body housing 102 and below the storage table 140. The motion sensor 120 is a sensor using infrared rays or the like, and has a function of detecting a traveler who has entered the area of the booth 701 shown by a broken line in FIG.

The pillar portion 200 has a hollow structure, and as shown in FIG. 3, the front surface excluding the light source 210 arranged at the upper portion is a half mirror (mirror surface). As shown in FIG. 2, inside the pillar portion 200 on the back side of the half mirror, a first fraud detection unit 510 including a third face recognition camera 73 and a depth sensor 511, a fourth face recognition camera 74, and a first face recognition camera 74. A thermal camera 81 and a second thermal camera 82 are provided. The light source 210 is composed of, for example, an LED, and functions as an illumination that illuminates the traveler so that the face of the traveler can be clearly imaged by the face recognition cameras 71 to 74.

In front of a mirror, a smooth surface of resin or glass, etc., where the face or figure is reflected, people tend to pay attention to the mirror surface on which they are reflected and check their hairstyle and clothes. The aim of installing the mirror surface is to guide the traveler to naturally pay attention to the mirror surface when he / she comes in front of the face recognition gate device 10, and to take an image of the front face of the traveler. For this reason, the front surface of the pillar portion 200 is a mirror surface that the traveler pays attention to, and the third face recognition camera 73 and the fourth face recognition camera 74 are arranged in the vicinity of the mirror surface portion that is the focus of attention.

The vicinity of the mirror surface portion referred to here is a peripheral portion including the mirror surface portion. As described above, in addition to arranging the third face recognition camera 73 and the fourth face recognition camera 74 on the back side of the half mirror which is the mirror surface portion, the third face recognition camera 73 and the fourth face recognition camera 74 may be arranged at a position close to the mirror surface portion on the outside of the mirror surface portion. Further, the height of the third face recognition camera 73 and the height of the fourth face recognition camera 74 are different so as to cope with the height difference of the traveler. In this way, by arranging the camera so that the position of the traveler's face is included in the imaging range at a position where it is easy to capture the traveler's face to confirm the reflection, it becomes easier to capture the traveler's face image. ing.

The first fraud detection unit 510 has a function of acquiring a still image and a moving image of a traveler and detecting the skeleton of the traveler and the movement of the traveler. The first fraud detection unit 510 has an image sensor that functions as a third face recognition camera 73, and a depth sensor 511. The first fraud detection unit 510 uses an image sensor to acquire a color visible light still image and a visible light moving image of a traveler. The depth sensor 511 includes an irradiation unit that irradiates infrared spot light, and an infrared camera that receives reflected light of each spot light. The first fraud detection unit 510 uses the depth sensor 511 to recognize the position and depth (depth) of the object from the time until the projected infrared rays are reflected and returned. As the first fraud detection unit 510 having an image sensor (third face recognition camera 73) and a depth sensor 511, for example, Kinect (registered trademark) manufactured by Microsoft Corporation is used. For example, a traveler may try to impersonate another person by using an image of another person's face. The face authentication gate device 10 uses the first fraud detection unit 510 to detect this fraud based on the fact that the face portion has no depth, that is, the face has no unevenness.

The first thermal camera 81 and the second thermal camera 82 have a function of capturing an image of a traveler and detecting the temperature. For example, a traveler may try to impersonate another person by wearing a mask in the shape of another person's face. Since the mask has irregularities similar to those of the face, there is a possibility that fraud cannot be detected only by the depth information. In such a case, the face recognition gate device 10 uses the first thermal camera 81 and the second thermal camera to detect this fraud based on the abnormal temperature of the face portion.

The third face recognition camera 73 and the fourth face recognition camera 74 have a function of capturing a traveler in order to acquire a face image of the traveler used for face recognition processing. The third face recognition camera 73 and the fourth face recognition camera 74 can acquire a color visible light image of a traveler's face.

The indicator light unit 300 includes a booth number display unit 310, a staff call display unit 320, a second fraud detection unit 520 including a first surveillance camera 91 and a depth sensor 521, and a second surveillance camera 92. The indicator light unit 300 further has a state display unit 330 on the back side indicating the state of the face authentication gate device 10 by using a plurality of lamps (see FIGS. 4A and 15), but details thereof. Will be described later.

As shown in FIG. 2, the booth number display unit 310 and a part of the lamps constituting the staff call display unit 320 are arranged in front of the indicator light unit 300. The second fraud detection unit 520 is arranged on the front side of the lower surface of the indicator light unit 300. The second surveillance camera 92 is arranged on the lower surface of the arm portion 340 extending from the right side surface of the indicator light portion 300 to the right front side.

The booth number display unit 310 has a function of displaying an identification number for distinguishing each of the plurality of booths 701 shown in FIG. The identification number is also used as an identification number for identifying each face recognition gate device 10. Of the indicator light portion 300 shown in FIG. 2, the portion from the lower surface to the upper surface of the arm portion 340 is fixed to the pillar portion 200. On the other hand, the booth number display unit 310 is rotatably provided. The booth number display unit 310 can be rotated in a direction that is easy for the traveler to see, depending on the orientation when the face recognition gate device 10 is installed.

The clerk call display unit 320 is used to call the airport clerk 611 waiting as a concierge in the unmanned examination space as shown in FIG. As shown in FIG. 3, the clerk call display unit 320 includes a horizontally long band-shaped lamp provided on the side surface of the indicator light unit 300 and a humanoid lamp provided on the front surface of the indicator light unit 300. The two lamps of the clerk call display unit 320 are normally off and turn on when calling the airport clerk 611. The face recognition gate device 10 has a function of detecting a predetermined operation performed by a traveler who has started immigration. For example, if the time from one operation to the next exceeds a preset predetermined time, the face recognition gate device 10 that detects this determines that it is necessary to assist the traveler's operation. The staff call display unit 320 is turned on. When the clerk call display unit 320 lights up, the waiting airport clerk 611 heads for the face recognition gate device 10 and assists the traveler's operation. When the airport clerk 611 performs a predetermined operation on the operation display unit 110, the clerk call display unit 320 is turned off.

The second fraud detection unit 520 has a function of acquiring a still image and a moving image of the traveler and detecting the skeleton of the traveler and the movement of the traveler. The second fraud detection unit 520 has an image sensor that functions as the first surveillance camera 91 and a depth sensor 521. The second fraud detection unit 520 is arranged on the lower surface of the indicator light unit 300 in an inclined state in which the image pickup range by the image sensor and the depth detection range by the depth sensor 521 are directed diagonally forward. As a result, the second fraud detection unit 520 can detect the skeleton and movement of the traveler in the booth 701. The second fraud detection unit 520 has the same configuration and function as the first fraud detection unit 510. Specifically, an image sensor is used to acquire a color visible light still image and a visible light moving image of a traveler, and a depth sensor 521 is used to detect the skeleton and movement of the traveler. .. For example, a traveler may move beyond the gate 12 even though the gate 12 is not open. Further, for example, when one traveler is permitted to enter the country and the gate 12 is opened, this traveler may pass through the gate 12 together with another traveler who is not permitted to enter the country. The face recognition gate device 10 uses the second fraud detection unit 520 to detect such fraud based on the movement of the traveler, the number of people passing through the gate 12, and the like.

The second surveillance camera 92 has a function of photographing the inside of the booth 701. The second surveillance camera 92 is used to monitor the immigration office. The second surveillance camera 92 is arranged on the lower surface of the tip of the arm portion 340 in an inclined state with the imaging range directed diagonally forward. As shown in FIG. 4A, the arm portion 340 extends in the right direction substantially parallel to the front surface of the indicator light portion 300, and then extends in a direction substantially parallel to the gate 12 at different angles. The second surveillance camera 92 captures an image of a traveler facing the operation display unit 110 from diagonally above on the right side.

As shown in FIG. 4A, the operation display unit 110 is arranged so that the screen is substantially parallel to one of the diagonal lines of the substantially square booth 701. In other words, the operation display unit 110 is arranged on one corner side of the booth 701, which is substantially square when viewed from above, with the screen slanted with respect to each side of the booth 701. A passport reading device 400 is arranged on the upper surface of the main body housing 102 on the front side of the traveler who operates the operation display unit 110.

The partition panel 101 is arranged on the left side of the booth 701 as shown in FIG. 4A. By arranging the face recognition gate devices 10 side by side as shown in FIG. 1, the partition panel 101 of the adjacent face recognition gate device 10 is arranged on the right side of the booth 701 shown in FIG. 4 (a). .. Further, a gate 12 is arranged on the upper side of FIG. 4A. The gate 12 is normally closed to prevent travelers from passing through. The traveler enters the booth 701 in which the left, right, and upper sides of FIG. 4 (a) are closed from the lower side of FIG. 4 (a).

As shown in FIG. 4A, the main body housing 102 has a substantially hexagonal shape when viewed from above, with a pair of opposite corners cut out from the four corners of a square having a vertical and horizontal dimension d1 of about 500 mm. It has a shape. The booth 701 is a square whose vertical and horizontal dimensions d2 are about 1250 mm when viewed from above. The width d1 of the main body of the gate device 11 for accommodating the three sets of panels 12a to 12c is about 500 mm, which is the same as the main body housing 102 of the face authentication gate device 10, and the gate 12 when the three sets of panels 12a to 12c are deployed. The width d3 of is about 650 mm. Therefore, with the gate 12 closed, the traveler who entered the booth 701 cannot proceed to the upper side of FIG. 4A. When entry is permitted by immigration by the face recognition gate device 10, the gate device 11 stores the panels 12a to 12c inside the main body and opens the gate 12, and the traveler advances to the upper side of FIG. 4A and booths. You can get out of 701.

The height h1 from the floor surface (hatched portion) shown in FIG. 4B to the upper surface of the main body housing 102 is about 850 mm. On the other hand, the height of the gate device 11 to the upper surface of the main body is about 900 mm, and the gate 12 having a width d3 at the time of deployment of about 650 mm can be folded and stored inside the main body. The height h2 from the floor surface to the upper end of the face recognition gate device 10 is about 2550 mm, and various information displayed on the indicator light unit 300 can be confirmed from a distant place.

The face recognition gate device 10 includes a first face recognition camera 71 and a second face recognition camera 72 arranged in the vicinity of the operation display unit 110 of the main body 100, and a third face recognition camera 73 and a third face recognition camera 73 arranged on the pillar 200. A face image for face recognition processing is acquired by using a total of four cameras of the four face recognition cameras 74. The height c1 from the floor to the first face recognition camera 71 is about 1070 mm, the height c2 to the second face recognition camera 72 is about 1290 mm, the height c3 to the third face recognition camera 73 is about 1520 mm, and the fourth. The height c4 up to the face recognition camera 74 is about 1760 mm. By arranging the four cameras 71 to 74 at different heights, it is possible to capture a face image of a short traveler and a face image of a tall traveler. It is also possible to capture facial images of travelers using wheelchairs.

FIG. 5 is a block diagram showing a configuration example of an immigration system including a face authentication gate device 10. As shown in FIG. 5, the immigration system includes a face recognition gate device 10, a management server 20, a management terminal 30, and a monitoring terminal 40. The face recognition gate device 10, the management server 20, the management terminal 30, and the monitoring terminal 40 are connected to each other so as to be able to communicate with each other via a network 50 such as a LAN. FIG. 5 shows one face recognition gate device 10, a management server 20, a management terminal 30, and a monitoring terminal 40 as the minimum configuration of the immigration system, but the number is not particularly limited.

The face recognition gate device 10 includes, in addition to the constituent parts described with reference to FIGS. 2 to 4, a control unit 13 for controlling each of these parts, a storage unit 14 for storing various information, and an external device. It has a communication unit 15 for performing data communication between the two. For example, the control unit 13, the storage unit 14, and the communication unit 15 are configured by a computer device.

The control unit 13 controls each unit constituting the face recognition gate device 10. The functions and operations of the face recognition gate device 10 described in the present embodiment are realized by the control of each unit by the control unit 13. The control unit 13 is a face detection processing unit that detects a face from an image obtained by imaging a traveler with face authentication cameras 71 to 74 and cuts out an image of the face portion, and is an IC chip of a passport by a passport reader 400. It also functions as a face authentication processing unit that determines whether or not the face image read from the face image matches the face image obtained by imaging the traveler with the face authentication cameras 71 to 74 by the face authentication process. However, the face recognition gate device 10 may have a face detection processing unit and a face recognition processing unit in addition to the control unit 13.

The storage unit 14 is composed of a non-volatile memory such as a semiconductor memory or a hard disk. The storage unit 14 is used to store various data such as programs and data necessary for the operation of the control unit 13, data necessary for the operation of the face authentication gate device 10, images and information obtained by the face authentication gate device 10. To. The communication unit 15 transmits / receives data to / from the management server 20, the management terminal 30, and the monitoring terminal 40 via a network 50 such as a LAN.

The management server 20 is configured by, for example, a computer device. The management server 20 obtains information necessary for immigration by the face recognition gate device 10, information obtained from the IC chip and the face of the passport by the face recognition gate device 10, an image of a traveler captured by the face recognition gate device 10, and the like. It has a function to save and manage.

For example, the management server 20 uses information such as a traveler's face image, nationality, name, date of birth, passport number, examination result, etc. obtained by all face recognition gate devices 10 as an examination history. I manage it.

Further, for example, the management server 20 manages information such as a face image, nationality, name, date of birth, passport number, etc. of a person who needs to be confirmed by the examiner 610 at the time of immigration as a blacklist. If the information acquired by the face recognition gate device 10 from the passport of the traveler or the image captured by the traveler matches the information registered in the black list, the face recognition gate device 10 sets the gate 12 to the gate 12. Guide travelers to be examined by examiner 610 without opening. The blacklist is not limited to the mode managed by the management server 20, and may be, for example, a mode managed by another server device connected to the network 50. Further, the process of collating the information obtained by the face authentication gate device 10 with the information registered in the black list may be executed by the face authentication gate device 10 or by the management server 20. It may be an embodiment.

The management terminal 30 is composed of, for example, a computer device. The management terminal 30 has a function of displaying various information acquired from the face authentication gate device 10 and the management server 20 on the screen. The management terminal 30 is installed in the counter 600 as shown in FIG. The examiner 610 manages a plurality of face recognition gate devices 10 by using the management terminal 30 assigned to each person. In addition, the examiner 610 uses the management terminal 30 to perform immigration inspection for travelers.

The face recognition gate device 10 to be managed by one management terminal 30, that is, one examiner 610 can be set in advance. For example, the six face recognition gate devices 10a to 10f forming the left column shown in FIG. 1 are preset as management targets of the management terminal 30b used by the examiner 610b of the counter 600a. Further, the six face recognition gate devices 10g to 10l forming the right column are preset as management targets of the management terminal 30d used by the examiner 610d of the counter 600b.

FIG. 6 is a diagram showing an example of a screen displayed on the management terminal 30. As shown in FIG. 6, information on all six face recognition gate devices 10a to 10f to be managed is displayed on the screen of the management terminal 30b used by the examiner 610b shown in FIG. 1. Similarly, information on all six face recognition gate devices 10f to 10l managed by the examiner 610d is displayed on the screen of the management terminal 30d used by the examiner 610d.

The management terminal 30 displays the information acquired from each face authentication gate device 10 on the screen. As shown in FIG. 6, the information of one face recognition gate device 10 is displayed in one frame 901. In the upper part of the frame 901, the examination status, status, passport number and confirmation items are displayed as examination information.

In the examination status on the screen, information indicating which process is in the plurality of processes at the time of immigration examination is displayed. Immigration by the face recognition gate device 10 is performed by detecting a traveler who has entered the booth 701, acquiring information from a passport, collating with a blacklist, verifying the identity by face recognition processing, determining whether or not to enter the country, and determining whether the gate 12 is based on the judgment result. It is done in the process of opening or guiding to manned examination. Information indicating which of these processes is in is displayed as the examination status on the screen of the management terminal 30.

For example, when a traveler who has entered the booth 701 is detected by the motion sensor 120, "presence detection" is displayed in the examination status. When information acquisition is started from the IC chip of the passport and the face of the passport by the passport reading device 400, "P reading" is displayed in the examination status. When the information from the passport is acquired and the identity verification by the face recognition process is started, "face recognition" is displayed in the examination status. When opening the gate 12 for a traveler who is permitted to enter the country based on the result of identity verification, "Gate open" is displayed in the examination status.

In the passport number on the screen, the passport number obtained from the passport of the traveler by the passport reading device 400 is displayed. When the passport reading device 400 acquires the passport number, the display of the passport number on the screen is updated from "None" to the display of the acquired passport number.

Information indicating the processing status is displayed in the status on the screen. For example, after detecting a traveler who has entered the booth 701, "Waiting" is displayed while waiting for information acquisition from the passport. Further, for example, while the identity verification by the face authentication process is started and the face image of the traveler acquired by the face authentication cameras 71 to 74 and the face image acquired from the IC chip of the passport are compared and collated, " "Matching" is displayed.

Information to be notified to the examiner 610 is displayed in the confirmation items on the screen. For example, if the face image acquired from the IC chip of the passport and the face image acquired by the face recognition cameras 71 to 74 do not match, "face recognition NG" is displayed. Further, for example, when the information such as the passport number acquired from the IC chip of the passport matches the information registered in the blacklist, "blacklist" is displayed.

As shown in FIG. 6, the face image acquired by the passport reading device 400 from the IC chip of the passport is displayed as the IC face image under the examination information. Further, the face image acquired by photographing the traveler with the face recognition cameras 71 to 74 is displayed as the captured face image. Hereinafter, the face image acquired from the IC chip of the passport is described as "IC face image", and the face image acquired by imaging the traveler with the face recognition cameras 71 to 74 is described as "captured face image" to distinguish them. ..

The display of the captured face image on the screen is updated with a new face image every time a new face image is acquired by the face recognition cameras 71 to 74. When a plurality of face images can be acquired by the plurality of face recognition cameras 71 to 74, one face image selected from the obtained plurality of face images is displayed. For example, if the priority is set in advance for the plurality of face recognition cameras 71 to 74, the captured face image selected according to the priority is displayed. In addition, it is also possible to select one in advance from the plurality of face recognition cameras 71 to 74 and set to always display the image captured by the selected face recognition camera.

As shown in FIG. 6, an image obtained by the passport reader 400 capturing the inner face of the passport of the traveler is displayed as a passport image below the IC face image and the captured face image. Further, below that, an image of a traveler captured by the surveillance camera 91 is displayed as a surveillance camera image. The display of the surveillance camera image is updated with the newly captured image at a predetermined time interval. The surveillance camera image is not limited to the mode of using the image acquired by the first surveillance camera 91, but may be the mode of using the image acquired by the second surveillance camera 92.

Normally, the two examiners 610 manage the six face recognition gate devices 10 in charge while checking the screen display shown in FIG. 6, respectively. If necessary, one examiner 610 can manage all 12 face recognition gate devices 10 by performing a predetermined operation on the management terminal 30. For example, when the management terminal 30d used by the examiner 610d breaks down, the examiner 610b operates the management terminal 30b to control the terminal that manages the face authentication gate devices 10g to 10l from the management terminal 30d. Change to 30b. As a result, the screen display of the management terminal 30b, which had previously displayed the information of the six face recognition gate devices 10a to 10f, is switched, and the information of the twelve face recognition gate devices 10a to 10l is displayed on the screen. To.

FIG. 7 is a diagram showing an example of a screen displayed when the face authentication gate device 10 to be managed is changed. When the operation of changing the number of face recognition gate devices 10 to be managed from 6 to 12 is performed, the screen of the management terminal 30b displaying the information for 6 as shown in FIG. 6 is displayed in FIG. 7. As shown in, it is divided into two left and right regions 903 and 904. In the area 903 on the left side, information on the six face recognition gate devices 10a to 10f managed by the management terminal 30b is displayed. In the area 904 on the right side, information on the six face recognition gate devices 10g to 10l newly added to the management target of the management terminal 30b is displayed. By dividing the screen into two areas, the width of the frame 902 for displaying the information of one face recognition gate device 10 becomes half of the frame 901 shown in FIG. The management terminal 30 displays the same information as in the frame 901 in the frame 902 by changing the arrangement of each information and reducing and displaying at least a part of the information.

It is also possible to exclude the six face recognition gate devices 10g to 10l from the management target by performing a predetermined operation on the management terminal 30b with the information for 12 devices displayed on the screen. When the operation of returning the management target to the original six units is performed, the display on the screen of the management terminal 30b returns from the display mode shown in FIG. 7 to the display mode shown in FIG.

The operation of newly adding the face authentication gate device 10 managed by another management terminal 30 to the management target or removing it from the management target again can be easily executed by pressing a predetermined key on the keyboard provided with the management terminal 30. can do. Specifically, for example, each time a predetermined key is pressed on the management terminal 30b that manages six face recognition gate devices 10a to 10f for one row, between the screen display shown in FIG. 6 and the screen display shown in FIG. The display switches.

The switching of the face authentication gate device 10 to be managed is performed for the face authentication gate device 10 managed by each management terminal 30 among a plurality of management terminals 30 that manage the same unmanned examination space. For example, it is assumed that the 12 face recognition gate devices 10 are divided into three groups A to C by four, and each group is managed by three management terminals 30. In this case, on the management terminal 30 that displays and manages the information of the four face recognition gate devices 10 of the group A, each time the predetermined key is pressed, the display on the screen is the information display of the eight units of the groups A and B. , The information display of 8 units of groups A and C, the information display of 12 units of groups A to C, and the screen display are switched in order. Then, when the predetermined key is pressed again, the information display of the four group A units is restored. In the management terminal 30 to which the face recognition gate device 10 is not assigned, one group, two groups, or all three groups are selected from the groups A, B, and C, and the information of the selected group is input. It can be displayed on the screen.

In this way, the examiner 610 can switch the screen display of the management terminal 30 to check the status of the face recognition gate device 10 managed by another management terminal 30. For example, when the management terminal 30 breaks down or the examiner 610 in charge of the management terminal 30 leaves the seat, the face recognition gate device 10 managed by the management terminal 30 is added to the management target of another management terminal 30. Therefore, the use of all face recognition gate devices 10 can be continued.

When the IC face image acquired from the passport of the traveler by the face authentication gate device 10 and the captured face image acquired by imaging the traveler by the face authentication cameras 71 to 74 do not match, the management terminal 30 is shown in the figure. "Face recognition NG" is displayed in the confirmation items on the screen shown in 6. The management terminal 30 notifies the examiner 610 of the occurrence of the confirmation item by blinking the display of the confirmation item or emitting a predetermined notification sound. The examiner 610 who received the notification can perform a predetermined operation on the management terminal 30 to enlarge and display the information of the face authentication gate device 10 displayed as "face authentication NG".

FIG. 8 is a diagram showing a screen example of the management terminal 30 in which the information of the traveler for whom the confirmation item has occurred is enlarged and displayed. When the enlarged display is performed, the screen of the management terminal 30 is divided into two left and right areas 905 and 906. In the area 905 on the left side, surveillance camera images of all face recognition gate devices 10 managed by the management terminal 30 are displayed. The management terminal 30 displays the information of all the face recognition gate devices 10 in the area 905 by limiting the types of information to be displayed.

In the frame 907 of the right side area 906, the information of the face recognition gate device 10 in which the confirmation item has occurred is enlarged and displayed. Specifically, the examination status, status, passport number, confirmation items, IC face image, past face image, and captured face image of the face recognition gate device 10 that has become "face recognition NG" are displayed. Further, the management terminal 30 displays a gate opening button and a manned examination button in the frame 907. The information of another face recognition gate device 10 can be enlarged and displayed at the same time in the lower area 908 of the right side area 906 of the screen.

The management terminal 30 displays the IC face image and the captured face image on the screen in a state enlarged as compared with the normal display shown in FIG. Further, the management terminal 30 displays the past face image in the same size as the IC face image and the captured face image side by side. The past face image is a face image of a traveler obtained when the person is confirmed by immigration, immigration, etc. conducted in the past. The face image of the traveler acquired in the past is managed by the management server 20 in a state of being associated with the passport number and the like of the traveler. When the management terminal 30 enlarges and displays the information of the face authentication gate device 10 in which the confirmation item has occurred, the management terminal 30 inquires the management server 20 for the information based on the passport number. Then, if the past face image is recorded, it is acquired and displayed on the screen.

The examiner 610 compares the past facial image and the captured facial image when the identity is confirmed in the past. Then, when the traveler using the face recognition gate device 10 is the passport holder himself / herself and it is determined that entry is permitted, the examiner 610 presses the gate open button on the screen shown in FIG. I do. In response to this operation, the management terminal 30 transmits a command signal instructing the opening of the gate 12 to the face authentication gate device 10. After transmitting the command signal, the management terminal 30 cancels the enlarged display and displays the normal screen shown in FIG. The face recognition gate device 10 that has received the command signal controls the gate device 11 to open the gate 12. As a result, the traveler can enter the country through the gate 12.

On the other hand, if it is determined that entry is not permitted, the examiner 610 performs an operation of pressing the manned examination button on the screen shown in FIG. In response to this operation, the management terminal 30 transmits a command signal for instructing guidance to the manned examination to the face recognition gate device 10. After transmitting the command signal, the management terminal 30 cancels the enlarged display and displays the normal screen shown in FIG. Upon receiving the command signal, the face recognition gate device 10 displays information instructing the counter 600 to proceed to the examination by the examiner 610 on the screen of the operation display unit 110 with the gate 12 closed, and displays the traveler. invite. The traveler will pass through the unmanned examination space and proceed to the counter 600 to be examined by the examiner 610.

As shown in FIG. 8, after the information of the face authentication gate device 10 in which the confirmation item has occurred is enlarged and displayed, when a predetermined time elapses without operating the gate opening button and the manned examination button on the screen, the management terminal 30 Determines that a manned examination is necessary. The management terminal 30 determined to perform the manned examination cancels the enlarged display and displays the normal screen shown in FIG. Further, the face authentication gate device 10 determines that a manned examination is necessary if the command signal cannot be received from the management terminal 30 even after a predetermined time has elapsed since the confirmation item such as "face authentication NG" occurs. .. The face recognition gate device 10 determined to perform the manned examination executes a process of guiding the traveler to the counter 600 where the examiner 610 is located, as described above.

Even if a confirmation item occurs in the face recognition gate device 10, the management terminal 30 may not perform the enlarged display shown in FIG. 8 and may not request confirmation by the examiner 610. Specifically, for example, when the information acquired from the passport by the face recognition gate device 10 matches the registration information of the blacklist managed by the management server 20, the enlarged display and the confirmation by the examiner 610 are not performed. In this case, the face recognition gate device 10 determines that manned examination is necessary without waiting for the command signal from the management terminal 30. Further, it is also possible to set the face authentication gate device 10 to proceed with the determination process without waiting for the command signal from the management terminal 30 by only notifying the management terminal 30 of the occurrence of the confirmation items for all the confirmation items. can.

On the screen of the management terminal 30b of the examiner 610b in charge of the examination, the information of the travelers who are determined to require the manned examination is displayed in the order of determination. The examiner 610b examines the travelers who are determined to be subject to the manned examination in order while checking the screen display of the management terminal 30b.

FIG. 9 is a diagram showing an example of a screen displayed on the management terminal 30b for performing a manned examination. As shown in FIG. 9, on the screen of the management terminal 30b for performing the manned examination, the determination reason 909 determined to be the target of the manned examination, the IC face image and the captured face image of the traveler acquired by the face authentication gate device 10 are displayed. , The past face image acquired from the management server 20 based on the passport number, the passport image, and the related information managed by the management server 20 are displayed. The management server 20 manages past travel information, examination information, and the like in association with the passport number, in addition to the past facial image. The management terminal 30 inquires information from the management server 20 based on the passport number of the traveler, and if there is past information about the traveler, displays this as related information on the screen. FIG. 9 shows an example of a case where the face recognition process does not match. For example, if the information matches the blacklist registration information and is subject to a manned examination, the face is registered in the blacklist on the screen. Information is displayed as related information.

The I-zone image 910 cut out from each image is displayed below the IC face image, the past face image, and the captured face image shown in FIG. The I-zone image 910 is an image obtained by cutting out and enlarging an I-shaped region including the inner corner of the eye, the tip of the nose, and the lips as an I-zone from the face image. The characteristics of the I zone including the inner corner of the eye, the tip of the nose, and the lips are difficult to hide even with makeup, and even if the face looks different due to the difference in makeup, it may be possible to determine that they are the same person by comparing the I zone images 910. .. Therefore, when performing a manned examination, the management terminal 30 cuts out an I zone from each face image acquired from the face authentication gate device 10 and the management server 20 to generate an enlarged I zone image 910 and displays it on the screen. .. The examiner 610 examines the traveler while checking the information displayed on the screen. The examiner 610 can operate the management terminal 30 to select the captured face image of the traveler obtained by the face authentication gate device 10 and send it to the management server 20 together with information such as the passport number of the traveler. .. The management server 20 manages the captured face image received from the management terminal 30 as a past face image in association with information such as a passport number.

Next, the monitoring terminal 40 will be described. The monitoring terminal 40 shown in FIG. 5 is configured by, for example, a computer device. The monitoring terminal 40 is installed in another room away from the immigration office and is used by the observer to monitor the immigration office. The monitoring terminal 40 is connected to the second monitoring camera 92 provided in each face recognition gate device 10 via the network 50. The second surveillance camera 92 is directly connected to the network 50 without going through the communication unit 15.

The second surveillance camera 92 operates independently of the face recognition gate device 10. The monitoring terminal 40 acquires an image of the area including the booth 701 from diagonally above from the second monitoring camera 92 that continues to operate even after the power of the face recognition gate device 10 is turned off, and displays the image on the screen. indicate. On the screen of the surveillance terminal 40, in addition to the image captured by the second surveillance camera 92, the images of the surveillance cameras installed in various places in the airport are also displayed.

FIG. 10 is a diagram showing an example of a screen displayed on the monitoring terminal 40. The screen of the monitoring terminal 40 is divided into an enlarged display area 911 on the upper side and a list display area 912 on the lower side. In the list display area 912, the images obtained by all the surveillance cameras are listed in the order of the identification numbers assigned to the surveillance cameras.

The enlarged display area 911 is divided into four areas arranged side by side. The image of the surveillance camera satisfying a predetermined condition and the image of the manually selected surveillance camera are enlarged and displayed in the enlarged display area 911. Specifically, the surveillance terminal 40 analyzes an image obtained by the surveillance camera, executes motion detection and person detection processing, and enlarges and displays an image of the surveillance camera that has detected a predetermined operation or a predetermined person. Enlarge and display in 911. Further, when the observer who confirms the screen display of the surveillance terminal 40 performs an operation of selecting the image of the surveillance camera displayed in the list display area 912, the surveillance terminal 40 enlarges the selected image to the enlarged display area 911. Enlarge to display.

The surveillance terminal 40 displays both images so that the correspondence between the surveillance camera image enlarged and displayed in the enlarged display area 911 and the surveillance camera image listed in the list display area 912 can be easily recognized. Specifically, as shown in FIG. 10, the identification number No. The surveillance terminal 40 in which the image of the surveillance camera of No. 3 is enlarged and displayed in the frame at the left end of the enlarged display area 911 has been No. 1 until then. A graphic image such as an icon schematically showing the four frames of the enlarged display area 911 is displayed in the frame of the list display area 912 displaying the image of the surveillance camera of 3. Then, the monitoring terminal 40 displays the inside of the leftmost frame used for the enlarged display among the four frames shown in the graphic image in a color different from the inside of the other three frames.

Then No. When the image of the surveillance camera 8 is displayed in the enlarged display area 911, the monitoring terminal 40 enlarges and displays this image in the second frame from the left of the enlarged display area 911. At the same time as the monitoring terminal 40 starts the enlarged display, the monitoring terminal 40 has been No. 1 until then. A graphic image showing the four frames of the enlarged display area 911 is displayed in the frame of the list display area 912 that displayed the images of the surveillance cameras of 8. Then, of the four frames shown in the graphic image, the inside of the second frame from the left used for the enlarged display is displayed in a color different from the inside of the other three frames.

As described above, when the monitoring terminal 40 enlarges and displays one of the plurality of surveillance camera images displayed in the list display area 912 in the enlarged display area 911, the list display displaying this image until then. A graphic image schematically showing the enlarged display area 911 is displayed in the frame of the area 912. Then, on this graphic image, the position where the surveillance camera image displayed in the list display area 912 is enlarged and displayed in the enlarged display area 911 is shown. As a result, the observer can easily recognize the correspondence between the surveillance camera image enlarged and displayed in the enlarged display area 911 and the surveillance camera image displayed in the list display area 912.

Next, the passport reading device 400 will be described with reference to FIGS. 11 to 14. As shown in FIG. 5, the passport reading device 400 includes two position detection sensors 411 and 412, an image acquisition unit 420, and an IC information acquisition unit 430.

FIG. 11 is a perspective view for explaining the configuration of the passport reading device 400. FIG. 11 schematically shows a region from the upper part of the main body housing 102 of the face recognition gate device 10 to the lower part of the pillar 200. The operation display unit housing 104 arranged on the back side of the upper surface of the main body housing 102 has an inclined surface on the upper front surface. A first face recognition camera 71, an operation display unit 110, a second face recognition camera 72, and a face recognition button 111 are arranged on the inclined surface in this order from the top.

The passport mounting table 401 (401a, 401b) constituting the passport reading device 400 by the three inclined surfaces 401a, 401b, 401c combined in a hipped roof type on the upper surface of the main body housing 102 and in front of the operation display housing 104. , 401c) are formed.

The passport mounting table 401 is formed in a state where the back edge portions of the two inclined surfaces 401a and 401b combined in a mountain-shaped cross section are in contact with the front surface of the operation display unit housing 104. The two position detection sensors 411 and 412 are arranged in the lower part of the front surface of the operation display unit housing 104.

A transparent glass plate 421 is fitted in one of the inclined surfaces 401a forming the passport mounting table 401. The image acquisition unit 420 is arranged below the inclined surface 401a and inside the main body housing 102. The image acquisition unit 420 images the inner face of the passport mounted on the passport mounting table 401 in the open state. The other inclined surface 401b is made of an opaque resin. The IC information acquisition unit 430 is arranged below the inclined surface 401b and inside the main body housing 102. The IC information acquisition unit 430 acquires information from the IC chip embedded in the passport mounted on the passport mounting table 401.

FIG. 12 is a diagram showing the appearance of the passport mounting table 401. 12 (a) to 12 (c) show a top view, a front view, and a side view of the passport mounting table 401, respectively. In FIGS. 12 to 14, the front view is a view seen from the direction facing the operation display unit 110.

As shown in FIG. 12A, a top surface 401d made of a curved surface is formed at the joint portion of the two inclined surfaces 401a and 401b combined in a mountain-shaped cross section. The glass plate 421 forming the inclined surface 401a on the right side exceeds the top surface 401d and slightly reaches the inclined surface 401b. The inclined surface 401c is formed so as to smoothly connect the front side (lower side in FIG. 12A) of the two inclined surfaces 401a and 401b and the top surface 401d to the upper surface of the main body housing 102. .. The edges of the two inclined surfaces 401a and 401b and the top surface 401d on the inner side (upper side of FIG. 12A) are in contact with the front surface of the operation display unit housing 104. The height H of the passport mounting table 401 having a mountain-shaped cross section shown in FIG. 12B is about 30 mm, the width W is about 290 mm, and the angle α between the two inclined surfaces 401a and 401b is about 160 degrees. .. The depth D of the passport mounting table 401 shown in FIG. 12 (c) is about 170 mm.

As shown in FIGS. 12A and 12B, the position detection sensors 411 and 412 emit and receive detection light in the horizontal direction between the light emitting units 411a and 412a and the light receiving units 411b and 412b, respectively. It is a transmissive sensor.

FIG. 13 is a diagram showing a method of mounting a passport on the passport mounting table 401. 13 (a) shows a top view, and FIG. 13 (b) shows a schematic cross-sectional view seen from the front. Open the page of the passport 800 in which the image, name, passport number, etc. of the traveler are described, and place it on the passport mounting table 401 as shown in FIG. 13 (a). At this time, the passport 800 is placed so that the face of the inside of the passport 800 including the passport photograph, the passport number, etc. faces the glass plate 421. Further, as shown in FIG. 13B, the passport 800 is placed so that the spine portion connecting the front cover and the back cover overlaps the top portion (top surface 401d) of the passport mounting table 401 having a mountain-shaped cross section. do.

Booklet-type passports with multiple pages between the front and back covers have a back part due to the closing habit of trying to return to the closed state when placed on a desk with the front and back covers open. May emerge. The passport mounting table 401 is configured to include inclined surfaces 401a and 401b forming a mountain-shaped substantially triangular cross section. As a result, even if the passport 800 is placed and the hand is released as shown in FIG. 13B, the inner face of the ticket can be maintained in contact with the glass plate 421. The glass plate 421 reaches a region exceeding the top surface 401d of the mountain-shaped cross section, and the image acquisition unit 420 can image substantially the entire surface of the passport 800 including the passport photograph and the passport number. The passport reader 400 uses the image of the face of the passport acquired by the image acquisition unit 420 to acquire the face image of the traveler from the photograph of the face of the passport, and the nationality, name, etc. of the traveler printed on the face of the passport. Character recognition and acquisition of information. The face image of the passport or the face image obtained from the passport face is displayed on the screen of the operation display unit 110. Travelers can confirm that their passport is their own by looking at the facial image on the screen.

Setting information is prepared in advance as to which position on the inner face of the passport 800 the character recognition is performed. The passport reading device 400 executes character recognition processing on the face face image based on this setting information. The setting information may be stored in the storage unit 14 of the face authentication gate device 10, or the passport reading device 400 may be provided with a storage unit and stored in the storage unit.

On the inner face of the passport 800, a character string that is a key for acquiring information from the IC chip embedded in the passport 800 is printed. After performing character recognition processing on the face image acquired by the image acquisition unit 420 to obtain a key character string, the IC information acquisition unit 430 uses this key to obtain a face image and nationality registered in the IC chip. And so on. The IC information acquisition unit 430 functions as a non-contact type IC reader that reads information registered in the IC chip by using wireless communication technology.

FIG. 14 is a schematic diagram illustrating a method of detecting a passport by the passport reading device 400. 14 (a) shows a side view, and FIGS. 14 (b) to 14 (d) show a front view. As shown in FIG. 14A, the position detection sensor 411 is arranged so that the detection light passes slightly above the top of the passport mounting table 401. The position detection sensor 411 functions as a presence / absence detection sensor for detecting the presence / absence of the passport 800. The position detection sensor 412 is arranged above the position detection sensor 411. The position detection sensor 412 functions as a lift detection sensor that detects the passport 800 lifted from the passport mounting table 401 due to the closing habit.

When the passport mounting table 401 does not have the passport 800, as shown in FIG. 14B, both the position detection sensor 411 for presence / absence detection and the position detection sensor 412 for floating detection are in a translucent state.

When the passport 800 is placed on the passport mounting table 401, the detection light of the position detection sensor 411 for presence / absence detection is blocked by the passport 800 and becomes a light-shielded state as shown in FIG. 14 (c). At this time, if the inner face of the passport 800 is in contact with the passport mounting table 401, the position detection sensor 412 for detecting floating remains in the translucent state. When the position detection sensor 411 is in a translucent state and the position detection sensor 412 is in a light-shielding state in this way, the passport reading device 400 determines that a good passport 800 ticket face image can be acquired. Then, the passport reading device 400 starts the information acquisition process by the image acquisition unit 420 and the IC information acquisition unit 430.

When the passport 800 placed on the passport mounting table 401 floats due to the closing habit, both the position detection sensor 411 for presence / absence detection and the position detection sensor 412 for floating detection are in a light-shielded state as shown in FIG. 14 (d). Become. When the position detection sensor 411 is in a light-shielding state and the position detection sensor 412 is in a light-shielding state in this way, the passport reading device 400 determines that a good ticket face image cannot be obtained due to floating. In response to this determination, the control unit 13 of the face recognition gate device 10 displays on the screen of the operation display unit 110 information instructing the traveler to bring the face of the passport 800 into contact with the passport mounting table 401. When the traveler presses the passport 800 from above and the position detection sensor 412 for floating detection changes to a translucent state as shown in FIG. 14 (c), the passport reader 400 that recognizes this is an image acquisition unit. The inner face of the passport 800 is imaged by the 420, and information is acquired from the IC chip by the IC information acquisition unit 430.

The passport reading device 400, which acquires the face image and character information of the traveler from the face image of the passport 800 and also acquires the information from the IC chip, notifies the control unit 13 of the information indicating that the reading of the passport 800 is completed. Upon receiving the notification, the control unit 13 displays information indicating that the information acquisition from the passport 800 has been successful on the screen of the operation display unit 110. Upon receiving this display, the traveler can release the hand holding the passport 800.

The processing related to the image acquisition of the ticket face image by the image acquisition unit 420, the information acquisition from the ticket face image, and the information acquisition from the IC chip by the IC information acquisition unit 430 may be performed by the passport reading device 400, or face authentication. The control unit 13 of the gate device 10 may use the passport reading device 400.

In this way, the passport reading device 400 can acquire a good image of the face of the passport 800 even when the passport 800 has a closing habit. When the passport 800 emerges from the passport mounting table 401 due to the strong closing habit of the passport 800, the passport reading device 400 can detect this by the position detection sensor 412. Further, when the floating of the passport 800 is suppressed, the passport reading device 400 detects this by the position detection sensor 412 and images the face of the passport 800. As a result, the process of acquiring information from the passport 800 can be easily proceeded in a short time.

Next, the indicator light unit 300 will be described with reference to FIGS. 15 and 16. FIG. 15 is a perspective view showing the appearance of the indicator light unit 300 as viewed from the rear side. The indicator light unit 300 provided on the pillar portion 200 is divided into an upper portion provided with the booth number display unit 310 and a lower portion provided with the staff call display unit 320, the status display unit 330 and the arm portion 340. There is. The lower portion is fixed to the upper end of the pillar portion 200, while the booth number display portion 310 is rotatably provided with respect to the lower portion.

As shown in FIG. 15, the status display unit 330 includes a plurality of lamps of the first lamp 330a, the second lamp 330b, the third lamp 330c, and the fourth lamp 330d. The first lamp 330a to the fourth lamp 330d are composed of, for example, LEDs, and each lamp emits light in a different color. The first lamp 330a to the fourth lamp 330d have a strip shape provided long in the horizontal direction along the back surface of the indicator lamp unit 300. As a result, when the face recognition gate device 10 is installed as shown in FIG. 1, a plurality of examiners 610b to 610d can confirm the light emitting state of the state display unit 330.

The control unit 13 notifies the examiner 610 of a plurality of states by changing the combination of the light emitting states of the lamps 330a to 330d shown in FIG. FIG. 16 is a diagram for explaining an example of state display by the state display unit 330. As shown in FIG. 16, when all of the first lamp 330a to the fourth lamp 330d are turned off, it indicates that the face recognition gate device 10 is in a standby state in which the immigration of the traveler has not been performed. When the first lamp 330a is blinking, it indicates that an unauthorized passage has occurred through the gate 12 of the face recognition gate device 10. When the second lamp 330b is lit, it indicates that the face recognition gate device 10 is checking the passport of the traveler. When the third lamp 330c is blinking and the fourth lamp 330d is blinking, it indicates that the face recognition process of the traveler is being executed. In this way, the control unit 13 notifies the state by changing the lighting state of each of the lamps 330a to 330d of the status display unit 330 between extinguishing, lighting, and blinking. Further, the control unit 13 notifies the state by changing the combination of the lighting states of the plurality of lamps 330a to 330d of the state display unit 330. Thereby, as shown in FIG. 16, various states can be notified. The examiner 610 can confirm the status notification by the status display unit 330 and take appropriate measures.

Next, the flow of processing by the face authentication gate device 10 will be described with reference to FIG. Here, the process of determining whether or not the traveler can enter the country will be described by omitting the process of confirmation and determination by the examiner 610. FIG. 17 is a flowchart showing the flow of processing by the face authentication gate device 10. The control unit 13 of the face recognition gate device 10 monitors an intruder into the booth 701 using the motion sensor 120 (step S1; No). When a traveler who has entered the booth 701 is detected (step S1; Yes), the control unit 13 displays an explanation of the immigration procedure by face authentication on the operation display unit 110, and travels by the face authentication cameras 71 to 74. The process of acquiring the face image of the person is started (step S2).

Specifically, the control unit 13 displays "Please put your passport" on the operation display unit 110, and also uses the face recognition cameras 71 to 74 to take an image of the inside of the booth 701 to display the traveler. The face is detected from the captured image. If a face is detected from the captured image, the control unit 13 cuts out the face, creates an captured face image (captured face data), and stores it in the storage unit 14. While the control unit 13 is executing other processing, in parallel with this, the control unit 13 takes a picture of the traveler with the face recognition cameras 71 to 74, creates an imaged face image, and stores the image in the storage unit 14. Continue to run.

After creating the captured face image, the control unit 13 may extract a feature amount for face matching processing from the captured face image and store this feature amount (captured face data) in the storage unit 14. In this case, the process of extracting the feature amount from the captured face image at the time of the face matching process becomes unnecessary. In the following, the feature amount of the captured face image or the captured face image used for the face matching process will be referred to as captured face data. When the purpose is only the face matching process, either one of the captured face image and the feature amount of the captured face image may be stored in the storage unit 14. In the present embodiment, in order to leave the captured face image as a record of the examination result, only the captured face image and its feature amount or the captured face image are stored in the storage unit 14.

The face recognition cameras 71 to 74 are arranged in the vicinity of the operation display unit 110 for confirming the display and input operation, the pillar portion 200 in which the figure of the traveler is reflected on the half mirror, and the like, including the object of interest that the traveler pays attention to. ing. Further, the face recognition cameras 71 to 74 are arranged so that the imaging range is directed toward the face of the traveler. As a result, the face image of the traveler can be acquired with high probability.

The neighborhood including the attention target to be noticed by the traveler means a position where the face image of the traveler can be acquired by each of the face recognition cameras 71 to 74 when the traveler pays attention to the attention target. If the face image of the traveler can be acquired, the installation position is not limited to the same plane as the object of interest. In general, when face matching is performed with registered face data taken from the front, it is desirable that the face image of the traveler that can be acquired is a front face image including both eyes, nose and mouth. Further, a frontal facial image taken from within 30 degrees up, down, left and right is desirable. Therefore, it is desirable that the face recognition cameras 71 to 74 are arranged at a position where the front face image of the traveler can be acquired when the traveler pays attention to the object of interest. Furthermore, it is desirable that the orientation of the face when the traveler pays attention to the object of interest and the orientation of each face recognition camera 71 to 74 are set to be within 30 degrees.

More specifically, when confirming or operating the display, the traveler pays attention to the operation display unit 110 (display unit), which is the target of attention regardless of the height of the traveler, and turns his face. At this time, the face of the traveler seen from the operation display unit 110 side is the front face. Therefore, the face recognition cameras 71 to 72 installed in the vicinity of the operation display unit 110 can capture the front face image of the traveler. In the face recognition gate device 10, face recognition cameras 71 to 72 are installed above and below the operation display unit 110. Therefore, regardless of the height of the traveler and the vertical difference of the face of the part of the operation display unit 110 that the traveler pays attention to, at least one of the face recognition cameras 71 to 72 of the traveler. A frontal face image can be captured.

In addition, when confirming the appearance, the traveler faces the half mirror of the pillar portion 200 and turns his face to the part of the half mirror (mirror surface) where his / her face is reflected. At this time, the face of the traveler seen from the attention portion of the half mirror is a frontal face. The part of the half mirror that attracts attention differs depending on the height of the traveler, but as described above, the frontal face image does not have to be a frontal face, and there is an allowable range in the vertical angle. Therefore, at least one of the cameras can capture a frontal facial image within an allowable range. In the face recognition gate device 10, face recognition cameras 73 to 74 are vertically installed on the back side of the half mirror. Therefore, regardless of the height of the traveler, the front face image of the traveler can be captured by at least one of the face recognition cameras 73 to 74. Although the half mirror of the pillar portion 200 is a plane mirror, it is also possible to reduce the range of the mirror surface as a gentle convex mirror.

Although examples of the operation display unit 110 and the half mirror of the pillar portion 200 which is a mirror surface are given as the objects of interest, the objects of interest are not limited to these as long as they are of interest to the traveler. If a plurality of objects of interest are provided and a face recognition camera is installed in each, there is a high possibility that the front face image of the person to be authenticated can be captured. For example, a changing display such as a blinking display may be performed around a face recognition camera installed at an arbitrary position to be a target of attention, and the traveler may pay attention to this changing display. However, in order to shorten the time required for a series of processes including face recognition, focus on those that the traveler naturally pays attention to in the series of processes, and face recognition cameras 71 to 74 in the vicinity. It is desirable to install.

In particular, in a series of processes including face recognition, as will be described later, before the traveler places the passport on the passport mounting table 401 and the IC information acquisition unit 430 acquires the IC face image from the IC chip embedded in the passport. In addition, it is desirable to focus on things that travelers pay attention to naturally without being particularly aware of it. For example, the above-mentioned operation display unit 110 and the pillar unit 200 which is a mirror surface. This is because the traveler first confirms the display of the operation display unit 110, and confirms the face on the mirror surface of the pillar portion 200 prior to the face authentication. In addition, the passport mounting table 401, which is the focus of attention when placing the passport, can also be the subject of attention.

The control unit 13 captures the face image of the traveler with the face recognition cameras 71 to 74, and continuously executes the process of storing the captured face data in the storage unit 14, while the passport is placed on the passport mounting table 401. It is monitored whether or not it is present (step S3; No). When the position detection sensor 411 detects that the passport is placed on the passport mounting table 401 (step S3; Yes), the control unit 13 determines whether the passport placed on the passport mounting table 401 is abnormal (step S4). ). An abnormality in the passport is a state in which information cannot be normally obtained from the passport.

Specifically, for example, when the position detection sensor 412 detects the floating of the passport, the control unit 13 determines that the passport has an abnormality (step S4; Yes). In this case, the control unit 13 displays information instructing the traveler to bring the passport into close contact with the passport mounting table 401 on the screen of the operation display unit 110 (step S5).

Even if the image of the traveler (passport photograph) cannot be acquired from the image of the passport face image captured by the image acquisition unit 420 of the passport reader 400, the control unit 13 determines that there is an abnormality (step S4; Yes). In addition, even if the key used to acquire the information of the IC chip, which is set in advance to be acquired from the passport image, and the character information (MRZ information) including the nationality, name, etc. of the traveler cannot be acquired from the ticket image. The control unit 13 determines that there is an abnormality (step S4; Yes). In this case, for example, the passport may be placed in the wrong orientation. Therefore, the control unit 13 confirms that there is no error in the orientation of the passport, and if there is an error in the orientation, displays information instructing the traveler to reposition the passport in the correct orientation on the screen of the operation display unit 110 ( Step S5).

In addition, if it is not possible to obtain textual information such as the traveler's name from the passport's face image, it is possible that the cause is the passport cover attached to the passport by the traveler. For example, in a passport issued in Japan, since the character information is at the end of the two-page spread, the inside of the passport cover may hide the character information. Therefore, the control unit 13 displays on the screen of the operation display unit 110 information instructing the traveler to remove the cover, if the passport is attached, when the text information cannot be acquired from the face image of the passport. (Step S5).

When the abnormality in the passport is resolved (step S4; No), the control unit 13 acquires information from the passport by the passport reading device 400 (step S6). Specifically, the image acquisition unit 420 captures the inner face of the passport to acquire the face image, and also performs character recognition processing on the face image to perform characters such as the nationality, name, passport number, and date of birth of the traveler. Get information. In addition, using the key acquired from the face image by character recognition processing, the IC information acquisition unit 430 can obtain information such as the IC face image, nationality, name, date of birth, and passport number from the IC chip embedded in the passport. get. The control unit 13 displays the image of the passport face image or the image of the passport photograph acquired from the passport ticket face image on the operation display unit 110. This display screen confirms that the passport is his or her own.

The control unit 13 determines whether or not the traveler is subject to the manned examination based on the information acquired from the IC chip of the passport and the information acquired from the face image of the passport (step S7).

Specifically, for example, the name and passport number obtained from the IC chip of the passport, and the name and passport number obtained from the face image of the passport that match the registration information of the blacklist managed by the management server 20. If there is, the control unit 13 determines that it is the target of the manned examination (step S7; Yes). Further, for example, if the information such as the IC face image, nationality, name, and date of birth obtained from the IC chip of the passport does not match the information on the passport face image, the passport may be forged. Therefore, the control unit 13 determines that it is the target of the manned examination (step S7; Yes). Further, even if there is information that cannot be obtained from the passport even if steps S4 and S5 are repeated a predetermined number of times, the control unit 13 determines that a manned examination is necessary (step S7; Yes).

The control unit 13 determined to be the target of the manned examination displays information instructing the traveler to proceed to the counter 600b and undergo the examination by the examiner 610c on the screen of the operation display unit 110 (step S8). , The immigration examination by the face recognition gate device 10 is completed. In response to this, the traveler will undergo a manned examination by the examiner 610c at the counter 600b.

For travelers who are determined not to be subject to manned examination by the determination based on the information obtained from the passport (step S7; No), the identity verification process by face verification is performed (step S9). Specifically, the control unit 13 compares the IC face image acquired from the passport with each of the plurality of captured face data created before acquiring the IC face image and stored in the storage unit 14, and whether they match. Executes face matching processing to determine whether or not.

In the face matching process that starts first after acquiring the IC face image, an image that matches the IC face image is included in the plurality of captured face data created before the acquisition of the IC face image and stored in the storage unit 14. In the process of confirming the presence or absence, the IC face image is compared with each of the plurality of captured face data. Specifically, the face matching processing unit included in the control unit 13 extracts the feature amount from the IC face image, and compares the feature amount of the IC face image with the feature amount of the captured face image which is the captured face data. When the similarity is larger than a predetermined threshold value, it is determined that the IC face image and the captured face data match. Further, when the captured face data is the captured face image, the face matching processing unit extracts the feature amount of the captured face image prior to the face matching processing. In FIG. 17, for convenience, the face matching process is shown as step S8 after the determination process in step S7, but in reality, the IC face image is acquired in step S6 and immediately in parallel with the determination process. The face matching process is started.

The control unit 13 determines whether or not the traveler is the holder of the passport based on the result of the face matching process performed on the captured face data read from the storage unit 14 and the IC face image (). Step S10). If the IC face image does not match the captured face data, the control unit 13 does not determine that the traveler is the passport holder (step S10; No), and subsequently, the motion sensor 120 detects the traveler. It is determined whether or not a predetermined time has elapsed since the detection (step S11).

If the predetermined time has not elapsed (step S11; No), the control unit 13 subsequently determines whether or not the input operation of pressing the face authentication button 111 by the traveler has been performed (step S12). If the input operation of pressing the face authentication button 111 has not yet been performed (step S12; No), the control unit 13 returns to step S9 and another captured face data and IC face image stored in the storage unit 14. Performs face matching processing with.

In this way, the control unit 13 repeatedly executes the face matching process of reading the captured face data stored in the storage unit 14 one by one and comparing them with the IC face image. The process of capturing an image of a traveler with the face recognition cameras 71 to 74 and storing the captured face data in the storage unit 14 is continuously performed in parallel with other processes. In the face matching process for comparing the captured face data stored in the storage unit 14 with the IC face image, if the IC face image does not match the captured face data, the storage unit 14 compares the captured face image with the IC face image. It is executed repeatedly while there is no captured face data.

When the face matching process is completed between all the captured face data stored in the storage unit 14 and the IC face image, the control unit 13 acquires the IC face image and the IC face image, and then the face recognition cameras 71 to 74. The face matching process is continued with the captured face image newly obtained and created in. Specifically, each time the captured face image is obtained by the face recognition cameras 71 to 74, the face matching processing unit included in the control unit 13 extracts the feature amount from the obtained captured face image, and the IC already extracted. Performs face matching processing with the feature amount of the face image.

If the captured face data matching the IC face image is obtained while the face matching process is being continued, the control unit 13 determines that the traveler is the owner of the passport (step S10; Yes). Subsequently, the control unit 13 determines whether or not the input operation of pressing the face authentication button 111 to instruct the start of identity verification has been performed by the traveler (step S15).

If the traveler has already pressed the face recognition button 111 (step S15; Yes), the control unit 13 controls the gate device 11 to open the gate 12 (step S16). The control unit 13 displays information indicating that entry is permitted on the screen of the operation display unit 110, notifies the traveler of the examination result (step S17), and ends the process.

On the other hand, if the traveler has not yet pressed the face authentication button 111, the control unit 13 continues to monitor the face authentication button 111 (step S15; No). Then, when the traveler performs an input operation of pressing the face authentication button 111 (step S15; Yes), the control unit 13 that detects this opens the gate 12 (step S16) and informs the traveler. Notification (step S17) is performed to end the process.

A traveler who has been notified of an entry permit enters the country through an open gate 12. After the traveler has passed through the gate 12, the control unit 13 closes the gate 12. In this way, the control unit 13 gates until the traveler performs an input operation of pressing the face authentication button 111 after obtaining the result that the traveler is the holder of the passport by the face matching process. 12 is not opened and the result is not notified.

When a predetermined time elapses after the traveler is detected by the motion sensor 120 (step S11; Yes) without obtaining the captured face data matching the IC face image (step S10; No), the control unit 13 determines. Judge that manned examination is necessary. That is, the control unit 13 ends the face matching process as a time-out. Subsequently, the control unit 13 determines whether or not the input operation of pressing the face authentication button 111 by the traveler has been performed (step S18).

If the traveler has already performed an input operation of pressing the face authentication button 111 (step S18; Yes), the control unit 13 operates information instructing the traveler to proceed to the counter 600b and be examined by the examiner 610c. It is displayed on the screen of the display unit 110 (step S19), and the process is terminated. On the other hand, if the traveler has not yet pressed the face authentication button 111, the control unit 13 continues to monitor the face authentication button 111 (step S18; No). Then, when the traveler performs an input operation of pressing the face authentication button 111 (step S18; Yes), the control unit 13 that detects this performs a guidance display instructing the manned examination (step). S19), the process is terminated. A traveler who is instructed to perform a manned examination will leave the booth 701, proceed to the counter 600b, and be examined by the examiner 610c.

The captured face data that matches the IC face image cannot be obtained (step S10; No), and before a predetermined time elapses after the traveler is detected by the motion sensor 120 (step S11; No), the person travels. When a person performs an input operation of pressing the face recognition button 111 (step S12; Yes), the control unit 13 provides guidance to the traveler in order to acquire good captured face data suitable for face recognition processing. It is determined whether or not it needs to be performed (step S13). When it is determined that the guidance is necessary (step S13; Yes), the control unit 13 displays the guidance on the screen of the operation display unit 110 (step S14). When it is determined that the guidance to the traveler is unnecessary (step S13; No), the control unit 13 returns to step S9 and continues the process.

For example, when the face collation process performed earlier compares the captured face data with the IC face image, the result is that the difference is large only in the mouth part and it cannot be determined that the person is the same person. The traveler may be wearing a mask. Therefore, the control unit 13 determines that guidance to the traveler is necessary (step S13; Yes), and displays information instructing the removal of the mask or the like on the screen of the operation display unit 110 (step S14). ).

Further, for example, if there is a large difference between the captured face data and the IC face image only in the parts of both eyes and it is not possible to determine that they are the same person, the traveler is wearing sunglasses or the like. there is a possibility. Therefore, the control unit 13 determines that guidance to the traveler is necessary (step S13; Yes), and displays information instructing the sunglasses and the like to be removed on the screen of the operation display unit 110 (step S14). ).

As described above, the control unit 13 needs to guide the traveler when the region where the difference between the captured face data and the IC face image is large is concentrated on a part of the face during the face matching process. Is determined. Then, in order to acquire the captured face data suitable for the face recognition process, the guidance display to the traveler is executed. This may, for example, instruct you to take off your hat if wearing a hat may be the cause of the face matching mismatch, or the bangs on your forehead may be the cause of the face matching mismatch. You can also instruct them to raise their bangs by hand.

The process of acquiring the captured face data by the face recognition cameras 71 to 74 and saving it in the storage unit 14 started by detecting the traveler with the motion sensor 120 continues even after the result of the identity verification by the face verification process is obtained. It is done. The process of accumulating the captured face data in the storage unit 14 ends at a predetermined timing. Specifically, for example, at the timing of permitting the traveler to enter the country and opening the gate 12 (step S16) or the timing of instructing the traveler to perform the manned examination (steps S8 and S19), the control unit 13 is newly added. The process of acquiring the captured face data is terminated. All the captured face data acquired by the face recognition cameras 71 to 74 during the immigration examination of the traveler is stored in the storage unit 14. The control unit 13 that has finished acquiring the captured face data selects the captured face data having the highest score indicating the degree of similarity to the IC face image from the captured face data stored in the storage unit 14. The captured face image related to the selected captured face data is sent from the face recognition gate device 10 to the management server 20 together with information such as the name and passport number obtained from the passport of the traveler, the examination result, and the time required for the examination. Will be sent. The management server 20 manages the information received from the face authentication gate device 10 as information about the traveler in association with the passport number.

As described above, when the face authentication gate device 10 detects the traveler who has entered the booth 701 by the motion sensor 120, the face authentication gate device 10 acquires and stores the face image of the traveler by using the four face authentication cameras 71 to 74. The process of saving in the unit 14 is started. Then, this process is continuously executed while other processes are performed, and the imaged face data is accumulated in the storage unit 14. When the IC face image is acquired by the passport reading device 400, the face authentication gate device 10 starts a face matching process for comparing each of the plurality of captured face data stored in the storage unit 14 with the IC face image. As a result, the processing result can be obtained earlier than in the case where the face image is acquired and the face matching process is performed after receiving the operation of the face authentication button 111. When the traveler proceeds with the operation while checking the operation guide and the like displayed on the operation display unit 110 and presses the face authentication button 111 to instruct the start of the identity verification, the traveler has already performed the face verification process. It may have been confirmed. In this case, the traveler can open the gate 12 and pass the traveler immediately after performing an input operation of pressing the face authentication button 111.

The face authentication gate device 10 can also control the timing at which the gate 12 is opened to allow the traveler to pass through when the identity verification result by the face verification process is obtained in a short time. Specifically, for example, when the traveler presses the face authentication button 111 15 seconds after the traveler is detected by the motion sensor 120, it has already been confirmed that the traveler is the passport holder. In some cases, the gate 12 can be set to open after waiting for 20 seconds from the detection by the motion sensor 120. If 20 seconds have passed when the face recognition button 111 is pressed, the gate 12 is immediately opened.

In this way, in the face recognition gate device 10, the shortest time from the detection by the motion sensor 120 to the opening of the gate 12 can be set. When the shortest time is set, the control unit 13 waits until the shortest time elapses and opens the gate 12, even if the gate 12 can be opened before the shortest time elapses. The shortest setting can be changed by operating the management terminal 30. For example, the examiner 610 operates the management terminal 30 to change the set time according to the degree of congestion at the immigration checkpoint, the degree of congestion at the baggage claim area for receiving baggage after entry, and the like. This makes it possible to facilitate the flow of travelers from passing through immigration to receiving baggage and leaving the airport.

It is also possible to set to omit the input operation of pressing the face authentication button 111. Specifically, the process of step S15 shown in FIG. 17 is omitted, and after confirming that the traveler is the passport holder himself / herself, the process of step S16 is subsequently executed to open the gate 12. Similarly, if the process of step S18 is omitted and a predetermined time elapses after the traveler is detected by the motion sensor 120 and before the identity verification is completed (step S11; Yes), the process of step S19 is subsequently executed. Invite travelers to undergo a manned examination. Similarly, regarding the process of step S13 for determining the necessity of displaying the guidance to the traveler, the process of step S12 is omitted, and the determination is executed regardless of the operation of the face authentication button 111.

If the input operation of pressing the face authentication button 111 is omitted, subsequent processing such as opening of the gate 12 is started at the timing when the face matching processing of the face authentication gate device 10 is completed, so that the face authentication gate device 10 is operated. Travelers may feel uncomfortable. However, since it is not necessary to wait for the input operation of pressing the authentication button 111, a series of processes including face authentication can be performed in the shortest time. On the other hand, if an input operation for pressing the face authentication button 111 is provided, the subsequent processing is started at the timing when the traveler performs the operation, so that the traveler does not feel uncomfortable and a series of processing including face authentication is performed. Can be done in a short time.

In either case, before the IC face image is acquired from the IC chip of the passport, the captured face data of the traveler is acquired using the face recognition cameras 71 to 74 installed in the vicinity of the attention target, so that the IC face image is acquired. At the time of acquisition, the captured face data suitable for face matching has already been acquired. Therefore, after the acquisition of the IC face image, the face matching process between the IC face image and the captured face data is immediately performed, and it can be determined whether or not the traveler from which the captured face data is acquired is the passport holder. Therefore, a series of processes including face recognition can be performed in a short time.

Regarding the processing of steps S13 and S14, the determination of the necessity of guidance to the traveler is performed based on the result of the face matching processing. Specifically, the process of collating the captured face data with the IC face image is performed by setting a large number of sample points on each face image and comparing the features at each sample point. The degree of matching of the features at each sample point is expressed by the evaluation points, and the higher the degree of matching, the higher the evaluation points. Details of the face matching process using sample points are described in, for example, Japanese Patent No. 4187494. When the collation result that the captured face data and the IC face image match cannot be obtained and the sample points having low evaluation points are locally concentrated, the control unit 13 needs to guide the traveler. Determination (step S13; Yes). FIG. 18 is an example for explaining an example in which the control unit 13 determines that guidance to a traveler is necessary. A collation target area 920 is set on the face image as shown by a broken line in FIG. A large number of sample points are set in the collation target area 920, and a collation process for comparing two face images is performed. When the two face images do not match and the sample points with low evaluation points are concentrated in the area above the eyes and on both lateral sides of the face, as shown in the shaded area in FIG. 18A, the control unit 13 Displays on the screen of the operation display unit 110 information prompting the traveler to confirm whether the face is hidden by the hair. Similarly, when the sample points with low evaluation points are concentrated in the area above the eyes as shown in FIG. 18B, the control unit 13 displays information prompting confirmation whether the face is hidden by the hat. do. As shown in FIG. 18 (c), when the sample points having low evaluation points are concentrated in the area around the eyes, the control unit 13 displays information prompting the user to remove the spectacles if they are wearing spectacles. As shown in FIG. 18D, when sample points with low evaluation points are concentrated in the peripheral area of the mouth, the control unit 13 prompts the user to remove the mask, muffler, or other attachment if it is worn. Is displayed.

The guidance display prompting the traveler to expose his / her face may be set to start the process immediately after the traveler is detected in step S1. Specifically, the control unit 13 compares the captured face data of the traveler acquired by the face recognition cameras 71 to 74 with the standard face image prepared in advance after detecting the traveler with the motion sensor 120. A standard face image is an average face image created based on a large number of face images. By comparing with the standard face image, it is possible to confirm whether or not the captured face data includes feature parts such as eyes, nose, and mouth. In the comparison between the captured face data of the traveler and the standard face image, a large number of sample points are set and the features at each sample point are compared in the same manner as when the captured face data and the IC face image are collated.

Before acquiring the IC face image of the traveler from the passport in step S6, the face matching process using the standard face image was started, and the eyes, nose, mouth, etc. could not be detected from the captured face data of the traveler. In this case, the control unit 13 executes the guidance display. Specifically, if a hat, sunglasses, a mask, a muffler, or the like is worn, information prompting the user to remove the hat, sunglasses, mask, muffler, or the like to expose the face is displayed on the screen of the operation display unit 110. Processing related to identity verification by starting face matching processing with standard face images immediately after detecting a traveler and displaying guidance in advance if there is a possibility that a face image suitable for identity verification cannot be captured. You can save time.

As a result of the comparison, especially when the sample points having low evaluation points are locally concentrated around both eyes, the control unit 13 determines that the sunglasses are worn. Further, when the sample points having low evaluation points are locally concentrated, particularly around the mouth or around the nose and mouth, the control unit 13 determines that the mask is worn. Sunglasses and masks account for a large proportion of the various wears that hinder face recognition, and if guidance is displayed and guided to be removed at an early stage prior to face matching processing, a series of processing including face recognition will be possible. The time required can be shortened.

On the other hand, in the case of a hat or muffler in which the main parts of the face such as eyes, nose and mouth are not largely hidden, it is not always necessary to display the guidance and have the wearer removed prior to the face matching process. This is because if the number of people who display the guidance to remove the attached object increases, the time required for a series of processes including face recognition may increase. Prior to the face matching process, the guidance process for removing the attached object is limited to the attached object having a high possibility that the face matching process will fail.

The determination as to whether or not the person is wearing sunglasses or a mask is not limited to the above method. For example, a face image of a person wearing sunglasses is collected and an average face image of the person wearing sunglasses is created in advance. When the degree of similarity between the average face image of the person wearing sunglasses and the captured face data is greater than a predetermined value, it may be determined that the traveler is wearing sunglasses. To determine whether or not a mask is worn, the face images of the person wearing the mask are collected to create an average face image of the mask wearer in advance, and the average face image of the mask wearer and the captured face data are used. When the similarity of is greater than a predetermined value, it may be determined that the traveler is wearing a mask.

The face recognition gate device 10 places the passport on the passport mounting table 401, corrects the position of the passport, confirms that the photograph of the passport is oneself, and captures images with the face recognition cameras 71 to 74 during the immigration examination of the traveler. Etc., and inform the traveler about the processing contents and operation method at each stage. At this time, the face recognition gate device 10 determines the proficiency level of the traveler based on the past immigration data managed by the management server 20, and executes guidance in an manner according to the proficiency level.

FIG. 19 is a diagram showing a setting example in which the guidance method is set according to the proficiency level of the traveler. The management server 20 manages the time required for a predetermined operation in association with the passport number when the traveler enters the country using the face authentication gate device 10. For example, the management server 20 manages the time required from the detection of entering the booth 701 by the motion sensor 120 to passing through the gate 12 in association with the passport number of the traveler.

When the control unit 13 of the face recognition gate device 10 acquires the passport number from the passport by the passport reading device 400, the control unit 13 inquires about the past required time managed by the management server 20 based on the passport number. If there is no record of the required time, or if the required time is 20 seconds or more, the control unit 13 executes guidance to the traveler in the beginner mode based on the setting shown in FIG. When the required time is 11 seconds or more and less than 20 seconds, the control unit 13 executes guidance to the traveler in the intermediate mode. If the required time is less than 10 seconds, the control unit 13 executes guidance to the traveler in the advanced mode.

As shown in FIG. 19, the transition speed of the screen when displaying a plurality of guidance displays in order on the screen of the operation display unit 110 is higher in the order of advanced mode, intermediate mode, and beginner mode. The screen transition speed of each mode can be set by the number of seconds for displaying one screen. Further, in the beginner mode and the intermediate mode, the guidance is displayed by a graphic display using a lot of still images and moving images, and in the advanced mode, the guidance is displayed by a text display. In addition, when guiding the operation of a predetermined process consisting of multiple operations, in the beginner mode, one operation is displayed on one screen and guided on multiple screens, and in the intermediate mode, multiple operations are displayed on one screen and multiple screens are displayed. In the advanced mode, all operations are displayed on one screen and the operations are guided on one screen.

In this way, in the face authentication gate device 10, when performing operation guidance, the screen transition speed when displaying a plurality of screens, the display mode of information displayed on the screen, the amount of information displayed on one screen, and the like are determined. It can be changed according to the proficiency level related to the operation of the authentication gate device 10. As a result, a traveler who is not accustomed to the operation can proceed with the operation while slowly checking the operation guidance in order. On the other hand, a traveler who is accustomed to the operation can confirm the operation guidance in a short time and proceed with the operation quickly.

The face recognition gate device 10 also uses the proficiency level of the traveler to determine when calling the concierge by turning on the lamp of the staff call display unit 320. Specifically, since it takes time for a traveler in the beginner mode to proceed with each operation, a long determination time is set until it is determined that assistance by the concierge is necessary. On the other hand, since the traveler in the advanced mode can proceed with the operation in a short time, the judgment time until it is determined that the assistance by the concierge is necessary is set shorter than that in the beginner mode. For travelers in the intermediate mode, the judgment time until it is judged that assistance by the concierge is necessary is set to be shorter than the judgment time in the beginner mode and longer than the judgment time in the advanced mode.

For example, for an operation that can be completed within 2 seconds on average for travelers in advanced mode and 4 seconds on average for travelers in beginner mode, the judgment time for advanced mode is 5 seconds and the judgment time for beginner mode is 5 seconds. Set to 7 seconds and set the intermediate mode judgment time to 6 seconds. When monitoring this operation, the control unit 13 determines that assistance by the concierge is necessary when the traveler in the beginner mode cannot complete the operation within 7 seconds, and lights the staff call display unit 320. On the other hand, if the operation cannot be completed within 5 seconds for the traveler in the advanced mode, it is determined that assistance by the concierge is necessary, and the staff call display unit 320 is turned on. In this way, by changing the time until it is determined that assistance by the concierge is necessary according to the proficiency level of the traveler, it is possible to assist each traveler at an appropriate timing. The determination time for determining whether or not assistance by the concierge is necessary can be set individually according to the type of operation, the content of the operation, and the like.

The face recognition gate device 10 is designed to determine whether or not the traveler has confirmed the information displayed on the screen of the operation display unit 110 when guiding the operation contents to the traveler. FIG. 20 is a flowchart showing the flow of operation guidance. The process shown in FIG. 20 is performed when guiding each operation.

First, the control unit 13 displays information indicating the operation content on the screen of the operation display unit 110 (step S21). The control unit 13 determines whether or not the traveler has confirmed the screen of the operation display unit 110 from the face images of the traveler captured by the face recognition cameras 71 to 74 (step S22). The control unit 13 compares the face image of the traveler with the standard face image prepared in advance for determining the orientation of the face. As the standard face image for determining the face angle and the method for determining the face angle, for example, in addition to the method described in Japanese Patent No. 5628570, other methods may be used. The control unit 13 determines from the angle of the traveler's face whether or not the traveler is facing the operation display unit 110, that is, whether or not the traveler has confirmed the screen of the operation display unit 110.

When it is determined that the traveler has not confirmed the information on the screen (step S22; No), the control unit 13 has elapsed a predetermined predetermined time since the operation content is displayed on the operation display unit 110. Whether or not it is determined (step S23).

Until the predetermined time elapses (step S23; No), the process of confirming whether or not the traveler has confirmed the information on the screen is continued. If the traveler confirms the information on the screen before the predetermined time elapses (step S22; Yes), the process proceeds to the next step. On the other hand, if the traveler does not confirm the information on the screen even after the lapse of a predetermined time (step S23; No), the control unit 13 executes voice guidance (step S24).

Specifically, when the angle of the traveler's face continues to deviate from the direction of the face looking at the screen of the operation display unit 110, the control unit 13 outputs a voice. The control unit 13 outputs, for example, a voice instructing to confirm the information on the screen. Further, for example, the operation content to be performed by the traveler is output by voice. The angle range of the face that can be detected from the face image is limited to a predetermined range both vertically and horizontally from the direction of the face of the traveler looking at the screen of the operation display unit 110. This predetermined range is defined as an angle range in which the orientation of the face is within the range of the screen, for example, within ± 10 °, in consideration of the size of the scene of the operation display unit 110. The control unit 13 determines that the traveler has not confirmed the information on the screen when the face angle satisfies the preset condition or when the face angle cannot be detected (step S22; No).
After the traveler confirms the information on the screen or provides voice guidance, the control unit 13 determines whether or not it is necessary to display the operation support information (step S25). The operation support information is information for guiding the operation contents to the traveler in detail and in an easy-to-understand manner. The time from displaying the operation content to executing the display of the operation support information is preset for each operation. Until this time is exceeded, the control unit 13 determines that the display of the operation support information is unnecessary (step S25; No). When the predetermined time set for each operation is exceeded, the control unit 13 determines that it is necessary to display the operation support information (step S25; Yes), and displays the operation support information on the screen of the operation display unit 110. (Step S26).

The control unit 13 determines whether or not the traveler has completed the operation (step S27). When the traveler has not completed the operation (step S27; No), the control unit 13 determines whether or not a preset predetermined time has elapsed after displaying the operation content on the operation display unit 110. (Step S28). The time required from displaying the operation content to completing the operation is preset for each operation. If this time is not exceeded (step S28; No), the control unit 13 returns to step S25 and continues the process. Then, if the predetermined time set for each operation is exceeded without the traveler completing the operation (step S28; Yes), the control unit 13 determines that assistance by the airport staff 611 is required. Subsequently, the control unit 13 executes a call process of the airport clerk 611 by the clerk call display unit 320 (step S29), and ends the process. The airport clerk 611, who stands by as a concierge in the unmanned examination space, starts assisting the traveler when he / she confirms the display of the clerk call display unit 320. On the other hand, if the traveler finishes the operation (step S27; Yes) until the predetermined time set for each operation elapses (step S28; No), the control unit 13 ends the process.

In the present embodiment, the case where immigration is performed with the layout shown in FIG. 1 has been described, but the layout when arranging a plurality of face recognition gate devices 10 is not limited to this. 21 to 24 are views showing an example of a layout in which a plurality of face recognition gate devices 10 are arranged. In each of the layouts shown in FIGS. 21 to 24, immigration can be performed by using the face recognition gate device 10 as described above.

The layout shown in FIG. 21 shows an example in which a traveler who enters the immigration control area from one entrance and undergoes a manned examination and a traveler who advances to an unmanned examination space are separated on the way. In the unmanned examination space, a plurality of face recognition gate devices 10 are arranged side by side in two rows. The six face recognition gate devices 10 in one row are arranged so that the passage between the two rows of face recognition gate devices 10 becomes narrower as the distance from the counter 600 increases.

Travelers who are admitted to the country by the face recognition gate device 10 in the left column pass through the gate 12 to the left outside. Travelers who are admitted to the country by the face recognition gate device 10 in the right column pass through the gate 12 to the right outside. Travelers who are permitted to enter the country pass through the gate 12 to the left and right outside of the face recognition gate devices 10 arranged in two rows, respectively, but they join in the middle and pass by the side of the same counter 600 to enter the country. do. Travelers who are not admitted to the country by immigration by the face recognition gate device 10 return to the passage that entered the unmanned examination space and are examined by the examiner 610 at the counter 600.

In the case of this layout, one examiner 610 examines travelers who undergo manned examination without proceeding to the unmanned examination space at the counter 600 on the right side. At the counter 600 in the center, one examiner 610 on the right side examines travelers who have not been admitted to the country by immigration by the face recognition gate device 10. The examiner 610 on the left side of the central counter 600 does not examine the traveler and manages all the face recognition gate devices 10.

The layout shown in FIG. 22 shows an example in which an unmanned examination space is provided in the center, a manned examination space is provided on the left side of the unmanned examination space, and a manned examination space is provided on the right side of the unmanned examination space. There is one entrance in each of the unmanned examination space, the manned examination space on the left side, and the manned examination space on the right side.

In the unmanned examination space, a plurality of face recognition gate devices 10 are arranged side by side in two rows. The six face recognition gate devices 10 in one row are arranged so that the passage between the two rows of face recognition gate devices 10 becomes narrower as the counter 600 approaches.

Travelers who are admitted to the country by the face recognition gate device 10 in the left column pass through the gate 12 to the left outside. Travelers who are admitted to the country by the face recognition gate device 10 in the right column pass through the gate 12 to the right outside. A traveler who has passed through the gate 12 on the right side of the unmanned examination space and a traveler who has passed through the gate 12 on the left side enter the country from both outsides of the counter 600 in the center through different passages. Travelers who are not admitted to the country by immigration by the face recognition gate device 10 proceed toward the counter 600 in the center between the face recognition gate devices 10 arranged in two rows and are examined by the examiner 610. ..

In the case of this layout, one examiner 610 on the left side in the counter 600 on the right side examines travelers undergoing manned examination in the manned examination space on the right side. In addition, one examiner 610 on the right side in the counter 600 on the left side examines travelers undergoing manned examination in the manned examination space on the left side. At the central counter 600, two examiners 610 examine travelers who have not been admitted to the country by immigration by the face recognition gate device 10. In addition, two examiners 610 of the central counter 600 manage the face recognition gate device 10 one by one in a row.

The layout shown in FIG. 23 shows an example in which an unmanned examination space is provided in the center, a manned examination space is provided on the left side of the unmanned examination space, and a manned examination space is provided on the right side of the unmanned examination space. There is one entrance in each of the unmanned examination space, the manned examination space on the left side, and the manned examination space on the right side.

In the unmanned examination space, a plurality of face recognition gate devices 10 are arranged side by side in two rows. The six face recognition gate devices 10 in one row are arranged so that the passages between the face recognition gate devices 10 in two rows have the same width.

Travelers who are admitted to the country by the face recognition gate device 10 in the left column pass through the gate 12 to the left outside. Travelers who are admitted to the country by the face recognition gate device 10 in the right column pass through the gate 12 to the right outside. A traveler who has passed through the gate 12 on the right side of the unmanned examination space and a traveler who has passed through the gate 12 on the left side enter the country from both outsides of the counter 600 in the center through different passages. Travelers who are not permitted to enter the country by the face recognition gate device 10 proceed toward the counter 600 in the center between the face recognition gate devices 10 arranged in two rows and are examined by the examiner 610.

In the case of this layout, one examiner 610 on the left side in the counter 600 on the right side examines travelers undergoing manned examination in the manned examination space on the right side. In addition, one examiner 610 on the right side in the counter 600 on the left side examines travelers undergoing manned examination in the manned examination space on the left side. At the central counter 600, one examiner 610 on the right side examines travelers who have not been admitted to the country by immigration by the face recognition gate device 10. On the other hand, the examiner 610 on the left side of the central counter 600 manages all the face recognition gate devices 10.

The layout shown in FIG. 24 shows an example in which two unmanned examination spaces and two manned examination spaces are provided. Two rows of face recognition gate devices 10 are arranged on both sides of a passage through which a traveler who has passed immigration by the face recognition gate device 10 passes. An unmanned examination space is provided on the left outer side of the face recognition gate device 10 in the left column, and a manned examination space is provided on the left outer side thereof. Further, an unmanned examination space is provided on the right outer side of the face recognition gate device 10 in the right column, and a manned examination space is provided on the right outer side thereof. There is one entrance in each of the two manned examination spaces and the two unmanned examination spaces.

The six face recognition gate devices 10 in one row are arranged so that the passages between the face recognition gate devices 10 in two rows have the same width. A traveler who has passed through the gate 12 on the right side from the unmanned examination space on the left side and a traveler who has passed through the gate 12 on the left side from the unmanned examination space on the right side both enter the country through the central passage.

Travelers who have not been admitted to the country after undergoing immigration inspection by the face recognition gate device 10 in the unmanned examination space on the left side proceed to the counter 600 in the unmanned examination space on the left side. Travelers who have not been admitted to the country after undergoing immigration inspection by the face recognition gate device 10 in the unmanned examination space on the right side proceed to the counter 600 in the unmanned examination space on the right side. Gate devices 11 that restrict passage to the counter 600 are arranged in each of the two unmanned examination spaces on the left and right.

In the case of this layout, as shown in FIG. 24, one examiner 610 on the left side in the counter 600 on the right side examines travelers undergoing manned examination in the manned examination space on the right side. In addition, one examiner 610 on the right side in the counter 600 on the left side examines travelers undergoing manned examination in the manned examination space on the left side.

Two examiners 610 of the counter 600 in the center manage the face recognition gate device 10 one by one in a row. Further, the two examiners 610 of the central counter 600 can open and close the gate 12 of the gate device 11 that restricts the passage from the unmanned examination space to the counter 600 by using the management terminal 30. The examiner 610 on the left side controls the gate 12 of the unmanned examination space on the left side, and examines each traveler who is not permitted to enter the country by the immigration examination by the face recognition gate device 10 through the counter 600. Similarly, the examiner 610 on the right side controls the opening and closing of the gate 12 of the unmanned examination space on the right side, and passes the travelers who are not permitted to enter the country by the immigration inspection by the face recognition gate device 10 to the counter 600 one by one. To examine.

In this embodiment, an example of using a plurality of face recognition gate devices 10 is described by taking immigration at an airport as an example, but the usage mode of the face recognition gate device 10 is not limited to this. The face recognition gate device 10 can also be used for immigration. It is also possible to use only one face recognition gate device 10 for immigration. Further, the gate 12 is not opened and closed by the gate device 11, and the predetermined person in charge restricts the passage of the gate that only the person whose identity can be confirmed can pass based on the determination result by the face authentication gate device 10. It can also be used.

In the present embodiment, an example of acquiring a face image from the IC chip of the passport by the passport reading device 400 is shown, but the acquisition mode of the face image used for the face authentication process is not limited to this. For example, the face recognition gate device 10 may include an IC chip reader instead of the passport reading device 400 as a reading device that reads information from an IC chip that is a storage medium. Further, the face image registered in the IC chip may be acquired from a medium such as a paper or resin card or ticket having an IC chip instead of a booklet such as a passport. For example, an ID card such as an employee ID card or a student ID card recording a person's face image, an individual number card, a driver's license, an IC ticket recording the owner's face image, or the like.

For example, the face recognition gate device 10 equipped with an IC chip reader instead of the passport reading device 400 is used for admission to a venue where a concert or a sports competition is held, admission to a building such as an art museum or a museum, or the like. A card with an IC chip that records the face image of the holder is issued as a permit to the holder who has been permitted to enter or enter the museum in advance, and the face is used when visiting or visiting the museum. The examination by the authentication gate device 10 is performed. As described above, the identity can be confirmed by the face matching process between the face image acquired from the IC chip of the permit by the IC chip reader and the captured face data captured by the face recognition cameras 71 to 74. Further, if the opening and closing of the gate 12 is controlled based on the result of the identity verification, it is possible to allow only the holder of the permit to enter or enter the building.

Further, in the present embodiment, an embodiment in which a face matching process is performed using the captured face data and the registered face data using the IC face image registered in advance as the face image of the holder in the IC chip as the registered face data is shown. However, there may be a mode in which a feature amount related to the face image of the holder is registered in the IC chip and the face matching process is performed using the feature amount as the registered face data. Also in this case, it is possible to realize each of the above-mentioned processes by performing identity verification by face matching process.

Further, in the present embodiment, an embodiment in which the passport reader 400 or the IC chip reader is used as the acquisition unit for acquiring the registered face data is shown. However, the face image of the holder or the registered face data which is the feature amount of the face image is registered in an external device such as a server, and the acquisition unit acquires the identification information such as the ID number of the holder. , The mode may be such that the registered face data associated with the identification information is acquired from an external device based on the acquired identification information. Also in this case, it is possible to realize each of the above-mentioned processes by performing identity verification by face matching processing using the registered face data acquired by the acquisition unit.

As described above, when performing face recognition processing using a paper or resin medium having an IC chip other than a passport, the holder is not essential, and identification such as an ID number that identifies the owner of the medium is not required. Processing can proceed using information, media-specific identification information, and the like. Further, when the registered face data is registered in an external device such as a server, if the identification information associated with the registered face data can be acquired, the identification information is not limited to the mode of recording the identification information on the IC chip, and the magnetic information can be obtained. An IC chip is not essential as long as identification information can be obtained by reading or recognizing printed characters. In this case, the acquisition unit first reads the identification information about the medium, receives the registered face data associated with the read identification information from the external device, and acquires the registered face data.

Further, in the present embodiment, the face authentication button 111 is pressed as an input operation performed by the traveler who is the person to be authenticated from the operation unit, and while apparently an instruction to start the identity verification is performed, the result of the identity verification is obtained in the face verification process. An example of triggering a process to be performed based on the above has been shown, but the input operation as a trigger is not limited to the start instruction of identity verification. For example, an input operation for accepting the agreement displayed on the screen, an input operation for declaring the owner of the medium related to face recognition, an input operation for deciding to purchase the product displayed on the screen, etc. If the trigger is used, each of the above-mentioned processes can be performed without making the person to be authenticated aware of face recognition.

Further, in the present embodiment, as the best embodiment for performing a series of processes including face recognition in a short time, the captured face data is created prior to the acquisition of the registered face data by the acquisition unit, and the registered face data is obtained. An example of performing face matching processing immediately after acquisition is shown, but the order of data acquisition is not limited to this. Similar to the conventional face matching device, for example, a face image is obtained after reading and acquiring registered face data from a medium equipped with an IC chip or reading and acquiring registered face data stored in a storage unit. It may be an embodiment in which an image is taken, an imaged face data is created, and a face matching process is performed. Even when the processing is performed in this order, it is possible to perform the processing without a sense of discomfort based on the result of identity verification by face verification, as described above, using the input operation of the authenticated person as a trigger.

As described above, according to the face recognition gate device 10 according to the present embodiment, the face image acquired from the IC chip of the passport by the passport reading device 400 and the face image obtained by capturing the traveler with the face recognition cameras 71 to 74. The gate 12 of the gate device 11 can be opened to allow the traveler to pass only when it can be confirmed that the traveler is the holder of the passport by performing face recognition processing with the user.

Further, according to the face authentication gate device 10 provided with an IC chip reader instead of the passport reader 400 as a reading device, a permit in which an IC chip in which a face image is recorded is embedded is issued in advance, and a visitor can use it. Face matching processing is performed between the face image obtained from the permit possessed by the IC chip reader and the face image of the visitor captured by the face authentication cameras 71 to 74, and the visitor is the holder of the permit. Only when it can be confirmed that the gate 12 of the gate device 11 can be opened to allow admission.

According to the face recognition gate device 10, a face image acquired by imaging a person to be authenticated such as a traveler or a visitor who receives face recognition, or captured face data which is a feature amount of the face image, is registered in advance. Performing a face matching process for collating a face image or registered face data which is a feature amount of the face image, and accurately determining that the person to be authenticated is the registered person in a short time. Can be done. As a result, it is possible to reduce the burden on the person in charge of examination who examines the person to be authenticated and confirms his / her identity. In addition, the burden on the certified person can be reduced by shortening the waiting time for examination.

As described above, the face matching device and the face matching method according to the present invention are useful for performing the process until the result of identity verification by face authentication is obtained in a short time.

10 Face recognition gate device 11 Gate device 12 Gate 13 Control unit 14 Storage unit 15 Communication unit 20 Management server 30 Management terminal 31 Calling unit 40 Surveillance terminal 50 Network 71-74 Face recognition cameras 81, 82 Thermal cameras 91, 92 Surveillance cameras 100 Main unit 101 Partition panel 110 Operation display unit 111 Face recognition button 120 Human sensor 130 Call unit 200 Pillar unit 210 Light source 300 Indicator light unit 310 Booth number display unit 320 Staff call display unit 330 Status display unit 400 Passport reader 401 Passport Stand 411, 412 Position detection sensor 420 Image acquisition unit 421 Glass plate 430 Information acquisition unit 510, 520 Fraud detection unit 511, 521 Depth sensor 700 Partition

Claims (13)

A face matching process is performed to collate the face image of the person to be authenticated or the captured face data which is the feature amount of the face image with the face image registered in advance or the registered face data which is the feature amount of the face image. A face matching device for confirming whether or not the person to be authenticated is the person to whom the registered face data is registered.
A camera that captures the image of the person to be authenticated and
A storage unit that detects a face from the image captured by the camera and stores the captured face data created by the camera.
The acquisition unit that acquires the registered face data,
The face matching process for collating the captured face data stored in the storage unit with the registered face data acquired by the acquisition unit before the registered face data is acquired is executed, and the person to be authenticated is the person himself / herself. A control unit that confirms the existence and performs a predetermined process based on the result of the face matching process .
An input operation unit that accepts an input operation of the authenticated person instructing the start of the face matching process by the control unit.
Equipped with
The control unit started the face matching process before the input operation was performed by the authenticated person in the input operation unit, and the face matching process was able to confirm the identity of the authenticated person. The predetermined process is performed on condition that the input operation is performed by the person to be authenticated in the input operation unit.
A face matching device characterized by that. Further equipped with the object of interest that the person to be authenticated pays attention to,
The face matching device according to claim 1, wherein the camera is installed in the vicinity of the target of interest. Further equipped with the object of interest that the person to be authenticated pays attention to,
The face matching device according to claim 1 or 2, wherein the camera is installed at a position where the front face image of the person to be authenticated can be acquired when the person to be authenticated pays attention to the object of interest. .. Further equipped with the object of interest that the person to be authenticated pays attention to,
The camera is according to any one of claims 1 to 3, wherein the camera is installed in the vicinity of the attention target to be noticed by the authenticated person before the acquisition unit acquires the registered face data. The face matching device described. The object of interest is the display unit.
The face matching device according to claim 4, wherein the display unit displays the registered face data before the acquisition unit acquires the registered face data. The object of interest is a mirror surface.
The face matching device according to claim 4, wherein the mirror surface reflects the face of the person to be authenticated before the acquisition unit acquires the registered face data. The attention target that the person to be authenticated pays attention to
Further equipped with multiple cameras,
The face matching device according to any one of claims 1 to 6, wherein a plurality of the cameras are installed in the vicinity of the target of interest. The plurality of attention targets that the authenticated person pays attention to,
Further equipped with multiple cameras,
The face matching device according to any one of claims 1 to 7, wherein the camera is installed in the vicinity of each of the objects of interest. A display unit is further provided at a position lower than the face of the person to be authenticated as an object of interest to be paid attention to by the person to be authenticated.
The camera includes a camera arranged in the vicinity of the display unit and a camera arranged at a position higher than the camera, and is characterized in that the person to be authenticated is imaged by a plurality of cameras to acquire an image. The face matching device according to any one of claims 1 to 8. Furthermore, it is equipped with a gate part that opens the gate based on the result of identity verification and allows passage.
The face matching device according to any one of claims 1 to 9, wherein the predetermined process performed by the control unit is opening / closing control of the gate unit. The control unit
After executing a face matching process for collating the captured face data stored in the storage unit with the registered face data acquired by the acquisition unit before the acquisition unit acquires the registered face data.
The camera captures the person to be authenticated to create new captured face data, and each time a new captured face data is acquired, a face matching process is executed between the captured face data and the registered face data. The face matching device according to any one of claims 1 to 10, characterized in that. A face matching device for collating a captured face image or captured face data which is a feature amount of the face image with a pre-registered face image or registered face data which is a feature amount of the face image.
With the camera
A storage unit that detects a face from the image captured by the camera and stores the captured face data created by the camera.
The acquisition unit that acquires the registered face data,
A control unit that performs face matching processing for collating the captured face data stored in the storage unit before the registered face data is acquired with the registered face data acquired by the acquisition unit.
An input operation unit that accepts an input operation instructing the start of the face matching process by the control unit.
Equipped with
The control unit starts the face matching process before the input operation is performed by the input operation unit, and the condition is that the face matching process is successful and that the input operation is performed by the input operation unit. Performs the specified processing
A face matching device characterized by that. A face matching process is performed to collate the face image of the person to be authenticated or the captured face data which is the feature amount of the face image with the face image registered in advance or the registered face data which is the feature amount of the face image. A face matching method for confirming whether or not the registered face data is the registered person.
The imaging process in which the camera captures the image of the person to be authenticated,
A storage step of detecting a face from an image captured by the camera and storing the captured face data created in the storage unit, and a storage step.
The acquisition process in which the acquisition department acquires the registered face data,
Before the registered face data is acquired, the face collation process for collating the captured face data stored in the storage unit with the registered face data is executed to confirm whether or not the authenticated person is the person himself / herself. Confirmation process and
The reception process that accepts the input operation of the person to be authenticated that instructs the start of the confirmation process, and
A predetermined processing step of executing a predetermined process based on the result of the face matching process is included.
The definite face matching process is started before the input operation by the authenticated person is accepted, and the identity verification of the authenticated person can be confirmed by the face matching process and the input operation by the authenticated person is accepted. The predetermined process is executed on the condition that
A face matching method characterized by that.

Images