JP5128454B2 - Wrinkle detection device, wrinkle detection method and program - Google Patents

Description

  The present invention relates to a wrinkle detection apparatus, a wrinkle detection method, and a program for detecting the position of a wrinkle from a face image.

  There are methods for determining the state of eyes based on a face image, measuring the direction of the person's line of sight, and estimating the degree of arousal of the person. In order to determine the eye state, it is necessary to accurately detect the eye part of the face image.

  For example, Patent Document 1 describes a technique for detecting the position of an eye part (eyelid) from a driver's face image. The technique of Patent Document 1 extracts and labels edges from an edge image of a face image, and among the labeled edges, only the upper edge whose area (number of pixels) is equal to or smaller than a predetermined value and whose width is equal to or larger than a predetermined value. Extracted as an edge having the characteristics of the upper eyelid. Then, if there is an edge having the lower eyelid feature under the extracted edge having the upper eyelid feature in the tracking window, the pair of the two edges is extracted as the upper and lower eyelid candidates and matched with the template. A technique is disclosed in which the position of the eye part (eyelid) can be detected with high accuracy by examining the correlation and detecting it as a vertical eyelid if the correlation is high.

  Further, Patent Document 2 detects the upper eyelid candidate and the lower eyelid candidate edge from the horizontal edge and the vertical edge of the face image based on the degree of matching with a predetermined condition. Then, the combination of the detected upper eyelid candidate and the lower eyelid candidate is compared, and the combination that most closely matches the predetermined condition with the upper eyelid candidate and the lower eyelid candidate is determined to be the upper eyelid and the lower eyelid. A technique capable of detecting a pair of cocoons is disclosed.

  Further, in Patent Document 3, the position of the nostril is searched from the moving image of the face, the region for searching the upper eyelid and the lower eyelid is determined from the nostril position, and the edge lines that are candidates for the upper eyelid and the lower eyelid are determined from the region. To extract. A technique is disclosed in which, among the extracted edge lines, the center of gravity of the upper eyelid candidate moves in the vertical direction, and the edge line where the center of gravity of the lower eyelid candidate does not move is determined as the eyelid.

Japanese Unexamined Patent Publication No. 2000-137792 JP 2008-140270 A JP 2008-226073 A

  In the techniques of Patent Documents 1 to 3, the edge corresponding to the heel is assumed to be continuous with a considerable length, and an edge having a predetermined length or less is excluded as a heel candidate. However, depending on the light environment in which the face image is taken, the edge corresponding to the eyelid is not always continuous. For example, when red eyes occur in a face image photographed at night, the edges corresponding to eyelids may be interrupted without being continuous due to red-eye emission of the eyeballs. In such a case, in the techniques of Patent Documents 1 to 3, each broken edge is labeled as a different wrinkle candidate, and as a result, each labeled edge is erased by filtering because of its small length, and is not detected as a wrinkle. There is a case.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a wrinkle detection device, a wrinkle detection method, and a program that accurately detect wrinkles without being affected by the light environment.

To achieve the above object, a wrinkle detection device according to a first aspect of the present invention includes a face image acquisition unit that acquires a plurality of face images obtained by continuously capturing the face of a subject, and a wrinkle search from the face images.瞼 search area detection means for detecting an area, edge detection means for detecting edges that are candidates for upper and lower eyelids from the heel search area detected by the heel search area detection means, and edges detected by the edge detection means an edge labeling means for applying a label to each edge group is a set of connection to the, on the basis of the change in the movement over time of each edge group to which the label has been granted by the edge labeling unit, each imparted with label each other interrupted to determine the same edge group from among the plurality of edge groups have a same label discriminating means, e it is determined that it is the same with the same label determination means Among re-labeling means for assigning the same label to each other and edge groups that have been labeled by the re-labeling means, the edge groups that change from top to bottom and from bottom to top within a predetermined time An upper eyelid discriminating means for discriminating an edge group corresponding to the upper eyelid discriminating means, an edge group located in a predetermined area below the edge group corresponding to the upper eyelid discriminated by the upper eyelid discriminating means, an edge group corresponding to the lower eyelid And a lower eyelid discriminating means for discriminating.
In order to achieve the above object, a wrinkle detection device according to a second aspect of the present invention includes a face image acquisition unit that acquires a plurality of face images obtained by continuously photographing a subject's face, and a wrinkle search from the face image.瞼 search area detection means for detecting an area, edge detection means for detecting edges that are candidates for upper and lower eyelids from the heel search area detected by the heel search area detection means, and edges detected by the edge detection means An edge labeling unit that assigns a label to each edge group that is a set of connected to each other, and a plurality of edges to which a label is attached based on a change in movement over time of each edge group to which the label is attached by the edge labeling unit The same label discriminating means for discriminating the same edge group from the groups and the edge groups discriminated by the same label discriminating means as the same label group. Among the re-labeling means for applying the label and the edge groups already labeled by the re-labeling means, the edge group that changes from the upper side to the lower side and from the lower side to the upper side within a predetermined time is determined as the edge group corresponding to the upper eyelid. Upper eyelid discriminating means for discriminating an edge group located in a predetermined area below the edge group corresponding to the upper eyelid discriminated by the upper eyelid discriminating means as an edge group corresponding to the lower eyelid. A predetermined curve is fitted to the edge group corresponding to the upper eyelid and the lower eyelid determined by the upper eyelid determining means or the lower eyelid determining means, and the pixels on the fitted curve are equivalent to the upper eyelid and the lower eyelid. Wrinkle detecting means for detecting as an edge line, and the same label discriminating means is adapted to detect a face image photographed one time before the latest face image. Template data is created from the position information of the edge group, pattern matching using the created template data is performed on the latest face image, and the edge group of the latest face image determined to have the highest correlation as a result of pattern matching The edge group correspondence that distinguishes and associates the edge group of the face image photographed one time earlier than the latest face image from which the template data was created as the same edge group And a change in the movement of the edge group over time is determined by tracing the edge group associated with the edge group association means .

  The edge labeling means may comprise edge erasing means for erasing edges that are not connected by a predetermined number or more of the edges detected by the edge detecting means.

  The same label discriminating means discriminates and associates the same edge group between the edge group of the face image photographed one time before the latest face image and the edge group of the latest face image. An edge group associating unit may be further provided, and a change in the movement of the edge group over time may be determined by tracing the edge group associated with the edge group associating unit.

  The edge group associating means creates template data from edge group position information of a face image taken one temporally before the latest face image, and performs pattern matching using the created template data. The edge group of the latest face image that is determined to have the highest correlation as a result of pattern matching is performed one time before the latest face image from which the template data was created. You may discriminate | determine that it is the same edge group as the edge group of the image | photographed face image.

A predetermined curve is fitted to the edge group corresponding to the upper eyelid and the lower eyelid determined by the upper eyelid determining means or the lower eyelid determining means, and the pixels on the fitted curve are the edges corresponding to the upper eyelid and the lower eyelid. You may further provide the wrinkle detection means detected as a line.

  The upper eyelid determining means includes an edge group that has been labeled by the re-labeling means, an edge group that changes from the upper side to the lower side and a lower side to the upper side within a predetermined time, and the upper eyelid that is fitted to the edge group. An edge group located in the vicinity of a curve serving as a template may be determined as an edge group corresponding to the upper eyelid.

In order to achieve the above object, a wrinkle detection method according to a third aspect of the present invention includes a face image acquisition step of acquiring a plurality of face images obtained by continuously photographing a subject's face, and a wrinkle search from the face images.瞼 search region detection step for detecting a region, edge detection step for detecting edges that are candidates for upper eyelid and lower eyelid from the heel search region detected in the heel search region detection step, and edges detected in the edge detection step an edge labeling step of applying a label to each edge group is a set of connection to the said label edge labeling step is based on the change in movement over time of each edge group granted each imparted with label each other interrupted to determine the same edge group from among the plurality of edge groups have a same label determining step, the same label A re-labeling step that assigns the same label to edge groups that are determined to be the same in another step, and among the edge groups that have been labeled in the re-labeling step, from above and below and within a predetermined time from below An upper eyelid determining step for determining an edge group changing upward as an edge group corresponding to the upper eyelid, and an edge group located in a predetermined area below the edge group corresponding to the upper eyelid determined in the upper eyelid determining step. A lower eyelid determining step for determining an edge group corresponding to the lower eyelid.
In order to achieve the above object, a wrinkle detection method according to a fourth aspect of the present invention includes a face image acquisition step of acquiring a plurality of face images obtained by continuously photographing a subject's face, and a wrinkle search from the face images.瞼 search region detection step for detecting a region, edge detection step for detecting edges that are candidates for upper eyelid and lower eyelid from the heel search region detected in the heel search region detection step, and edges detected in the edge detection step An edge labeling step for assigning a label to each edge group that is a set of connected to each other, and a plurality of edges to which a label is attached based on a change in movement over time of each edge group to which the label is assigned in the edge labeling step In the same label discriminating step for discriminating the same edge group from the group, and in the same label discriminating step, it is determined that they are the same. Re-labeling step for assigning the same label to each edge group, and among the edge groups that have been labeled in the re-labeling step, the edge groups that change from top to bottom and from bottom to top within a predetermined time An upper eyelid determining step for determining an edge group corresponding to an eyelid, and an edge group located in a predetermined area below the edge group corresponding to the upper eyelid determined in the upper eyelid determining step as an edge group corresponding to the lower eyelid The lower eyelid determining step for determining, and the upper eyelid determining step or the upper eyelid determined in the lower eyelid determining step, fitting a predetermined curve to the edge group corresponding to the lower eyelid, the pixels on the fitted curve, A wrinkle detecting step for detecting an edge line corresponding to an upper eyelid and a lower eyelid, and the same In the bell discrimination step, template data is created from the position information of the edge group of the face image taken one temporally before the latest face image, and pattern matching using the created template data is made the latest face image. The edge group of the latest face image determined to have the highest correlation as a result of the pattern matching and the latest face image from which the template data was created were taken one temporally before It further comprises an edge group association step for discriminating and associating the edge group of the face image with the same edge group, and by tracing the edge group associated in the edge group association step, the edge group It is characterized by discriminating a change in the movement of the chronologically over time.

In order to achieve the above object, a program according to a fifth aspect of the present invention provides a computer, a face image acquisition unit that acquires a plurality of face images obtained by continuously photographing a subject's face, and a wrinkle search from the face images.瞼 search area detection means for detecting an area, edge detection means for detecting edges that are candidates for upper and lower eyelids from the heel search area detected by the heel search area detection means, and connection of edges detected by the edge detection means Edge labeling means for assigning a label to each edge group that is a set to be attached, and each of the assigned labels is discontinuous with each other based on a change in movement of each edge group to which the label is assigned by the edge labeling means over time The same label discriminating means for discriminating the same edge group from among a plurality of edge groups, and discriminating that the same label discriminating means is the same Among re-labeling means for assigning the same label to the edge groups, and among edge groups already labeled by the re-labeling means, the edge groups that change from top to bottom and from bottom to top within a predetermined time An upper eyelid discriminating means for discriminating an edge group corresponding to the upper eyelid discriminating means, and an edge group located in a predetermined region below the edge group corresponding to the upper eyelid discriminated by the upper eyelid discriminating means being discriminated as an edge group corresponding to the lower eyelid It is made to function as a lower arm discriminating means to perform.
In order to achieve the above object, a program according to a sixth aspect of the present invention provides a computer, a face image acquisition unit that acquires a plurality of face images obtained by continuously photographing a subject's face, and a wrinkle search from the face images.瞼 search area detection means for detecting an area, edge detection means for detecting edges that are candidates for upper and lower eyelids from the heel search area detected by the heel search area detection means, and connection of edges detected by the edge detection means An edge labeling unit that assigns a label to each edge group that is a set of the plurality of edge groups to which a label is attached based on a change in movement of each edge group that has been given a label by the edge labeling unit over time The same edge group from the same label discriminating means and the same label discriminating means, Among the edge groups that have been labeled by the re-labeling means, the edge groups that change from the upper side to the lower side and the lower side to the upper side within a predetermined time are determined as edge groups corresponding to the upper eyelid. An upper eyelid discriminating means for discriminating an edge group located in a predetermined area below the edge group corresponding to the upper eyelid determined by the upper eyelid discriminating means as an edge group corresponding to the lower eyelid, A predetermined curve is fitted to the edge group corresponding to the upper eyelid and the lower eyelid determined by the upper eyelid determining means or the lower eyelid determining means, and the pixels on the fitted curve are defined as edge lines corresponding to the upper eyelid and the lower eyelid. The same label discriminating means is a face photographed one time before the latest face image. Template data is created from the position information of the edge group of the image, pattern matching using the created template data is performed on the latest face image, and the latest face image determined to have the highest correlation as a result of pattern matching An edge group and an edge group of a face image photographed one time earlier than the latest face image from which the template data was created are identified as the same edge group and associated with each other It further comprises a group association means, and the change in the movement of the edge group over time is discriminated by tracing the edge group associated by the edge group association means.

  According to the wrinkle detection device of the present invention, even when an edge group corresponding to a wrinkle is interrupted, the same label is given based on the change in movement over time. Therefore, wrinkles can be accurately detected without being affected by the light environment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and description thereof will not be repeated. FIG. 1 is a block diagram showing a configuration of a wrinkle detection device 1 according to an embodiment of the present invention. The eyelid detection device 1 according to the present embodiment includes a camera 2 that captures a driver's face and generates a face image, an illumination light source 3 that illuminates the driver's face, and a eyelid detection device that detects the driver's eyes. 10 and a display device 4 connected to the wrinkle detection device 10. Note that the display device 4 is not an essential component, and is not particularly required if it is not necessary to display a face image, a detection result, a determination result, or the like.

  The camera 2 converts the lens image to an electrical signal using, for example, a CCD (Charge Coupled Device), and outputs digitized image data for each pixel. For example, the camera 2 captures a gradation image of the driver's face. The image data generated by the camera 2 includes not only the driver's face but also the background thereof.

  The display device 4 includes an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube), and displays an edge image generated from a face image photographed by the camera 2. Note that the display device 4 is not necessarily provided.

  The eyelid detection device 10 processes the image data acquired by the camera 2 to detect the left and right end positions, the vertical position, and the nostril position of the face. Then, based on these detected positions, a region for searching for wrinkles (a wrinkle search region) is set. Then, upper and lower eyelids are detected from the eye part search region.

  FIG. 2 is a block diagram illustrating an example of the configuration of the eyelid detection device 10. The eyelid detection device 10 includes a transmission / reception unit 11, an image memory 12, an external storage unit 13, a control unit 14, a main storage unit 15, a display control unit 16, and a light source control unit 17. The transmission / reception unit 11, the image memory 12, the external storage unit 13, the control unit 14, the main storage unit 15, the display control unit 16, and the light source control unit 17 are all connected to the control unit 14 via the internal bus 18.

  The transmission / reception unit 11 includes a modem or a network termination device, and a serial interface, a LAN (Local Area Network) interface, an NTSC interface, or the like connected thereto. The control unit 14 inputs image data from the camera 2 via the transmission / reception unit 11. The image memory 12 stores image data generated by the camera 2 and input via the transmission / reception unit 11.

  The external storage unit 13 includes a nonvolatile memory such as a flash memory, a hard disk, a DVD (Digital Versatile Disc), a DVD-RAM (Digital Versatile Disc Random Access Memory), a DVD-RW (Digital Versatile Disc ReWritable), and is controlled. A program for causing the unit 14 to perform processing is stored in advance, and data of the program is supplied to the control unit 14 according to an instruction from the control unit 14, and the data supplied from the control unit 14 is stored. Note that the image data may be stored in the external storage unit 13. The external storage unit 13 stores various threshold data and statistical data used for various determinations, template definition data, and the like, which are used in the eyelid search process described later.

  Further, the external storage unit 13 stores operators of horizontal edge detection and vertical edge detection Sobel filters, which are fixed data for edge detection as shown in FIGS. 3 (A) and 3 (B). . In the present invention, a series of points in the image where the luminance changes from light to dark or from dark to light in the horizontal direction is referred to as a horizontal edge. Since the points of the horizontal edge are generally continuous in the vertical direction, they are also called vertical edges. A series of points in which the luminance changes in the vertical direction from light to dark or from dark to light is called a vertical edge. Since the points of the vertical edges are generally continuous in the horizontal direction, they are also called horizontal edges.

  A horizontal edge (vertical edge) detection Sobel filter shown in FIG. 3A is an operator for extracting a boundary (vertical edge) of gradation differences continuous in the vertical direction as shown in FIG. is there. The Sobel filter for detecting a vertical edge (horizontal edge) shown in FIG. 3B is an operator for extracting a boundary (horizontal edge) of grayscale differences that are continuous in the horizontal direction as shown in FIG. It is.

  Returning to FIG. 2, the control unit 14 includes a CPU (Central Processing Unit) and the like, and controls the overall operation of the eyelid detection apparatus 10. Further, the control unit 14 executes a program stored in the external storage unit 13 to perform processing for detecting wrinkles from the face image (wrinkle detection processing). Details of the wrinkle detection process will be described later.

  The main storage unit 15 includes a RAM (Random-Access Memory) and the like, and is used as a work area for the control unit 14.

The display control unit 16 controls the display device 4 under the control of the control unit 14.
The light source control unit 17 controls turning on / off of the illumination light source 3 under the control of the control unit 14.

  Next, the operation of the wrinkle detection device 1 will be described. The operation of the eyelid detection device 1 is performed by the control unit 14 in cooperation with the camera 2, the transmission / reception unit 11, the image memory 12, the external storage unit 13, and the main storage unit 15.

  For example, the control unit 14 in the computer generates an internal interrupt every 1/30 seconds, for example, every time the internal timer elapses, and executes the eyelid detection process shown in FIG.

  First, the control unit 14 performs preprocessing, captures a driver's face image, and generates an image with an enhanced vertical edge and an image with an enhanced horizontal edge (step S100).

FIG. 5 shows details of the preprocessing (step S100).
When the preprocessing is started, first, the control unit 14 performs a capture process of capturing the face image of the subject (driver) captured by the camera 2 via the transmission / reception unit 11 and storing it in the image memory 12 (step S110). ).

  Next, the control unit 14 performs a coordinate conversion process, and performs a process of thinning out the face image pixels stored in the image memory 12 to such an extent that a Sobel filter process described later can be performed (step S120). Since this process is performed for speeding up, it may be performed as necessary.

  Subsequently, the control unit 14 uses the vertical edge detection operator (FIG. 3A) stored in the external storage unit 13 to process the face image after coordinate conversion, and emphasizes the vertical edge in the face image. Generated image. In addition, the control unit 14 processes the face image after coordinate conversion using the horizontal edge detection operator (FIG. 3B) stored in the external storage unit 13 to emphasize the horizontal edge in the face image. The generated image is generated (step S130). This is the end of the preprocessing.

  Returning to FIG. 4, when the preprocessing is completed, the control unit 14 executes a wrinkle search region setting process for setting a region for searching the position of the wrinkle (step S <b> 200).

FIG. 6 shows details of the eyelid search area setting process (step S200).
When the eyelid search area setting process is started, the control unit 14 detects a face area from the face image (step S210).
For example, the control unit 14 calculates a face contour edge from the face image. Alternatively, the face contour is extracted by pattern matching of the face contour. Then, the control unit 14 detects the edge of the eyebrow from the top and the edge of the mouth from the bottom within the range of the extracted face contour, and is surrounded by the left and right edges of the face, and the upper and lower eyebrows positions. An area may be detected as a face area.

Subsequently, the control unit 14 sets a nostril search region from the detected face region based on statistical data (step S220). For example, as shown in FIG. 7, when a face area F having a height H and a width W is detected in the process of step S210, the nostril search area is a rectangular shape having a height a ₁ · H and a width a ₂ · W. area, at the position of a ₃ · H on mouth under the position M, the horizontal direction in the center of the face region F, may be set as the nostril search region N. Note that a ₁ , a ₂ , and a ₃ are constants calculated from a statistical face image or the like in order to detect the nostril search region.

  Returning to FIG. 6, subsequently, the control unit 14 scans the nostril search region of the face image to obtain the position of the nostril (step S230). For example, template image data of the nostril is prepared in advance, matching processing is performed in the nostril search region using the template, and the position of the nostril is obtained.

Subsequently, the control unit 14 sets a wrinkle search region from the obtained nostril position based on statistical data (step S240).
For example, as shown in FIG. 8, when the position of the nostrils of the (barycentric position) Cn obtained in step S230, the in position above the upper b ₁ · H of the center of gravity position Cn of the nostrils, the center of gravity of the lateral nostril locations away between b ₂ · W only around the position Cn, vertical b ₃ · H, two rectangular region having a width b ₄ · W may be set as the eye search region E. Note that b ₁ , b ₂ , b ₃ , and b ₄ are constants calculated from a statistical face image or the like in order to detect an eye search region. Thus, the wrinkle search area setting process ends.

  Returning to FIG. 4, subsequently, the control unit 14 executes an edge labeling process of detecting and labeling candidate edges corresponding to the eyelid from the eyelid search region (step S300).

  The edge labeling process (step S300) will be described in detail below using the flowchart of FIG.

  When the edge labeling process is started, first, the control unit 14 uses the horizontal edge detection Sobel filter shown in FIG. 3B from the wrinkle search area of the face image to detect the horizontal edges in the wrinkle search area. Is extracted (step S310). At this time, a point (pixel) that moves from a bright pixel to a dark pixel from the top to the bottom of the haze search region is a negative edge, and a point (pixel) that moves from a dark pixel to a bright pixel is a positive edge. .

Subsequently, in the edge image from which the horizontal edge is extracted, the control unit 14 distinguishes and groups the minus edge and the plus edge for each connected edge, and labels the grouped edge set (hereinafter referred to as an edge group). Labeling (identification information) is performed (step S320). In this labeling, edges that are not connected by a predetermined number of pixels may be erased as noise.
This completes the edge labeling process (step S300).

  Returning to FIG. 4, when the edge labeling process is completed, the control unit 14 performs the same labeling in the eyelid search region of the face image (past face image) photographed immediately before the face image currently being processed. Edge association processing is performed in which each edge group is associated with the same edge group labeled in the eyelid search area of the currently processed face image (step S400).

Details of the edge association processing (step S400) will be described with reference to FIG.
When the edge association processing is started, the control unit 14 acquires from the external storage unit 13 a face image (past face image) taken immediately before the face image currently being processed (step S410). ). At this time, the wrinkle search area definition data, edge group position information, and the like in the past face image are also acquired at the same time.

  Subsequently, the control unit 14 selects one of the labeled edge groups in the past face image (step S420).

  Subsequently, the control unit 14 divides the selected edge group into regions each having a predetermined pixel width in the X direction (horizontal direction), and acquires edge position information as template data for each divided region (step S430). .

  Subsequently, the control unit 14 performs pattern matching in the eyelid search region of the current face image using the acquired template data, and determines an edge group having the highest correlation with the template data (step S440).

  Then, the control unit 14 associates the edge group determined to have the highest correlation with the edge group of the past face image that is the basis of the template data as the same edge (step S450). By this processing, the same edge groups are associated with each other between images continuously photographed at predetermined time intervals. Then, by tracing the associated edge group in the shooting order of a plurality of continuously shot images, it is possible to examine the change in the movement of the edge group over time.

  Subsequently, the control unit 14 determines whether or not there is an edge group that has not been selected in Step S420 in the past face image (Step S460).

  When it is determined that there is an edge group for which template data has not been created (step S460; Yes), the control unit 14 selects one edge group for which template data has not been created from the past face image (step S420). The template data of the selected edge group is acquired (step S430), and the process of determining the edge group of the current face image having the highest correlation and associating it with the same edge (steps S440 and S450) is repeated.

  When it is determined that there is no edge group for which template data has not been created (step S460; No), the control unit 14 examines changes in the feature point over time for each edge group, and determines the amount of movement and the movement timing. Labeling movement processing for resetting the same label with the close edge groups as the same edge group is performed (step S470). This completes the edge association processing.

  Returning to FIG. 4, when the edge association processing ends, the control unit 14 executes the upper eyelid detection processing for detecting the upper eyelid edge (step S <b> 500).

  Specifically, the control unit 14 identifies and identifies a positive edge group that moves from the lower direction to the upper direction within a predetermined time and then moves from the lower direction to the upper direction (that is, moves in a V shape). The detected edge group is detected as an edge corresponding to the upper eyelid.

  Subsequently, when the upper eyelid detection process ends, the control unit 14 detects a negative edge group located below the upper eyelid edge group as an edge corresponding to the lower eyelid (step S600). This is the end of the wrinkle detection process.

  Next, the wrinkle detection process will be described with a specific example.

  FIG. 11A shows an image of the eyelid in the eyelid search area. The eyelid image shown in FIG. 11A has a red-eye phenomenon due to the light environment at the time of photographing. Therefore, the visibility near the center of the upper and lower eyelid line (near the eyeball) is deteriorated due to red-eye emission of the eyeball.

  When the image of FIG. 11A is the processing target of the edge labeling process (step S300), first, by extracting the horizontal edge line (step S310), the image shown in FIG. A horizontal edge is extracted as shown in FIG. In FIG. 11B, a positive edge (labeled A on FIG. 11B) is extracted upward, and a negative edge (marked B on FIG. 11B) is extracted downward. Yes. Due to red-eye light emission, in the edge image of FIG. 11B, the edges corresponding to the upper eyelid and the lower eyelid are interrupted at the center portion.

  Subsequently, by the labeling process (step S320), as shown in FIG. 11C, the positive edge group corresponding to the upper eyelid is G + 1, G + 2, and the negative edge group corresponding to the lower eyelid is G-1, G -2, and a label (identification information). In FIG. 11C, a predetermined number of unconnected edges are eliminated as noise.

  Next, the case where the edge association processing (step S400) is performed with the labeled edge image of FIG. 11C as a processing target will be described.

  First, the past face image shown in FIG. 11D is acquired from the external storage unit 13 (step S410). In the past face image shown in FIG. 11D, labels of G + 3 and G + 4 are assigned to the plus edge group, and G-3 and G-4 are assigned to the minus edge group.

  Subsequently, an edge group G + 3 is selected from the past face image shown in FIG. 11D (step S420), and the selected edge group G + 3 is moved in the X direction (horizontal direction) as shown in FIG. The image is divided into areas each having a pixel width, and edge position information is acquired as template data for each area (step S430). The pixel width divided in the X direction may be divided by a predetermined pixel width other than one pixel.

  Then, as shown in FIG. 11F, using the acquired template data, pattern matching is performed within the eyelid search region of the current face image, and an edge group having the highest correlation with the template data is determined (step S440). ). In the example of FIG. 11F, the edge group having the highest correlation with the template data is determined as the edge group G + 1.

  Then, the edge group G + 1 determined to have the highest correlation is associated with the edge group G + 3 of the past face image that is the basis of the template data as the same edge group (step S450).

  Similarly, for other edge groups of the past face image, template data is created, pattern matching is performed, and processing for associating with one of the edge groups of the current face image is performed. Finally, the edge groups G + 3, G + 4, G-3, and G-4 of the past face image are the same as the edge groups G + 1, G + 2, G-1, and G-2 of the current face image. Corresponding as being an edge.

  Subsequently, the control unit 14 examines a change in the movement of the feature point over time for each edge group. For example, FIG. 12A is a diagram showing a time change for each edge group with the maximum value in the Y direction (height) with the nostril position as the origin as a feature point. Note that the feature points are not limited to this, and for example, the minimum value or average value in the Y direction of the edge group, the center of gravity position, and the like may be used as the feature points.

  As can be seen from FIG. 12A, the edge groups G + 1 and G + 2 and the edge groups G-1 and G-2 are the same edge group because the movement amount and the movement timing are close to each other. Determine. Then, as shown in FIG. 13A, the label of the edge group is reset (step S470). That is, the edge group G + 2 is reset to the edge group G + 1, the edge group G-2 is reset to the edge group G-1, and the label is reset. With this process, even edge groups that have been interrupted due to the influence of red-eye emission or the like are labeled as the same group. This completes the edge association processing (step S400).

  Subsequently, the upper eyelid detection process (step S500) is performed, and in the edge image whose label is reset in the process of step S470, as shown in FIG. Changes in movement are examined. Then, from FIG. 12B, the edge group G + 1 that changes from the upper direction to the lower direction within a predetermined time and then moves from the lower direction to the upper direction (that is, moves in a V shape) is defined as an edge corresponding to the upper eyelid. To detect.

  Subsequently, the lower eyelid detection process (step S600) is performed, and the edge group G-1 located in a predetermined region below the edge group G + 1 is detected as the lower eyelid (step S600). Finally, as shown in FIG. 13B, edge groups corresponding to the lower eyelid and the upper eyelid are determined. Thus, the wrinkle detection process ends.

  As described above, in the present embodiment, even when the red-eye phenomenon or the like occurs and the edge group corresponding to the eyelid is interrupted, the same edge group is discriminated from the change in the movement over time and the same labeling is performed. To do. Then, the edge group corresponding to the upper eyelid is detected from the change in the movement of the edge group thus labeled over time, and it corresponds to the lower eyelid from a predetermined area below the edge group corresponding to the detected upper eyelid. Detect the edge group to be used. Therefore, wrinkles can be accurately detected without being affected by the light environment.

In addition, this invention is not limited to the said embodiment, A various deformation | transformation and application are possible.
For example, in the present embodiment, the nostril position is first detected, and the wrinkle search area is obtained based on the detected nostril position. A known method may be used as a method for obtaining the eyelid search region.

  Further, after discriminating the edge groups corresponding to the upper eyelid and the lower eyelid in the above embodiment, as shown in FIG. 14, a predetermined curve such as a Bezier curve, a spline curve, or a regression curve is fitted to the edge group of each eyelid. The pixels on the fitted curve may be edge lines corresponding to the upper eyelid and the lower eyelid. By doing in this way, the edge line interrupted by the influence of the light environment is complemented, and a clearer wrinkle line can be detected.

  In the above embodiment, in the upper eyelid detection process in step S500, a positive edge group moving in a V shape is specified, and the specified edge group is detected as an edge corresponding to the upper eyelid. However, the edges of the upper eyelid and the upper part of the eyelid move less during the opening and closing of the eyes than the edges of other parts of the upper eyelid. For this reason, in the upper eyelid detection process, there is a possibility that the V-shaped movement of the edge is not specified at the corners of the eyes and the corners of the eyes, and the edge corresponding to the upper eyelid may not be detected.

In order to solve this problem, the upper eyelid detection process may be performed according to the procedure shown in FIG.
First, the control unit 14 examines the movement of the plus edge over time, and identifies the plus edge group moving in a V shape as an edge group corresponding to the upper eyelid (step S510). This process is the same as the upper eyelid detection process of the above embodiment.

  Subsequently, the control unit 14 fits a curve (template curve) serving as a template for the upper eyelid edge created from predetermined statistical data or the like to the edge group specified in step S510 (step S520). Then, the control unit 14 also detects a positive edge group located in the vicinity of the fitted curve as an edge group corresponding to the upper eyelid (step S530).

The above-described upper eyelid detection process will be described with a specific example.
For example, consider a case where four plus edge groups C, D, E, and F as shown in FIG. First, when the above-described upper eyelid detection process is started, the temporal movement of each edge group is examined as shown in FIG. 16B, and only the edge groups D and E moving in a V shape are upper eyelids. (Step S510). Note that the edge blocks C and F correspond to the corners of the eyes and the top of the eyes, respectively, so that V-shaped movement is not detected, and the upper eyelid is not specified at this stage. Subsequently, as shown in FIG. 16C, a template curve is fitted to the specified edge groups D and E (step S520), and the edge groups C and F located in the vicinity of the template curve are also It is detected as an edge group of the eyelid (step S530). In summary, finally, as shown in FIG. 16D, all four edge groups of plus edge groups C, D, E, and F are detected as upper eyelids.
Therefore, by performing such an upper eyelid detection process, an edge group such as the corner of the eye or the eye with little change in movement over time is also detected as an upper eyelid, and the accuracy of the eyelid detection can be further improved. .

  In addition, the hardware configuration and the flowchart described above are merely examples, and can be arbitrarily changed and modified.

  The central part that performs processing of the eyelid detection device 10 including the transmission / reception unit 11, the image memory 12, the external storage unit 13, the control unit 14, the main storage unit 15, etc. This can be realized using a computer system. For example, a computer program for executing the above operation is stored and distributed on a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. Thus, the wrinkle detection device 10 that executes the above-described processing may be configured. Alternatively, the eyelid detection device 10 may be configured by storing the computer program in a storage device included in a server device on a communication network such as the Internet and downloading the computer program by a normal computer system.

  Further, when the function of the eyelid detection device 10 is realized by sharing of an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program portion is stored in a recording medium or a storage device. May be.

  It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the computer program distributed via the network. The computer program may be activated and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

It is a block diagram of an eye detection device concerning an embodiment of the present invention. It is a block diagram which shows the structure of the eye detection apparatus shown in FIG. It is a figure which shows the example of the operator for a vertical edge and a horizontal edge detection. It is a flowchart for demonstrating a wrinkle detection process. It is a flowchart for demonstrating the pre-processing in the flowchart of FIG. It is a flowchart for demonstrating the haze search area | region setting process in the flowchart of FIG. It is a figure which shows the example of a nostril search area | region. It is a figure which shows the example of a haze search area | region. It is a flowchart for demonstrating the edge labeling process in the flowchart of FIG. It is a flowchart for demonstrating the edge matching process in the flowchart of FIG. It is a figure for demonstrating the specific example of a wrinkle detection process. It is the figure which showed the change of a motion with time for every edge group. It is a figure for demonstrating the specific example of a wrinkle detection process. It is the figure which showed the specific example which fits a curve to the edge line of a ridge. It is a flowchart for demonstrating the modification of an upper eyelid detection process. It is a specific example for demonstrating the modification of an upper eyelid detection process.

Explanation of symbols

2 camera 3 illumination light source 4 display device 10 eyelid detection device 11 transmission / reception unit 12 image memory (face image storage means)
13 External Storage Unit 14 Control Unit (Wrinkle Search Area Detection Unit, Edge Detection Unit, Edge Labeling Unit, Same Label Discriminating Unit, Relabeling Unit, Upper Eyelid Discriminating Unit, Lower Eyelid Discriminating Unit)
15 Main Memory 16 Display Control Unit 17 Light Source Control Unit 18 Internal Bus

Claims (11)

Face image acquisition means for acquiring a plurality of face images obtained by continuously photographing the face of the subject;
A wrinkle search area detecting means for detecting a wrinkle search area from the face image;
Edge detection means for detecting edges that are candidates for upper eyelid and lower eyelid from the eyelid search area detected by the eyelid search area detection means;
Edge labeling means for assigning a label to each edge group that is a set of connected edges detected by the edge detection means;
The same edge group is identified from among a plurality of edge groups each having a label cut off from each other based on a change in movement of each edge group to which a label is assigned by the edge labeling means over time. Label discrimination means;
Re-labeling means for giving the same label to the edge groups determined to be the same by the same label determining means;
Of the edge groups that have been labeled by the re-labeling means, the upper eyelid determining means for determining an edge group that changes from the upper side to the lower side and from the lower side to the upper side within a predetermined time as an edge group corresponding to the upper eyelid,
Lower eyelid determining means for determining an edge group located in a predetermined region below the edge group corresponding to the upper eyelid determined by the upper eyelid determining means as an edge group corresponding to the lower eyelid;
A wrinkle detection device comprising: The same label discriminating means is
Edge group association means for discriminating and associating the same edge group between the edge group of the face image photographed one time before the latest face image and the edge group of the latest face image Prepared,
By tracing the edge group associated with the edge group association means, the change in the movement of the edge group over time is determined.
The wrinkle detection device according to claim 1 . The edge group association means is
Create template data from the position information of the edge group of the face image taken one time before the latest face image,
Perform pattern matching using the created template data on the latest face image,
The edge group of the latest face image that has been determined to have the highest correlation as a result of pattern matching is the edge of the face image that was photographed one time before the latest face image from which the template data was created. It is determined that the edge group is the same as the group.
The wrinkle detection apparatus according to claim 2 . A predetermined curve is fitted to the edge group corresponding to the upper eyelid and the lower eyelid determined by the upper eyelid determining means or the lower eyelid determining means, and the pixels on the fitted curve are the edges corresponding to the upper eyelid and the lower eyelid. Further comprising wrinkle detection means for detecting as a line ,
The wrinkle detection device according to any one of claims 1 to 3 , wherein   Face image acquisition means for acquiring a plurality of face images obtained by continuously photographing the face of the subject;
  A wrinkle search area detecting means for detecting a wrinkle search area from the face image;
  Edge detection means for detecting edges that are candidates for upper eyelid and lower eyelid from the eyelid search area detected by the eyelid search area detection means;
  Edge labeling means for assigning a label to each edge group that is a set of connected edges detected by the edge detection means;
  The same label discriminating means for discriminating the same edge group from among a plurality of edge groups to which a label is attached based on a change in movement of each edge group to which a label is given by the edge labeling means with time.
  Re-labeling means for giving the same label to the edge groups determined to be the same by the same label determining means;
  Of the edge groups that have been labeled by the re-labeling means, the upper eyelid determining means for determining an edge group that changes from the upper side to the lower side and from the lower side to the upper side within a predetermined time as an edge group corresponding to the upper eyelid,
  Lower eyelid determining means for determining an edge group located in a predetermined region below the edge group corresponding to the upper eyelid determined by the upper eyelid determining means as an edge group corresponding to the lower eyelid;
  A predetermined curve is fitted to the edge group corresponding to the upper eyelid and the lower eyelid determined by the upper eyelid determining means or the lower eyelid determining means, and the pixels on the fitted curve are the edges corresponding to the upper eyelid and the lower eyelid. Wrinkle detecting means for detecting as a line,
  With
  The same label discriminating means is
    Template data is created from the edge group position information of the face image taken immediately before the latest face image, and pattern matching using the created template data is performed on the latest face image. The edge group of the latest face image determined to have the highest correlation as a result of the matching, and the edge group of the face image taken immediately before the latest face image from which the template data was created Is further provided with an edge group association means for identifying and associating with the same edge group,
    By following the edge group associated with the edge group association means, the change in the movement of the edge group over time is determined.
  A wrinkle detection device characterized by that. The edge labeling means includes edge erasing means for erasing edges that are not connected by a predetermined number of pixels or more among the edges detected by the edge detecting means.
The wrinkle detection device according to any one of claims 1 to 5, wherein The upper eyelid determining means is
Among the edge groups that have been labeled by the re-labeling means, the edge groups that change from the top to the bottom and the bottom to the top within a predetermined time, and the vicinity of the curve that becomes the upper eyelid template fitted to the edge groups The edge group located is determined as the edge group corresponding to the upper eyelid,
The wrinkle detection device according to any one of claims 1 to 6 , wherein A face image acquisition step of acquiring a plurality of face images obtained by continuously photographing the face of the subject;
A wrinkle search area detecting step for detecting a wrinkle search area from the face image;
An edge detection step of detecting edges that are candidates for upper eyelid and lower eyelid from the eyelid search region detected in the eyelid search region detection step;
An edge labeling step for assigning a label to each edge group, which is a set of connected edges detected in the edge detection step;
Based on a change in movement of each edge group with a label in the edge labeling step over time, the same edge group is discriminated from among a plurality of edge groups each having a label interrupted. A label determination step;
Re-labeling step of assigning the same label to the edge groups determined to be the same in the same label determination step;
Of the edge groups that have been labeled in the re-labeling step, an upper eye determining step for determining an edge group that changes from the upper side to the lower side and from the lower side to the upper side within a predetermined time as an edge group corresponding to the upper eyelid,
A lower eyelid determining step for determining an edge group located in a predetermined region below the edge group corresponding to the upper eyelid determined in the upper eyelid determining step as an edge group corresponding to the lower eyelid;
A wrinkle detection method comprising:   A face image acquisition step of acquiring a plurality of face images obtained by continuously photographing the face of the subject;
  A wrinkle search area detecting step for detecting a wrinkle search area from the face image;
  An edge detection step of detecting edges that are candidates for upper eyelid and lower eyelid from the eyelid search region detected in the eyelid search region detection step;
  An edge labeling step for assigning a label to each edge group, which is a set of connected edges detected in the edge detection step;
  The same label discrimination step for discriminating the same edge group from among a plurality of edge groups to which a label is attached based on a change in movement over time of each edge group to which a label is given in the edge labeling step;
  Re-labeling step of assigning the same label to the edge groups determined to be the same in the same label determination step;
  Of the edge groups that have been labeled in the re-labeling step, an upper eye determining step for determining an edge group that changes from the upper side to the lower side and from the lower side to the upper side within a predetermined time as an edge group corresponding to the upper eyelid,
  A lower eyelid determining step for determining an edge group located in a predetermined region below the edge group corresponding to the upper eyelid determined in the upper eyelid determining step as an edge group corresponding to the lower eyelid;
  A predetermined curve is fitted to the edge group corresponding to the upper eyelid and the lower eyelid determined in the upper eyelid determining step or the lower eyelid determining step, and pixels on the fitted curve are set to the edges corresponding to the upper eyelid and the lower eyelid. A wrinkle detection step for detecting as a line;
  With
  The same label discrimination step includes
    Template data is created from the edge group position information of the face image taken immediately before the latest face image, and pattern matching using the created template data is performed on the latest face image. The edge group of the latest face image determined to have the highest correlation as a result of the matching, and the edge group of the face image taken immediately before the latest face image from which the template data was created Are further provided with an edge group association step for identifying and associating them with the same edge group,
    By following the edge group associated in the step of associating with the edge group, a change in the movement of the edge group over time is determined.
  A wrinkle detection method characterized by the above. Computer
Face image acquisition means for acquiring a plurality of face images obtained by continuously photographing the face of the subject;
A wrinkle search area detecting means for detecting a wrinkle search area from the face image;
Edge detection means for detecting edges that are candidates for upper eyelid and lower eyelid from the eyelid search area detected by the eyelid search area detection means;
Edge labeling means for assigning a label to each edge group which is a set of connected edges detected by the edge detection means;
The same edge group is identified from among a plurality of edge groups each having a label cut off from each other based on a change in movement of each edge group to which a label is assigned by the edge labeling means over time. Label discrimination means,
Re-labeling means for assigning the same label to edge groups determined to be the same by the same label determining means;
An upper eyelid discriminating means for discriminating, from among the edge groups already labeled by the relabeling means, an edge group that changes from the upper side to the lower side and from the lower side to the upper side within a predetermined time, as an edge group corresponding to the upper eyelid
Lower eyelid determining means for determining an edge group located in a predetermined region below the edge group corresponding to the upper eyelid determined by the upper eyelid determining means as an edge group corresponding to the lower eyelid;
A program characterized by functioning as Computer
  Face image acquisition means for acquiring a plurality of face images obtained by continuously photographing the face of the subject;
  A wrinkle search area detecting means for detecting a wrinkle search area from the face image;
  Edge detection means for detecting edges that are candidates for upper eyelid and lower eyelid from the eyelid search area detected by the eyelid search area detection means;
  Edge labeling means for assigning a label to each edge group which is a set of connected edges detected by the edge detection means;
  Same label discriminating means for discriminating the same edge group from among a plurality of edge groups to which a label is attached based on a change in movement of each edge group to which a label is given by the edge labeling means with time.
  Re-labeling means for assigning the same label to edge groups determined to be the same by the same label determining means;
  An upper eyelid discriminating means for discriminating, from among the edge groups already labeled by the relabeling means, an edge group that changes from the upper side to the lower side and from the lower side to the upper side within a predetermined time, as an edge group corresponding to the upper eyelid;
  Lower eyelid determining means for determining an edge group located in a predetermined region below the edge group corresponding to the upper eyelid determined by the upper eyelid determining means as an edge group corresponding to the lower eyelid;
  A predetermined curve is fitted to the edge group corresponding to the upper eyelid and the lower eyelid determined by the upper eyelid determining means or the lower eyelid determining means, and the pixels on the fitted curve are the edges corresponding to the upper eyelid and the lower eyelid. Wrinkle detection means for detecting as a line,
  Function as
  The same label discriminating means is
    Template data is created from the edge group position information of the face image taken immediately before the latest face image, and pattern matching using the created template data is performed on the latest face image. The edge group of the latest face image determined to have the highest correlation as a result of the matching, and the edge group of the face image taken immediately before the latest face image from which the template data was created Is further provided with an edge group association means for identifying and associating with the same edge group,
    By following the edge group associated with the edge group association means, the change in the movement of the edge group over time is determined.
  A program characterized by that.