JP2008538998A - Use of time when recognizing a person in an image - Google Patents

Abstract

  A method for identifying a particular at least one person of interest from a digital image collection acquired over a long period of time includes providing a digital image collection in which each digital image includes one or more persons. Storing in the digital database an appearance model having a set of features that are distance measurements related to facial features of a particular person of interest at a certain time; and providing a personal recognition and classification device; The personal recognition and classification device includes the step of identifying a specific person of interest in the images of the digital image collection using the appearance model and the accompanying time.

Description

  The present invention relates to image processing, and more particularly to identifying or recognizing a person of interest contained in a stored digital image or video.

  With the advent of digital photography, consumers are holding large amounts of digital images and video. The average number of images taken by a single person with a digital camera is increasing year by year. As a result, organizing and searching for images and videos is already a problem for the typical consumer. At present, the duration of a digital image taken by a typical consumer is only a few years. The problem of organizing and searching will become more serious as digital images and videos accumulate longer.

  One of the most basic problems in consumer photography is the person. In addition, certain people often appear in accumulated digital images and videos. For example, most images taken by a new parent generally include their baby. Consumers want the ability to find all images, including a specific person, among the images they have taken. However, this is a technically very difficult problem. For example, when a baby grows up and walks, and then grows up as a child, the appearance changes dramatically.

  There are many image processing packages that attempt to recognize people for security or other purposes. To name a few, there are face recognition software FaceVACS from Cognitech Systems, and face recognition SDKs from Imagis Technologies and Identitys. These packages are primarily intended for security-type applications, where a person faces the camera with a neutral look and uniform lighting under a uniform look. These methods are not suitable for use with personal consumer images. This is because in this field, there are various poses, lighting, facial expressions, and face sizes included in images. Furthermore, because such a system is intended for use with adult faces, it cannot successfully recognize images of specific persons of all ages of interest.

  Several image processing techniques for detecting a person's age from a digital image have been reported. For example, Lobo and Kwon describe in US Pat. No. 5,481,650 a method for classifying the age of a person included in a digital image. They measure the face and use a snake to analyze wrinkles and classify the age of the person into the following categories: They are babies (up to about 3 years old), young people (3 to 40 years old), middle-aged and elderly (over 40 years old). In this method, the problem of recognizing who the person included in the image is not discussed. Various other image processing methods for recognizing a face are known. For example, US Patent Application Publication No. 2004/0247177 A1 uses various unique faces to clarify the characteristics of the pixel density pattern of a single target face. There are various problems with any of the above methods, and the person you are interested in may be mistaken.

  One object of the present invention is to provide an improved method for identifying a single person contained in an accumulated image. It has been found that it is particularly effective to classify unknown persons by using an appearance model having features based on measured distances between features of the faces of known persons of interest. In adults, the measurement of the distance between facial features generally does not change over time, improving the search accuracy for the person of interest. Appearance models that use distance measurements can be used effectively in children whose facial features change rapidly as they grow up.

The purpose is to identify at least one particular person of interest from a digital image collection acquired over a long period of time, comprising:
Providing a digital image collection in which each digital image includes one or more persons;
Storing in the digital database an appearance model with a set of features that is a distance measure of the facial features of a particular person of interest at an associated time;
A method comprising: providing a personal recognition and classification device, wherein the personal recognition and classification device identifies a particular person of interest in an image of a digital image collection using an appearance model and associated time Achieved.

  One advantageous effect of the present invention is that one image and its associated time can be used to identify the person of interest contained in the digital image or video. In other words, in the present invention, when an image of a baby is used, it can be expressed as a photograph of the same person who later became a child or an adult.

  The above features and objects of the present invention, as well as other features and objects, and methods for realizing the same will become more apparent from the following description of one embodiment of the present invention with reference to the accompanying drawings. Furthermore, the present invention itself will be better understood from this description.

  In the following description, some embodiments of the present invention will be described as software programs. Those skilled in the art will appreciate that hardware equivalent to such software can be configured. Since image processing algorithms and image processing systems are well known, the description here is used in particular in combination with algorithms and systems that form part of the method according to the invention, or more directly with the method according to the invention. Turn to algorithms and systems. Other aspects of such algorithms and systems and the hardware or software for generating and processing the image signals involved are not specifically shown or described herein, but are conventional. You can choose from known systems, algorithms, components, and elements. As will be understood from the description given in the following part of this specification, it is common that all the description is realized by software, and it is possible for those skilled in the art.

  The present invention can be realized as computer hardware and a computerized device. For example, the method of the present invention can be implemented in a digital camera, digital printer, Internet server, kiosk, personal computer. Referring to FIG. 1, a computer system that implements the present invention is shown. While this computer system is shown for purposes of illustrating a preferred embodiment, the present invention is not limited to the illustrated computer system, and any electronic processing system (eg, digital camera, home computer, The invention can be used in kiosks, small or large photofinishers), or any other system for processing digital images. The computer system includes a microprocessor-based unit 20 (also referred to herein as a digital image processor) for performing installation and processing of software programs, as well as other processing functions. The digital image processor 20 processes an image from the image acquisition device 10 (eg, camera, scanner, computer image generation software). By processing the digital image using the digital image processor 20, the overall brightness, tone scale, image structure, etc. of the digital image are adjusted, and a satisfactory image is generated from the image output device 30. You can make it. The digital image processor 20 exchanges data and instructions as an interface with the overall control computer 40 (which may be a microprocessor-based unit). The overall control computer 40 and the digital image processor 20 can be two different microprocessors, or each function can be implemented by a single microprocessor. The digital image processor 20 often outputs the image to an image output device 30 (eg, a printer that prints the image). A display device 50 is electrically connected to the digital image processor 20 and displays user-related information associated with the software, for example, via a graphical user interface. Since the keyboard 60 is also connected to the microprocessor-based unit 20 through the overall control computer 40, the user can input information to the software. As is well known in the prior art, it is possible to move a selector on the display device 50 using a mouse instead of the keyboard 60 for input, and select an item on which the selector overlaps.

  As a means of inputting software programs and other information to the computer 40 and the digital image processor 20 for general control, a compact disk-read-only memory (CD-ROM) that generally contains software is used for overall control. Insert into computer 40. In addition, since the floppy disk can also accommodate a software program, the floppy disk is inserted into the overall control computer 40 for inputting the software program. . Further, as is well known in the prior art, the overall control computer 40 can be programmed to store a software program therein. The overall control computer 40 can be connected to an external network (for example, a local area network or the Internet) (for example, a telephone line or a wireless connection).

  The image can also be displayed on the display device 50 through a personal computer card (PC card). A PC card is, for example, a previously known PCMCIA card (based on the specification of the Personal Computer Memory Card International Association), in which a digitized image is contained in the card in electronic form.

  The image output device 30 provides the final image. The image output device 30 can be a printer or other output means for providing a paper or other hard copy final image. The image output device 30 can also be an output device that provides the final image as a digital file. The image output device 30 can also combine outputs (printed images on a memory unit such as a CD or DVD and a digital file).

  A digital image includes one or more digital image channels or color components. Each digital image channel is a two-dimensional array of pixels. Each pixel value is related to the amount of light that the image acquisition device receives corresponding to the physical area of the pixel. In color image applications, a digital image often consists of red, green and blue digital image channels. The use of moving images can be considered as a series of digital images. One skilled in the art will appreciate that the present invention can be applied to digital image channels for any of the applications mentioned in this specification. Although described as a two-dimensional array of pixel values with digital image channels arranged in rows and columns, those skilled in the art will appreciate that the present invention can be applied to arrays that are not surrounded by straight lines with the same effect. Like. A person skilled in the art can replace the original pixel value with the pixel value processed in the digital image processing step described below to replace the same processing step with a new digital having the processed pixel value while retaining the original pixel value. You will also understand that creating an image is functionally equivalent to describing it.

  Note that the present invention can be realized by a combination of software and hardware, and that the present invention is not limited to a plurality of devices physically connected to each other or a plurality of devices existing at the same physical location. I want to be. One or more of the devices shown in FIG. 1 can be remotely located and connected through a network. One or more of the devices can be wirelessly connected directly or through a network, for example by a radio frequency link.

  The present invention can be utilized in a variety of user contexts and environments. Examples of contexts and environments include large digital photofinishing stores (including processing steps or processing steps such as receiving, digitally processing, and printing out a film), small digital photofinishing stores (receiving film, digitally Processing and printing out), printing at home (scanning film or digital images at home, digital processing and printing out), desktop software (algorithm applied to digital printing to improve quality) Software that simply changes digital printing, performs digital processing (receives digital images from the media or through the web, digitally processes them, outputs the images to the media in digital form, or images in digital form to the web Output to Or a hard-copy print), kiosk (receives digital input or scanned input, digitally processes and outputs in digital form or hard copy), portable device (eg, processing unit, display unit, PDAs or mobile phones that can be used as any of the units to give processing instructions), services offered through the World Wide Web, and so on.

  The present invention can be independent in each case, or can be an element of a larger system solution. In addition, the human interface (eg, scanning and input, digital processing, display to the user (if necessary), user request or processing command input (if necessary), output), respectively, is the same physical on the same device. May be in different locations, on different devices, and in different physical locations, and communication between devices and locations may involve public network connections, private network connections, and medium-based exchanges. It can be realized through either. If in accordance with the above description of the present invention, the method of the present invention is fully automated, user input (complete manual input or partial manual input), and results are Or allowing the operator to accept / reject, metadata (metadata that can be supplied by the user, metadata supplied by the measuring device (eg camera), metadata determined by the algorithm) Can do. In addition, algorithms can be used to interface with various user interface schemes related to workflow.

  The present invention includes combinations of the embodiments described in this specification. Reference to “a particular embodiment” or the like means a reference to a feature present in at least one embodiment of the invention. Where reference is made separately to "one embodiment" or "special embodiment", it does not necessarily refer to the same embodiment. However, such embodiments are not incompatible with each other, unless expressly stated otherwise or apparent to those skilled in the art.

  One embodiment of the present invention is shown in FIG. The digital image or video 104 is analyzed to recognize who is included in it. Recognizing the person included in the accumulated digital image or video 104 can create a database 120 for the digital image or video 104 and reveal who the person included in the digital image or video 104 is it can. As a result, by asking the question 122 to the database 120, it is possible to find images and videos including the person who is looking for. Database 120 can be in any form. Further, the database 120 can be distributed over a number of storage locations, for example by inserting a person classification 118 into the header of a digital image or video 104 file. For example, a question 122 looking for an image “gym” returns a question answer 124 that includes a group of digital images or videos 104 that include the person “gym” of interest. The question answer 124 can be returned. The question answer 124 is a group of digital images or videos 104 containing the desired person.

  The digital image or video 104 is transferred to the acquisition time retrieval device 108. The acquisition time extraction device 108 clarifies the time when the digital image or video 104 was acquired and outputs the image acquisition time 112. The image acquisition time 112 of the digital image or video 104 is revealed by the acquisition time retrieval device 108 using one of several methods. The acquisition time is often embedded in the header of the digital image or video 104 file. The image or video acquisition device can store information associated with the image or video in the header of the file, for example in the EXIF image format (described at www.exif.org). The “Date / Time” input relates to the date and time when the image was acquired. The image acquisition time 112 may be revealed by detecting the date (which is often done at the acquisition time) when a digital image or video is obtained from the scanning film and is usually printed on the lower left side of the image. is there. The date on which the photo is printed is often printed on the back of the print. There are also film systems in which the film includes a magnetic layer for storing information such as the date of acquisition. The acquisition time extraction device 108 uses an optimum method for extracting the image acquisition time 112 of the image. The source of the digital image 104 is preferably a digital camera, and the acquisition time extraction device 108 preferably extracts the acquisition time from the image file information.

  The image acquisition time 112 can accurately indicate the time up to the minute, such as 10:17 am on March 27, 2004. Alternatively, the image acquisition time 112 may not be as detailed as in 2004 or March 2004. The image acquisition time 112 can be in the form of a probability distribution function, such as March 27 ± 2 of 2004, with 95% confidence. The image acquisition time 112 is input to the personal recognition classification device 114.

  The digital image or video 104 is also input as an input to the person finding device 106, and the person included in the digital image or video is automatically detected. Person discovery device 106 identifies person features 110 with respect to one or more persons included in the digital image or video. The person is preferably found by detecting the face. Methods for detecting human faces are well known in the field of digital image processing. For example, one face detection method for finding human faces in images is described in the following papers: Jones, MJ, Viola, P., "Rapid Multi-View Face Detection", Computer Vision and Pattern Recognition IEEE Conference (CVPR) on June 2003. Furthermore, when a face is detected, well-known methods (eg Yuille et al., “Feature extraction from faces using deformable templates”, Int. Journal of Comp. Vis., Vol. 8, No. 2) 1992, pp. 99-111), it is also possible to specify the position where facial features (for example, eyes, nose, mouth, etc.) are present. The author describes a method that uses energy minimization by template matching to identify mouth, eye, and iris / sclera boundaries. The facial features are TF Cootes and CJ Taylor's “Constrained Active Appearance Model” (8th International Conference on Computer Vision, Vol. 1, pp. 748-754, published by IEEE Computer Society, July 2001) It can also be found using the active appearance model described in. In this preferred embodiment, based on the human face active shape model described by Bolin and Chen in "Facial Feature Discovery Automated System for Portrait Images" (Procedure at IS & T PICS Conference, 2002). A method for identifying facial feature points is used.

  The person finding device 106 outputs a person feature 110 that is a quantitative description of the person detected by the person finding device 106. The person finding device 106 preferably outputs a set of person features 110 for each detected person. The human feature is preferably the location of 82 feature points for a particular facial feature. The features are found using a method similar to the above-described active appearance model of Cootes et al. FIG. 3A and FIG. 3B show facial feature points related to a single facial image as illustrations. Alternatively, the human features 110 can be further expressed using quantifiable descriptors such as eye color, skin color, face shape, and the like. The feature of the person can be a distance between specific feature points or an angle formed by lines connecting the specific feature points. Color information can be easily retrieved from the digital image or video 104 once the person and facial features are identified by the person finding device 106. The person feature 110 is input to the personal recognition classifier 114.

  Alternatively, various facial features can be used. For example, the facial similarity metric described by M. Turk and A. Pentland in “Eigenfaces for Recognition” (Journal of Cognitive Neuroscience, Vol. 3, No. 1, pp. 71-86, 1991) One embodiment based on this is possible. A face descriptor is obtained by projecting an image of a face into a group of principal component functions that describe the variation in facial appearance. The similarity between any two faces is measured by calculating the Euclidean distance for the features obtained by projecting each face onto the same set of functions. In this case, the appearance model can describe the average value and covariance of a plurality of images related to the same face projected onto these principal component functions. A similarity metric (e.g., calculated by the person classifier 132 of FIG. 5) can calculate the Mahalanobis distance of the new face against the average value of the model.

Appearance models and similarity measures could include combinations of several types of features that are not in common with each other (eg, eigenfaces, facial measurements, color / texture information, wavelet features, etc.). When a set of appearance models 116 ₁ -116 _N stored in a digital database is input to the personal recognition classifier 114, who is detected by the person finding device 106 and represented by the person features 110? Is specified. The appearance models 116 _{1 to} 116 _N include a set of features relating to a specific person of interest. In addition, each appearance model is accompanied by a time. The appearance models 116 _{1 to} 116 _N represent the appearance (preferably a face) of a person who is interested in a specific time or age. The appearance model will be described in detail below. The person of interest is an important person to identify in the collected digital image and video 104. For example, in a family with small children, each child will be the person of interest. This is because identifying each child appearing in each photo and video 104 of the digital media collection would be useful and valuable to the family's owner. The purpose of the personal recognition and classification device 114 is to use the one or more of the appearance models 116 _{1 to} 116 _N , the image acquisition time 112, and the human feature 110 as a target for the person detected by the person discovery device 106. It is to clarify whether it is a person. This is accomplished by comparing the person's characteristics 110 for the detected person with the characteristics of the person's appearance model 116 of interest. The personal recognition classification device 114 outputs the person classification 118. The person classification 118 is preferably the probability (probability) that the person detected by the person finding device 106 is the person of interest. The person classification 118 can be a binary indicator (eg, true or false) that indicates a determination of the personal recognition classifier 114 as to whether the person associated with the person feature 110 is the person of interest.

  By applying the personal recognition and classification device 114 repeatedly to each person detected in each collected photo and video 104, all the people detected by the person discovery device 106 are specific persons of interest. Classified as certainty.

Furthermore, appearance models 117 _{1 to} 117 _{M of} other persons who are interested are also input to the personal recognition classification device 114. When each person found by the person finding device 106 generates one set of person features 110, the personal recognition and classification device 114 causes each detected person corresponding to one set of person features 110 to be of interest. A person classification 118 indicating the certainty of being a certain person is output. The person classification is a set of probabilities that the person feature 110 corresponding to the person found by the person finding device 106 is a specific person of interest (one probability for each person of interest). Is preferred.

  In summary, the personal recognition and classification device 114 uses the image acquisition time 112 and the features of the time-appearance appearance model 116 relating to the person of interest to confirm that the detected person is the object of interest. A person classification 118 describing the likelihood is generated.

  Information about the person classification 118 can then be stored in the database 120 with the person features 110 or with digital images or videos. When a question 122 is made on an image or video that includes a particular person of interest, the database 120 searches and an appropriate answer 124 to the question is displayed in the user interface (eg, the display device of FIG. 1). 50) returned to the user through.

The N appearance models 116 _{1 to} 116 _N represent the appearance of one person who is an object of interest. Each appearance model is accompanied by a time. For example, the appearance model 116 ₁ can represent the appearance when the person of interest is 1 year old, and the appearance model 116 ₂ can represent the appearance when the person of interest is 2 years old . In this example, the accompanying times are 1 year old and 2 years old. The time associated with the appearance model can be the age of the person of interest, or an absolute time such as a date. As with image acquisition time 112, the time associated with the appearance model can be a probability distribution function, or accurate (eg, age measured in days of the person of interest) or inaccurate (eg, object of interest). Age measured in 10-year units). The appearance model of the person of interest can also include the person's birthday. As with the image acquisition time 112 and the time associated with the appearance model, the date of birth should be accurate (eg birthday is June 26, 2002) or inaccurate (eg birthday is 1960s), or probability distribution function It can be expressed as

  For example, FIGS. 3A and 3B show two different appearance models of a particular person of interest. Figure 3A is a graphical representation of the appearance model when the particular person of interest is 1 year old, and Figure 3B is a graphical representation of the appearance model when the particular person of interest is 2 years old. is there. The illustration of each appearance model shows the face contour, the eye and eyebrow contour, and the position of the bottom of the nose. The display is standardized so that the left eye is at (0, 0) and the right eye is at (1, 0). Note that in the appearance model, the eyes appear smaller at age 2 than at age 1 because the face changes with growth.

A preferred method for generating an appearance model of a person of interest is shown in FIGS. 4A and 4B. The accumulated digital image or video 104 is processed by calculating the person's features 110 from a group of images containing the person of interest. When the person finding device 106 finds a person included in the image, a person feature 110 is calculated. The faces of people found are classified into similar face clusters. Each cluster must represent an individual or represent one particular appearance of an individual. This is described in United States Patent Application Publication 2003/0210808 A1 published on November 13, 2003, assigned to the assignee. The user confirms that the clustering is correct and that all the people in a single cluster are the same person of interest. Also, the acquisition time extraction device 108 extracts the image acquisition time 112 associated with each image or video 104. The image acquisition time 112 and the person feature 110 are input to the appearance model generation device 102. The appearance model generation device outputs a set of appearance models 116 _{1 to} 116 _N related to a person who is an object of interest. A user input 103 indicating the birthday of the person who is interested is also input to the appearance model generation device 102. The appearance model generation device generates a set of appearance models 116 _{1 to} 116 _N by using features related to a person who is an object of interest. Appearance model generation apparatus 102, for example of interest person using the characteristics associated with all matters detected by the person finding device with respect to that person at the time of 1 year old, to produce the appearance model 116 _1. Appearance model generator 102 then utilizes the feature of interest person associated with all matters detected by the person finding device with respect to that person at the time of two years old, to produce the appearance model 116 _2.

として定義される。ただし||Pn-P(n+1)||は、顔の特徴点nとmの間のユークリッド距離を意味する。特徴に関するこれらArc長を両目の間の距離で割って顔のサイズの違いを規格化する。点PCは、点0と点1の重心に位置する点である（この点は正確に両目の間にある）。ここで利用する顔の測定値は、人の顔において性別、年齢、魅力、民族を判断するのに関係することがわかっている人体測定値に由来するものである（参考：Farkas（編）による『頭と顔の人体測定』、第2版、レイヴン出版、ニューヨーク、1994年）。 Because the human face changes most rapidly during the growth period (0-16 years), the appearance model for children has a shorter time interval (with respect to the time associated with the appearance model) than the appearance model for the non-growth period There is a need to. For better performance, growing children need more appearance models than non-growing adults. For example, it is preferable to generate an appearance model by setting the accompanying time interval to 1 year between 0 to 6 years, and to every 3 years from 16 to 22 years, and every 5 years thereafter. The appearance model 116 relates to a person who is interested in any person whose age (when the person's birthday of interest is known; otherwise, the image acquisition time is used) is within a predetermined interval. It can be generated by calculating the mean and covariance of the features in the scene. In summary, the appearance model 116 includes features representing quantitative attributes of the face. In this preferred embodiment, these attributes of the face are measurements relating to facial features (eyes, nose, mouth, forehead, etc.). The features used are listed in Table 1. When calculating the features, reference is made to the points on the face indicated by numbers in FIG. 4B. Arc (Pn, Pm)

Is defined as However, || Pn-P (n + 1) || means the Euclidean distance between the facial feature points n and m. Divide these Arc lengths for features by the distance between both eyes to normalize the difference in face size. Point PC is a point located at the center of gravity of points 0 and 1 (this point is exactly between the eyes). The facial measurements used here are derived from human measurements that are known to be related to determining gender, age, attraction, and ethnicity in human faces (reference: Farkas) “Human and Body Measurement of Head and Face,” 2nd edition, Raven Publishing, New York, 1994).

  The appearance model describes the appearance of a person who is an object of interest at an accompanying time. An appearance model can include multiple submodels for a particular case. For example, in one appearance model, there can be one sub-model in which a person who is interested is wearing glasses and a second appearance model in which the person who is interested is not wearing glasses.

FIG. 5 shows in more detail the personal recognition and classification device 114 when the time associated with the appearance models 116 _{1 to} 116 _N is the age of the person of interest and the birthday of the person is known. . The age calculator 126 calculates the age at the time of image acquisition for the person of interest. For example, when the image acquisition time is November 2, 1999 and the birthday of the person who is interested is 1977, the age calculator 126 indicates that the age of the person who is interested is 7976-8340. Determined to be a day (21 years 10 months and 2 days to 22 years 10 months and 1 day). The age modeler 128 generates an age designation model 130 using the appearance models 116 _{1 to} 116 _N of the person who is interested and the age of the person who is interested calculated by the age calculator. The age modeler 128 selects the appearance model with the time (age) from among the _N appearance models 116 _{1 to} 116 N related to the person who is interested. It is preferred to select the appearance model that best matches the age. For example, when the age of the person who is interested is 7976-8340 days, the appearance model corresponding to the person who is interested in 22 years old is selected.

  If the age calculator 126 returns a negative time, this means that the image acquisition time is before the birthday of the person of interest. In typical consumer images and videos, it is extremely unusual to have a pre-birth image or video of an individual (except for medical imaging techniques such as ultrasound). Therefore, when the image acquisition time 112 is before the birthday of the person who is the object of interest (included in the appearance model 116), the personal recognition classifying device 114 is interested in the person detected by the person finding device 106. The person classification 118 indicating that the person is not the person who is the object of (or the probability that the person is the object of interest is very small or zero) is generated. Similarly, when the age calculator 126 determines that the age of the person who is interested in the image acquisition exceeds the expected life of the person (for example, 116 years), the person classifying device 132 is the person finding device. A person classification 118 is generated that indicates that the person detected by 106 is not the person of interest (or the probability of being a person of interest is very small or zero).

Alternatively, the appearance model 116 with the time associated with the person who is the object of interest may not exist near the image acquisition time 112. For example, a specific person who is interested may have only one appearance model 116 corresponding to that person who is interested at the age of two. This appearance model 116 may not be appropriate to determine whether the person of interest is a person in an image acquired at the age of 22. Appearance model 116 indicates that when the person who is interested is 0 to 6 years old, the age of the person is within one year from the age of the person, and when the person who is interested is 6 to 16 years old, Within 2 years from the age of the person, within 3 years from the person's age when the person of interest is between 16 and 22 years old, and when the person of interest is over 22 years of age May be appropriate within 5 years. If the model is not appropriate, the age modeler 128 selects one or more appearance models 116 ₁ -116 _N having an accompanying time closest to the age of the person of interest at the time of image acquisition time 112 . The age modeler 128 then corrects the selected appearance model to make up for the difference between the time associated with the appearance model and the age of the person of interest determined at the image acquisition time 112. This is achieved using a generation model for how a person's characteristics change with age. A generation model is a mathematical description of the average movement of facial features (eg eyes, mouth, nose, etc.) as a person ages. When you know the gender or ethnic background of the person you are interested in, you can use special models to describe the movement of facial features. To illustrate this idea, FIG. 6 shows an appearance model of a baby whose accompanying time is 3 months. This appearance model is modified by a generation model with the movement of facial feature points when a baby becomes an adult, forming the age-specific appearance model 130 when the baby becomes a 30-year-old adult as shown in FIG. To do. In this example, the generation model shows changes in facial feature points as the baby becomes an adult. However, similar models for aging can be used to model changes in facial structure between any two ages (eg, 3 and 5 years, or 5 and 10 years). Furthermore, by using this aging model, an age-specified corrected appearance model that is younger than the age associated with the appearance model can be created. For example, an appearance model associated with the age of 30 years can be modified to create an age specification model when the person of interest is 5 years old.

Alternatively, the age designation model 130 can be created by interpolating between two or more appearance models 116 _{1 to} 116 _N related to the person who is interested. For example, there are appearance models when the person of interest is 4 years old and 8 years old. Age calculator 126 determines that the person of interest is 6 years old when the image is acquired. Appearance models at the age of 4 and 8 can be combined (eg, as a weighted average) to create an estimated age-specific model when the person of interest is 6 years old.

  Returning to FIG. 5, the age designation model 130 generated by any of the above methods is input to the person classification device 132 together with the person feature 110 found by the person discovery device 106 to generate the person classification 118.

  Question 122 may include who the person is and the time (absolute time or age). The question can be, for example, searching for an image of “Pete” in 1999, or an image of “Pete” at the age of 34. The person classifier 132 calculates a similarity score between the face measurements calculated from the face images obtained from the user's image collection and the appearance model stored for that person at the age specified in the question 122. To do. A list of collected images is presented to the user in order of decreasing similarity. The similarity score is a weighted difference between a given face measurement and the model, and the weight is proportional to the importance of a particular face measurement that distinguishes individuals. If the weight is based on the variance of the measured values, for example, the Mahalanobis distance can be used. In one preferred embodiment, the distribution of differences between face measurements is calculated for different individuals, for example, for the same individual and from a large set of facial images that know who they are. Assuming a zero mean Gaussian distribution, there is an equiprobable point for each measurement of the face (the difference is the same probability if it is due to variations in the same person and two measurements from different people) Calculated. Since the weight used for the unknown difference is the reciprocal of this point, every difference is expressed as a fraction of an equal probability difference.

FIG. 8 illustrates another embodiment where the time associated with the appearance model 116 is a calendar date (ie, absolute time) and the birthday of the person of interest is not known. In this case, the time modeler 134 generates the time designation appearance model 136 by selecting the appearance model accompanied by the absolute time closest to the image acquisition time 112. This time-designated appearance model 136 is then transferred to the person classification device 132 to generate the person classification 118 as already described. Here, since the birthday of the person who is interested is not known, the appearance model of the person who is interested at a time other than the time associated with the appearance model 116 _{1 to} 116 _N for the person who is interested. It cannot be created using a general age model. In some cases, the age estimation device 142 can estimate the birthday of the person who is interested. If the birthday of the person who is interested can be successfully determined by the age estimation device 142, the information is added to the appearance model 116. Then, the method described with reference to FIG. 5 can be used.

  The age estimator 142 determines the age associated with the appearance model (and thus the birthday of the person of interest) or the age associated with the digital image from which the appearance model is derived. The age estimation device 142 can be, for example, the method described by Lobo in US Pat. No. 5,781,650. The certainty of the birthday of the person of interest is improved by the age estimator 142 estimating the age of multiple images or multiple appearance models and mixing the results of any estimation. Since the time difference between images and the time difference between appearance models is known, the certainty is further improved. Therefore, estimating the age incorrectly can be eliminated by mixing a large number of estimated values.

  FIG. 9 shows yet another embodiment of the present invention. In this embodiment, the age estimator 142 is applied to a person feature 110 relating to the detected person and to the original digital image or video 104. The birthday estimation device 144 can estimate the detected person's birthday using the estimated age from the age estimation device 142 and the image acquisition time 112 associated with the digital image or video 104. As already explained, this detected birthday is a probability distribution function. The detected birthday is compared with the birthday associated with the appearance model of the person who is interested by the person classification device 146, and a person classification 118 is generated. For example, FIG. 10 shows the estimated birthday and the birthday associated with the appearance model. The age estimating device 142 estimates that the birthday of the person detected by the person finding device 106 is a distribution indicated by a dotted line. The distribution of birthdays associated with the appearance model is indicated by a solid line. The person classification device 146 is configured to perform person classification (for example, the above two distributions) based on the estimated birthday distribution from the birthday estimation device 144 and the birthday associated with the appearance model 116 (possibly considering other factors). To determine the sum of the square roots of the products. The person classifier 146 may calculate a product of distributions, for example, and turn it into a person class 118 that indicates the probability or likelihood that the detected person is the person of interest.

  FIG. 9 has an advantage that it is ensured that a person who is interested is not detected in an image having an image acquisition date and time earlier than the birthday of the person. For purposes of explanation, assume that the age estimator 142 simply estimates that any person in the image 104 that may be discovered by the person finder 106 is at least 0 years old. Next, the birthday estimation device 144 confirms that the found person's birthday must be before the image acquisition time 112. Accordingly, in the embodiment of FIG. 9, the person is not found in any image acquired before the birthday associated with the person of interest. The personal recognition and classification device 114 searches for the person who is the object of interest accompanied by the birthday from the images 104 whose acquisition time is later than the birthday, and for the images 104 whose acquisition time is before the birthday Don't look inside.

  It has been found that the method described in this specification has several failure modes. For example, if a picture of a statue of George Washington at the age of 21 was acquired on March 14, 2005, the age calculator 126 in FIG. 5 shows that the person of interest (George Washington) is 273 years old. Therefore, the person classification device 132 outputs a person classification that the detected person is probably not George Washington. Because 273 years is far beyond human life expectancy. In other words, this method may not work in the relatively rare case where the digital image is an image of a person different from the actual person.

1 is a schematic diagram of a computer system capable of implementing the present invention. It is a flow chart of one embodiment of the present invention. It is an appearance model of one person who is an object of interest with different accompanying time. It is an appearance model of one person who is an object of interest with different accompanying time. 6 is a flow chart of an embodiment of the present invention for creating an appearance model with time attached. This is an appearance model of a person whose feature points are labeled. 2 is a more detailed flow chart of one embodiment of the present invention. An appearance model for children. This is a modified appearance model, showing the possible appearance when the child in Figure 6 grows up. 6 is a flow chart of another embodiment of the present invention. 6 is a flow chart of another embodiment of the present invention. It is a graph of the probability distribution regarding the estimated birthday and the birthday associated with the appearance model.

Explanation of symbols

10 Image acquisition device
20 Digital image processor
30 Image output device
40 Computer for overall control
50 display devices
60 keyboard
82 Features
102 Appearance model generator
103 User input
104 Digital image or video
106 Person discovery device
108 Acquisition time retrieval device
110 Characteristic
112 Image acquisition time
114 Personal recognition classification device
116 Appearance model
118 Person classification
120 database
122 Questions
124 questions
126 Age calculator
128 age modeler
130 Age-specified appearance model
132 Person classification device
134 Time Modeler
136 Time specified appearance model
142 Age estimation device
144 Birthday Estimator
146 Person classification device

Claims (24)

A method for identifying at least one particular person of interest from a digital image collection acquired over a long period of time, comprising:
Providing a digital image collection in which each digital image includes one or more persons;
Storing in the digital database an appearance model with a set of features that is a distance measure of the facial features of a particular person of interest at an associated time;
A step of providing a personal recognition and classification device, and using the appearance model and the accompanying time in the personal recognition and classification device to identify the specific person of interest in the image of the digital image collection Including methods.   2. The image acquisition time of each image included in the digital image collection is known, and the personal recognition and classification device uses the image acquisition time to identify the specific person of interest. the method of.   The method of claim 1, wherein the distance measurement is a ratio or angle with respect to a distance between facial features.   4. The method of claim 3, wherein the facial features include eyes, nose, eyebrows and mouth.   2. The method of claim 1, wherein for a child, more appearance models are utilized throughout the growth period.   The age of the person of interest at the time of image acquisition is known and the appearance model is modified to make up for the time difference between the time associated with the appearance model and the age of the person of interest The method according to claim 2.   The method of claim 1, further comprising at least two appearance models with different accompanying times.   The method of claim 1, wherein the feature includes a birthday of the particular person of interest.   The method according to claim 1, wherein the accompanying time is a date on a calendar or an age of the specific person of interest.   Further including at least two appearance models with different accompanying times, the personal recognition and classification device identifies the specific person of interest using an appearance model whose accompanying time is closest to the image acquisition time. The method according to claim 2.   3. The method according to claim 2, wherein the appearance model is modified to generate an age-designated appearance model, and the personal recognition classifier uses the age-designated appearance model to identify the specific person of interest.   It further includes at least two appearance models with different times, and generates an age-designated appearance model from the at least two appearance models, and the personal recognition classifier is an object of interest using the age-designated appearance model. The method according to claim 2, wherein the specific person is identified. A method for identifying at least one particular person of interest from a digital image collection acquired over a long period of time, comprising:
Providing a digital image collection in which each digital image includes one or more persons;
Appearance model with a set of features that is a distance measurement of the facial features of a person, derived from a previously taken digital image of the specific person of interest at a certain time in the digital database Memorizing step;
A step of providing a personal recognition and classification device, and using the appearance model and the accompanying time in the personal recognition and classification device to identify the specific person of interest in the image of the digital image collection Including methods.   14. The method of claim 13, wherein the digital image collection includes the previously captured digital image used in the storing step.   14. The image acquisition time of each image included in the digital image collection is known, and the personal recognition and classification device uses the image acquisition time to identify the specific person who is interested. the method of.   16. The method of claim 15, wherein the appearance model is modified to generate an age-designated appearance model, and the personal recognition and classification device identifies the specific person of interest using the age-designated appearance model.   It further includes at least two appearance models with different times, and generates an age-designated appearance model from the at least two appearance models, and the personal recognition classifier is an object of interest using the age-designated appearance model. 16. The method of claim 15, wherein the specific person is identified.   Further including at least two appearance models with different accompanying times, the personal recognition and classification device identifies the specific person of interest using an appearance model whose accompanying time is closest to the image acquisition time. The method of claim 15.   14. The method according to claim 13, wherein the stored feature is a measurement value relating to a facial feature of the particular person of interest, and the personal recognition classifier utilizes the facial feature.   14. The method of claim 13, further comprising at least two appearance models with different accompanying times.   21. The method of claim 20, wherein for a child, a greater number of appearance models are utilized throughout the growth period.   The method of claim 13, wherein the feature includes a birthday of the particular person of interest.   14. The method of claim 13, wherein the accompanying time is a calendar date or the age of the particular person of interest. A method for identifying at least one particular person of interest from a digital image collection acquired over a long period of time, comprising:
Providing a digital image collection in which the acquisition time of each digital image is known;
Storing in the digital database the appearance model of the particular person of interest and the birthday of the particular person of interest;
A method comprising the step of searching for a particular person of interest only in a group of digital images whose acquisition time is after the birthday of that particular person of interest.

Images