JP5790751B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device.

  The following subject tracking devices are known. The subject tracking device detects a person's face from an image and tracks the person's face between a plurality of frames (for example, Patent Document 1).

JP 2006-101186 A

  However, in the conventional subject tracking device, tracking of a face cannot be continued if the face of the person cannot be detected during tracking of the face of the person.

An image pickup apparatus according to a first aspect of the present invention includes an image pickup unit that repeatedly picks up an image, and performs tracking using a human face pattern from an image output by the image pickup unit, and corresponds to the person face pattern. A first tracking unit that detects an area, and a second tracking unit that performs tracking using the face color of the person from the image output from the imaging unit and detects a second area corresponding to the color of the person's face When the first tracking unit can detect the first area, information corresponding to the first area is displayed on the display unit, and when the second tracking unit can detect the second area, A control unit that controls to display information corresponding to a second area on the display unit, and when the first tracking unit can detect the first area, the focus adjustment area of the first area is selected, The first tracking unit provides the first area. A selection unit for selecting a focusing area of the second area when it can not detect, characterized in that it comprises a focusing control unit for the focusing control using the focusing area selected in said selection section .
An image pickup apparatus according to a sixth aspect of the present invention includes an image pickup unit that repeatedly picks up an image, and performs tracking using a human face pattern from an image output by the image pickup unit, and corresponds to the person face pattern. A first tracking unit for detecting an area;
Tracking is performed by detecting a region having the highest degree of matching with the template image obtained using the image output by the imaging unit from the image output by the imaging unit, and the degree of matching with the template image is the highest. A second tracking unit that detects a second area corresponding to the area; and when the first tracking unit detects the first area, information corresponding to the first area is displayed on the display unit, and the second tracking unit When the second area was detected by the tracking unit, the first area could be detected by the control unit that controls to display information corresponding to the second area on the display unit, and the first tracking unit. said first selecting the focusing area of the area, the selection unit for selecting the focusing area of the second area when it can not detect the first area by the first tracking section when, selecting the selector Using the focusing area is characterized in that it comprises a focusing control unit for the focusing control.

  According to the present invention, tracking of a specific target image can be continued even when the specific target image cannot be detected during tracking of the specific target image.

It is a block diagram which shows the structure of one Embodiment of a camera. It is a flowchart figure which shows the flow of a subject tracking process.

  FIG. 1 is a block diagram illustrating a configuration of an embodiment of a camera according to the present embodiment. The camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, and a monitor 106. The operation member 101 includes various operation buttons operated by the user. For example, the operation member 101 includes a power button, a release button, a zoom button, a cross button, an enter button, a play button, a delete button, and the like.

  The lens 102 is composed of a plurality of optical lenses, but is representatively represented by one lens in FIG. The lenses constituting the lens 102 include an AF lens (focusing lens) for AF (Auto Focus / automatic focus adjustment). The image sensor 103 is an image sensor such as a CCD or a CMOS, for example, and receives a light beam from the lens 102 to generate a light reception signal. Then, the generated light reception signal (image signal) is output to the control device 104.

  The control device 104 generates image data in a predetermined image format, for example, JPEG format (hereinafter referred to as “main image data”) based on the image signal input from the image sensor 103. Further, the control device 104 generates display image data, for example, thumbnail image data, based on the generated image data. The control device 104 generates an image file that includes the generated main image data and thumbnail image data, and further includes header information, and outputs the image file to the memory card slot 105. In the present embodiment, image file generation processing by the control device and recording processing of the generated image file on a memory card are referred to as photographing processing.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium, and the image file output from the control device 104 is written and recorded on the memory card. The memory card slot 105 reads an image file stored in the memory card based on an instruction from the control device 104.

  The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100, and the monitor 106 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. . In addition, when the operation member 101 is operated by the user and an instruction to start displaying the captured image (live view) on the monitor 106 is given, the control device 104 displays based on the image signal repeatedly acquired from the image sensor 103. Image data is generated and output to the monitor 106. As a result, a live view is displayed on the monitor 106.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and controls the camera 100. Note that the memory constituting the control device 104 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for the CPU to develop a program when the program is executed or as a buffer memory for temporarily recording data. The flash memory is a non-volatile memory in which data of a program executed by the control device 104, various parameters read during program execution, and the like are recorded.

  As shown in FIG. 1, the control device 104 includes a first tracking unit 104a, a second tracking unit 104b, a control unit 104c, a focus detection unit 104d, and a focus adjustment unit 104e. The processing by the control device 104 to be described is executed by each of these elements.

  Specifically, the first tracking unit 104a tracks a target by repeatedly detecting a pattern corresponding to a specific target (in this embodiment, a human face) from a plurality of images repeatedly obtained from the image sensor 103. To do. The second tracking unit 104b uses at least one of the target color and luminance as reference information, and tracks the target image by repeatedly detecting the target image from a plurality of images based on the reference information. When the tracking of the target by the first tracking unit 104a becomes impossible after the first tracking unit 104a starts tracking the target, the control unit 104c changes the tracking of the target image by the second tracking unit 104b. And control.

  The focus detection unit 104d detects the focus adjustment state of the lens 102 with respect to the position of the target image being tracked. The focus adjustment unit 104e performs focus adjustment of the lens 102 based on the focus adjustment state.

  The camera 100 according to the present embodiment has a tracking function for tracking a subject image in a through image over continuous frames (images) after starting live view. In the present embodiment, when the live view is started and the display of the through image is started, the control device 104 performs tracking using the face of a person in the frame (image) as a tracking target. Specifically, when the live view is started, the control device 104 executes the processing of the flowchart shown in FIG. 2 to track the face of an object over continuous frames (images). That is, the process shown in FIG. 2 is executed by the control device 104 as a program that starts when the user instructs the start of the live view.

  In step S10, the control device 104 generates display image data based on the image signal repeatedly acquired from the image sensor 103, and outputs this to the monitor 106 at a predetermined frame rate, thereby starting live view. Thereafter, the process proceeds to step S20, and the control device 104 clears the subject tracking area that is the subject of subject tracking calculation described later in the image. That is, the control device 104 clears the subject tracking area set in the past subject tracking in order to set the subject tracking area in the image by the processing described below. Then, it progresses to step S30.

  In step S30, the control device 104 acquires an image of the current frame, and proceeds to step S40. In step S40, the control device 104 determines whether the face tracking process for the image acquired in step S30 is the first process after the live view starts, or whether a predetermined time has elapsed since the previous process was executed. judge. If a positive determination is made in step S40, the process proceeds to step S50.

  In step S50, the control device 104 executes a known face detection calculation on the current frame image to detect a human face from the current frame image. For example, the control device 104 performs face detection by detecting a pattern corresponding to the face of a specific target person from the image of the current frame. Thereafter, the process proceeds to step S60, and the control device 104 determines whether or not a face has been detected from the image by face detection calculation. If a positive determination is made in step S60, the process proceeds to step S70.

  In step S70, the control device 104 records, in the flash memory, information regarding the position in the image and the size of the face area of the face area detected by the face detection calculation, that is, the range including the face of the person in the image. At this time, if the position and size of the face area in the image of the past frame has already been recorded, the control device 104 updates the information detected in the current frame. Further, the control device 104 displays information indicating the detected face area, for example, a frame surrounding the face area, on the through image displayed on the monitor 106. Thereafter, the process proceeds to step S80.

  In step S80, the control device 104 sets an AF area at a position corresponding to the detected face area. When a plurality of faces are detected from the image, the AF area is set for any one face, for example, the face area corresponding to the closest face. Alternatively, the AF area may be set for the largest face area among a plurality of face areas.

  After that, the process proceeds to step S90, and the control device 104 sets a predetermined range including the detected face area and its vicinity area as the subject tracking area, and proceeds to step S100. In step S100, the control device 104 updates the subject tracking template image (“setting” in the first time) with the detected image in the face area, and proceeds to step S110.

  In step S110, it is determined whether or not the end of the live view is instructed by operating the operation member 101 by the user. If a negative determination is made in step S110, the process returns to step S30 and is repeated. On the other hand, when an affirmative determination is made in step S110, the process ends.

  On the other hand, if a negative determination is made in step S40, the process proceeds to step S120, and the control device 104 determines whether or not the subject tracking area is cleared. If an affirmative determination is made in step S120, the process proceeds to step S50 described above. On the other hand, if a negative determination is made in step S120, the process proceeds to step S130.

  In step S130, the control device 104 executes a face tracking calculation using an image in the face area detected by the face detection calculation for the set subject tracking area. Specifically, the control device 104 executes a face tracking calculation by detecting a face included in a face area that has already been detected from the subject tracking area. In this manner, by limiting the face detection range to the subject tracking area, processing can be performed at a higher speed than when face detection calculation is performed on the entire current frame. In addition, it is possible to prevent erroneous tracking that tracks another face.

  Thereafter, the process proceeds to step S140, and the control device 104 determines whether or not a face exists in the subject tracking area. If a positive determination is made in step S140, the process proceeds to step S70 described above. In this case, in step S70, the control device 104 sets an area including the face of the subject detected by subject tracking calculation as a face area, and obtains information on the position of the face area in the frame, the size of the face area, and individual information of the face. Record in flash memory. At this time, if the position and size of the face area in the past frame have already been recorded, the control device 104 updates the information detected in the current frame. Further, the control device 104 displays information indicating the detected face area, for example, a frame surrounding the face area, on the through image displayed on the monitor 106.

  On the other hand, if a negative determination is made in step S140, the process proceeds to step S160, and the control device 104 executes subject tracking calculation. The subject tracking calculation will be described later.

  Next, processing when a negative determination is made in step S60 will be described. In this case, the process proceeds to step S150, and the control device 104 determines whether or not the subject tracking area is cleared. If an affirmative determination is made in step S150, the process returns to step S30 and is repeated. On the other hand, if a negative determination is made in step S150, the process proceeds to step S160.

  In step S160, the control device 104 uses the template image updated in step S100 and the image acquired in step S30 as the target within the set subject tracking area, and converts the subject face color information and luminance information into the subject face color information and luminance information. Based on the subject tracking calculation. Specifically, the control device 104 compares the image in the subject tracking area with the template image, and identifies the area in the subject tracking area that has the highest degree of matching with the template image, thereby performing subject tracking calculation. Execute. As a result, even when the subject's face cannot be detected from within the frame due to the subject temporarily facing down or sideways, the subject can be tracked continuously.

  Thereafter, the process proceeds to step S170, and the control device 104 identifies the subject area detected by the subject tracking calculation, that is, a position having a high degree of coincidence with the template image in the frame in the tracking region as the face area. . Then, information regarding the position of the specified face area in the frame and the size of the face area is recorded in the flash memory. At this time, if the position and size of the face area in the past frame have already been recorded, the control device 104 updates the information detected in the current frame. Further, the control device 104 displays information indicating the detected face area, for example, a frame surrounding the face area, on the through image displayed on the monitor 106. Then, it progresses to step S80 mentioned above.

  With the above processing, after detecting a human face by face detection calculation, the control device 104 can speed up the processing by switching the face detection range to face tracking calculation limited to the subject tracking area. it can. In addition, when a predetermined time has elapsed after switching to the face tracking calculation, it is possible to detect the face of a subject that newly appears during live view display by performing face detection processing again. Furthermore, if face detection calculation or face tracking calculation results in failure to detect a person's face, the process can be shifted to subject tracking calculation, even if the face cannot be detected temporarily. The person's face can be tracked continuously.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The control device 104 performs face tracking calculation on the face detected by the face detection calculation, and switches to subject tracking calculation when a face cannot be detected from the tracking area. Thus, even when a face cannot be detected temporarily during tracking of a person's face, tracking of the face can be continued.

(2) The control device 104 stops the subject tracking calculation and restarts the face tracking calculation when the person's face can be detected again after switching to the subject tracking calculation. As a result, when the face of a person can be detected again from the image, it is possible to return to the face tracking calculation with a high tracking capability for the face.

(3) After switching to the face tracking calculation, the control device 104 executes the face detection calculation again on the entire screen when a predetermined time has elapsed since the previous face detection calculation was executed. . Accordingly, it is possible to detect a face of a subject that newly appears during live view display and add a plurality of faces to the tracking target.

(4) The control device 104 sets the AF area based on the face area detected by the face detection calculation, face tracking calculation, or subject tracking calculation, and performs focusing. Thereby, the AF process can be executed so that the face of the person in the image is in focus.

(5) When a plurality of face areas are detected from the frame, the control device 104 targets any one face area, for example, the face area corresponding to the closest face or the largest face area. The AF area was set. Thereby, even when a plurality of faces are detected from the frame, it is possible to accurately focus on any one of the faces.

(6) The control device 104 updates the template image for subject tracking with the image of the face area detected by the face detection calculation, face tracking calculation, or subject tracking calculation. As a result, in the subject tracking calculation, template matching is performed using the image in the face area including the face of the subject and tracking is performed, so that even if the human face cannot be detected from within the frame, You can track your face.

-Modification-
The camera according to the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, the example in which the control device 104 detects and tracks a human face from a through image based on image data input in time series from the image sensor 103 has been described. However, the control device 104 may detect and track a person's face based on data output from an image sensor other than the image sensor 103.

(2) In the embodiment described above, the control device 104 sets a predetermined range including the face area set in the previous frame as the subject tracking area, and performs face tracking calculation in the next frame in the subject tracking area, or An example of subject tracking calculation has been described. However, the control device 104 may perform face tracking calculation or subject tracking calculation in the next frame for the image in the AF area set in the previous frame. As a result, the processing range in the next frame can be limited to the AF area to speed up the processing.

(3) When the camera 100 can set the scene mode at the time of shooting, the control device 104 can be used only when the mode of the camera 100 is set to a mode for shooting a person, for example, a portrait mode. The processing of the above-described embodiment, that is, the detection and tracking of a person's face may be performed. Alternatively, the user may forget to change the mode of the camera 100 to a mode other than the mode for shooting a person, for example, a landscape mode, and then change to a mode for shooting a person. The control device 104 may perform the processing of the above-described embodiment regardless of the set mode.

(4) When the control device 104 does not detect a person's face from the image even after performing face detection calculation on a predetermined number of frames after starting input of the through image, Alternatively, it may be determined that the photographing is not performed with a person as a subject, and the processing of the above-described embodiment, that is, detection / tracking of a person's face may be stopped. In this case, the control device 104 may perform the AF process by the AF method set by the user.

(5) In the above-described embodiment, when a plurality of face areas are detected from the frame, the control device 104 can select any one face area, for example, the face area corresponding to the closest face, The AF area is set for the largest face area. However, the user can select which face to focus on among the plurality of detected faces, and the control device 104 sets the AF area for the face area corresponding to the face selected by the user. You may make it set.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

DESCRIPTION OF SYMBOLS 100 Camera, 101 Operation member, 102 Lens, 103 Image sensor, 104 Control apparatus, 104a 1st tracking part, 104b 2nd tracking part, 104c Control part, 104d Focus detection part, 104e Focus adjustment part, 105 Memory card slot, 106 monitor

Claims (6)

An imaging unit that repeatedly captures images;
A first tracking unit that performs tracking using a human face pattern from an image output by the imaging unit and detects a first area corresponding to the human face pattern;
A second tracking unit that performs tracking using the face color of the person from the image output by the imaging unit and detects a second area corresponding to the color of the person's face;
When the first tracking unit can detect the first area, information corresponding to the first area is displayed on the display unit, and when the second tracking unit can detect the second area, the second area A control unit that controls to display information corresponding to an area on the display unit ;
When the first tracking unit can detect the first area, the focus adjustment area of the first area is selected. When the first tracking unit cannot detect the first area, the focus adjustment area of the second area is selected. A selection section to select;
An imaging apparatus, comprising: a focus adjustment control unit that performs focus adjustment control using the focus adjustment area selected by the selection unit. The imaging apparatus according to claim 1,
When the first tracking unit is able to detect the first area, the control unit can display a frame corresponding to the first area on the display unit, and the second tracking unit can detect the second area. An image pickup apparatus that controls to display a frame corresponding to the second area on the display unit . The imaging apparatus according to claim 1 or 2,
A setting unit for setting a tracking region including the first area when the first tracking unit can detect the first area;
The first tracking unit performs tracking using the face pattern of the person within the range of the tracking region of the image output by the imaging unit,
The second tracking unit performs tracking using the color of the person's face within the tracking region of the image output by the imaging unit when the first tracking unit cannot perform tracking.
The setting unit sets, as the tracking region, an area including at least one of the first area detected by the first tracking unit and the second area detected by the second tracking unit. An imaging device. The imaging apparatus according to claim 3 ,
The second tracking unit does not start tracking by the second tracking unit when the first tracking unit can detect the first area. The imaging apparatus according to any one of claims 1 to 4 ,
An acquisition unit that acquires, as a reference image, an image corresponding to at least one of the first area detected by the first tracking unit and the second area detected by the second tracking unit. An imaging device. An imaging unit that repeatedly captures images;
A first tracking unit that performs tracking using a human face pattern from an image output by the imaging unit and detects a first area corresponding to the human face pattern;
Tracking is performed by detecting a region having the highest degree of matching with the template image obtained using the image output by the imaging unit from the image output by the imaging unit, and the degree of matching with the template image is the highest. A second tracking unit for detecting a second area corresponding to the region;
When the first tracking unit can detect the first area, information corresponding to the first area is displayed on the display unit, and when the second tracking unit can detect the second area, the second area A control unit that controls to display information corresponding to an area on the display unit ;
When the first tracking unit can detect the first area, the focus adjustment area of the first area is selected. When the first tracking unit cannot detect the first area, the focus adjustment area of the second area is selected. A selection section to select;
An imaging apparatus, comprising: a focus adjustment control unit that performs focus adjustment control using the focus adjustment area selected by the selection unit.

Images