JP7536464B2 - IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

For example, imaging devices such as digital cameras and digital video cameras perform image processing to automatically detect specific subject patterns from an image. This image processing makes it possible to identify human face regions and the like. The technology of Patent Document 1 has been proposed as a related technology. The technology of Patent Document 1 detects AF/AE/WB evaluation values based on a captured image in which the amount of change relative to the image of the frame that is the target of face region detection is within a predetermined range. In recent years, convolutional neural networks (hereinafter referred to as CNN) have also been used to detect subjects from images. Technology related to CNN is disclosed in Non-Patent Document 1.

JP 2005-318554 A Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems 25 (NIPS’12), 2012

Dictionary data can be used to detect subjects. Dictionary data is generated for each subject. Therefore, when detecting multiple types of subjects, subject detection is performed while sequentially switching between multiple dictionary data. When subject detection is performed by sequentially switching between multiple dictionary data, it may take a long time to perform subject detection using dictionary data suitable for the target subject. Furthermore, when subject detection is performed using dictionary data from multiple dictionary data that has different characteristics from the target subject, the subject may be erroneously detected. The above problem becomes particularly noticeable when the number of dictionary data is large.

The present invention aims to efficiently select dictionary data when performing subject detection.

In order to achieve the above-mentioned object, the image processing device of the present invention comprises an estimation means for estimating the likelihood of the presence or absence of a subject for each of a plurality of dictionary data used to detect a subject based on captured image data, a subject detection means for detecting a subject from the image data using dictionary data set according to the estimated likelihood, and a control means for switching between the plurality of dictionary data set in the subject detection means, wherein the control means switches between the plurality of dictionary data other than dictionary data whose likelihood is less than a predetermined threshold value .

According to the present invention, it is possible to efficiently select dictionary data when performing subject detection.

FIG. FIG. 2 is a diagram illustrating the relationship between a system control unit and each unit. 3 is a flowchart showing a process flow of the present embodiment. FIG. 11 is a diagram illustrating an example of switching of dictionary data.

Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in each of the embodiments.

The present embodiment will be described below with reference to the drawings. The imaging device 100 of this embodiment generates an image signal from a subject image formed by a plurality of imaging optical systems, and performs subject detection. FIG. 1 is a cross-sectional view of the imaging device 100. The configuration of the imaging device 100 is not limited to the example of FIG. 1. The imaging device 100 has a camera body 101 and a photographing lens 102. The camera body 101 is, for example, a digital single-lens reflex camera body. The photographing lens 102 is detachably attached to the front of the camera body 101. The imaging device 100 has a system control unit 201. The system control unit 201 includes a CPU, a RAM, and a ROM. For example, a control program stored in the ROM may be expanded in the RAM, and the CPU may execute the control program, thereby realizing the processing of this embodiment. The CPU may be, for example, a multi-core CPU capable of parallel processing of multiple tasks.

The photographing lens 102 is an interchangeable lens. The camera body 101 and the photographing lens 102 are also electrically connected via a mount contact group 115. A focusing lens 113 and an aperture shutter 114 are provided inside the photographing lens 102. The amount of light taken into the photographing lens 102 and the focus can be adjusted by controlling via the mount contact group 115.

Next, the camera body 101 will be described. The camera body 101 has a quick return mirror composed of a main mirror 103 and a sub-mirror 104. The main mirror 103 is a half mirror that is obliquely installed on the photographing optical path in the viewfinder observation state and reflects the light beam incident from the photographing lens 102 to the viewfinder optical system. On the other hand, the transmitted light is incident on the distance measurement sensor 105 via the sub-mirror 104. The distance measurement sensor 105 forms a secondary image plane of the photographing lens 102 on a focus detection line sensor, thereby generating an AF image signal that can detect the focus adjustment state of the photographing lens 102 by a phase difference detection method. The generated AF image signal is transmitted to the system control unit 201. The system control unit 201 detects the focus state of the focusing lens 113 based on the AF image signal. The system control unit 201 also performs focus adjustment by controlling the drive of the focusing lens 113 based on the result of focus detection.

The focusing screen 106 is disposed on the intended imaging plane of the photographing lens 102 that constitutes the viewfinder optical system. The pentagon 107 is used to change the viewfinder optical path. The photometry sensor 108 generates image data having a luminance signal and a color difference signal from the irradiated light signal. The photometry sensor 108 generates an AE image signal from the light signal irradiated from the subject and transmits it to the system control unit 201. The system control unit 201 performs exposure control and the like using the received AE image signal. The system control unit 201 optimizes the focus exposure based on the subject detected by the subject detection unit 204, which will be described later. The camera body 101 is provided with an eyepiece 109. The photographer can check the shooting screen and shooting information by observing the focusing screen 106 through the eyepiece 109.

The camera body 101 also has a focal plane shutter 110 and an image sensor 111. In FIG. 2, the focal plane shutter 110 is written as "shutter." When exposure is performed, the main mirror 103 and the sub-mirror 104 move away from the photographing optical path, and the focal plane shutter 110 opens. This exposes the image sensor 111. The focal plane shutter 110 blocks light from the image sensor 111 when photographing is being performed. Also, when photographing is being performed, the focal plane shutter 110 is opened, and the subject light beam is directed to the image sensor 111.

The imaging sensor 111 as an imaging unit is composed of a CCD sensor, a CMOS sensor, etc. The imaging sensor 111 may include an infrared cut filter, a low-pass filter, etc. The imaging sensor 111 photoelectrically converts the subject image formed through the imaging optical system of the imaging lens 102, and transmits an image signal for generating a captured image to the system control unit 201. The system control unit 201 generates a captured image from the received image signal, stores it in the image storage unit 202, and displays it on the display unit 112 such as an LCD. The operation unit 203 detects user operations performed via the release button, switch, connected device, etc. provided on the camera body 101, and transmits a signal according to the operation content to the system control unit 201. When the release button is pressed halfway, the release switch SW1 is turned on, and a shooting preparation operation such as AF or AE is performed. When the release button is pressed all the way, the release switch SW2 is turned on, and a still image shooting operation is performed. The display unit 112 displays the most recently captured still image for a certain period of time so that the user can check the captured image.

FIG. 2 is a diagram showing the relationship between the system control unit 201 and each unit. The above-mentioned units are connected to the system control unit 201 as a control means, and the subject detection unit 204, dictionary estimation unit 205, and dictionary data storage unit 206 are also connected to the system control unit 201. The subject detection unit 204 and dictionary estimation unit 205 may be part of the system control unit 201, or may be provided separately from the system control unit 201. The system control unit 201, subject detection unit 204, and dictionary estimation unit 205 may form an image processing device. The image processing device may be installed in, for example, a smartphone, a tablet terminal, etc.

The subject detection unit 204, which serves as a subject detection means, performs subject detection based on dictionary data generated by machine learning. In this embodiment, the subject detection unit 204 uses dictionary data for each subject in order to detect multiple types of subjects. Each dictionary data is, for example, data in which the characteristics of the corresponding subject are registered. The subject detection unit 204 performs subject detection while sequentially switching between dictionary data for each subject. In this embodiment, the dictionary data for each subject is stored in the dictionary data storage unit 206. Therefore, multiple dictionary data are stored in the dictionary data storage unit 206. The system control unit 201 determines which dictionary data from the multiple dictionary data to use for subject detection based on the estimation result of the dictionary estimation unit 205.

The dictionary estimation unit 205 estimates the likelihood of the presence or absence of a subject for each dictionary data based on image data. The dictionary data storage unit 206 stores dictionary data for each subject. The dictionary estimation unit 205 outputs the likelihood of the presence or absence of a subject for each dictionary data as an estimation result in order to estimate which subject is included in the image. The subject detection unit 204 estimates the position of the subject in the image based on the captured image data and the estimated dictionary data. The subject detection unit 204 may estimate the position, size, reliability, etc. of the subject and output the estimated information. The subject detection unit 204 may output other information.

The dictionary data includes, for example, dictionary data for detecting "people" as a subject, dictionary data for detecting "animals", and dictionary data for detecting "vehicles". Furthermore, dictionary data for detecting "the whole person" and dictionary data for detecting "a person's face" may be stored separately in the dictionary data storage unit 206. The dictionary estimation unit 205 outputs, based on the image data, the likelihood of the presence or absence of a subject for each of the dictionary data for "people", "animals", and "vehicles" as a probability.

The subject detection unit 204 detects subjects from image data using dictionary data with high likelihood. In this embodiment, the dictionary estimation unit 205 is configured with a machine-learned CNN and outputs the likelihood for each dictionary data. The subject detection unit 204 is configured with a machine-learned CNN and estimates the position of the subject included in the image data. In this embodiment, the subject detection unit 204 and the dictionary estimation unit 205 are each configured with a different CNN (convolutional neural network). The subject detection unit 204 and the dictionary estimation unit 205 may be realized by a GPU (graphics processing unit) or a circuit specialized for estimation processing by CNN.

The machine learning of the CNN may be performed by any method. For example, a specific computer such as a server may perform machine learning of the CNN, and the imaging device 100 may obtain the trained CNN from the specific computer. For example, the CNN of the subject detection unit 204 may be trained by a specific computer performing supervised learning using the image data for training as input and the position of the subject corresponding to the image data for training as teacher data. The CNN of the dictionary estimation unit 205 may be trained by a specific computer performing supervised learning using the image data for training as input and dictionary data corresponding to the subject of the image data for training as teacher data. In this way, a trained CNN is generated. The CNN may be trained by the imaging device 100 or the image processing device described above.

Next, the process flow of this embodiment will be described. FIG. 3 is a flowchart showing the process flow of this embodiment. A series of processes from step S301 to step S310 in FIG. 3 corresponds to one frame (one piece of image data) by the imaging device 100. In step S301, the system control unit 201 determines whether the release switch SW1 or the release switch SW2 is on. If the system control unit 201 determines Yes in step S301, it advances one frame and advances the process to step S302. If the system control unit 201 determines No in step S301, it ends the process.

In step S302, the system control unit 201 causes the photometry sensor 108 to accumulate charge and reads out the generated image signal as an AE image signal, and causes the distance measurement sensor 105 to accumulate charge and reads out the generated image signal as an AF image signal. In FIG. 3, the AE image signal and the AF image signal are referred to as image signals. In step S303, the dictionary estimation unit 205 uses the AE image signal read out in step S302 as an input image and outputs the likelihood of the presence or absence of a subject for each dictionary data. In step S304, the system control unit 201 selects dictionary data based on the respective likelihoods of each dictionary data output by the dictionary estimation unit 205, and sets the dictionary data for subject detection. The system control unit 201 may, for example, select only one dictionary data with the highest likelihood, or may select multiple dictionary data with likelihoods greater than a predetermined threshold.

In step S305, the subject detection unit 204 performs subject detection using the dictionary data set in step S304, with the AE image signal read out in step S302 as an input image. At this time, the subject detection unit 204 outputs information such as the position, size, and reliability of the detected subject. At this time, the system control unit 201 may cause the display unit 112 to display the above information output by the subject detection unit 204. In addition, the system control unit 201 may cause the display unit 112 to display the likelihood information of each dictionary data output by the dictionary estimation unit 205.

In step S306, the system control unit 201 selects the focus detection point closest to the position of the subject detected in step S305, and detects the focus state of the selected focus detection point using the AF image signal acquired in step S302. If a subject is not detected in step S305, the system control unit 201 performs focus detection for all focus detection points, and then selects the focus detection point whose focus is closest to the imaging device 100. In step S307, the system control unit 201 adjusts the focus position of the focusing lens 113 based on the focus state of the focus detection point selected in step S306.

In step S308, the system control unit 201 performs automatic exposure calculations in a predetermined manner using the AE image signal read out in step S302, and determines the aperture value (AV value), shutter speed (TV value), ISO sensitivity (ISO value), etc. The AV value, TV value, and ISO value here are determined using a program diagram stored in advance. In step S309, the system control unit 201 determines whether the release switch SW2 is ON. If the system control unit 201 determines Yes in step S309, it advances the process to step S310. On the other hand, if the system control unit 201 determines No in step S309, it returns the process to step S302.

In step S310, the system control unit 201 moves the main mirror 103 and the sub-mirror 104 up to remove them from the optical path, exposes the imaging sensor 111, and captures an image. The exposed imaging sensor 111 generates an image signal and transmits the generated image signal to the system control unit 210. The system control unit 210 then generates image data based on the image signal received from the imaging sensor 111, stores the image in the image storage unit 202, and displays the image on the display unit 112.

Next, switching of dictionary data will be described. The system control unit 201 sets dictionary data in the subject detection unit 204 based on the likelihood of each dictionary data estimated by the dictionary estimation unit 205. In this embodiment, the system control unit 201 sets the order and period of dictionary data to be set in the subject detection unit 204 based on the likelihood of each dictionary data. Figure 4 is a diagram showing an example of switching of dictionary data. In the example of Figure 4, it is assumed that three dictionary data (dictionary data 1 to dictionary data 3) are stored in the dictionary data storage unit 206. However, many dictionary data may be stored in the dictionary data storage unit 206.

FIG. 4A shows an example of sequentially switching dictionary data set in the subject detection unit 204. "Detection of dictionary n (n is 1 to 3)" in FIG. 4 indicates the time when the subject detection unit 204 performs subject detection using dictionary n. "Image" indicates the time when image data is acquired. In the example of FIG. 4A, the dictionary data is sequentially and evenly switched in cycles of dictionary data 1, dictionary data 2, and dictionary data 3. Here, it is assumed that the dictionary data corresponding to the target subject is dictionary data 3. In this case, subject detection using dictionary data 3 is performed after subject detection using dictionary data 1 and subject detection using dictionary data 2. Therefore, it takes a long time until subject detection using dictionary data 3 is performed. In particular, when the number of dictionary data stored in the dictionary data storage unit 206 increases, it may take a very long time until subject detection using dictionary data corresponding to the target subject is performed.

Therefore, in this embodiment, the system control unit 201 switches the dictionary data according to the likelihood of each dictionary data stored in the dictionary data storage unit 206. The system control unit 201 excludes dictionary data whose likelihood estimated by the dictionary estimation unit 205 is less than a predetermined threshold from the targets to be set in the subject detection unit 204. When switching the dictionary data, the system control unit 201 increases the priority of the order in which dictionary data with a high estimated likelihood is set in the subject detection unit 204 and shortens the switching cycle.

In other words, the system control unit 201 schedules switching so that the order of setting the dictionary data to the subject detection unit 204 is earlier as the likelihood of the dictionary data increases. Also, the system control unit 201 schedules switching so that the cycle of switching the dictionary data to be set to the subject detection unit 204 is shorter as the likelihood of the dictionary data increases. The system control unit 201 may control only one of the order of the dictionary data to be set to the subject detection unit 204 and the cycle of switching the dictionary data, or may control both. In this embodiment, it is described that both order control and cycle control are performed.

Figure 4 (b) shows an example of dictionary data switching control performed by the system control unit 201. The dictionary estimation unit 205 estimates that the likelihood of dictionary data 1 is 0%, the likelihood of dictionary data 2 is 40%, and the likelihood of dictionary data 3 is 80%. Here, it is assumed that the above-mentioned predetermined threshold is set to 30%. The system control unit 201 excludes dictionary data 1 whose likelihood is less than the predetermined threshold from the targets to be set in the subject detection unit 204. For this reason, the system control unit 201 controls switching between dictionary data 2 and dictionary data 3 other than the dictionary data whose likelihood is less than the predetermined threshold. By excluding dictionary data with low likelihood from the targets to be set in the subject detection unit 204, the use of dictionary data that is not suitable for detecting the target subject is suppressed. This reduces erroneous detection of the subject.

As shown in FIG. 4B, the system control unit 201 first sets the dictionary data 3 with the highest likelihood of the three dictionary data to the object detection unit 204. Then, the system control unit 201 sets the dictionary data 2 with the next highest likelihood to the object detection unit 204. As a result, the system control unit 201 controls the order of switching the dictionary data. Then, the system control unit 201 sets the dictionary data 3 to the object detection unit 204 consecutively. As a result, the system control unit 201 controls the cycle of switching the dictionary data. In other words, when switching the dictionary data to be set in the object detection unit 204, the system control unit 201 advances the order of the dictionary data 3 with the highest likelihood and shortens the switching cycle. The dictionary data 3 with the highest estimated likelihood is likely to be suitable for detecting the target object. The system control unit 201 controls the order and cycle as described above, thereby shortening the time until the target object is detected.

The system control unit 201 may also change the threshold (reliability threshold) used by the subject detection unit 204 to detect a subject, depending on the likelihood of each dictionary data. For example, if the likelihood of the dictionary data set in the subject detection unit 204 is low, the system control unit 201 sets the reliability threshold for subject detection high. This makes it difficult to detect a subject using dictionary data with low likelihood. On the other hand, if the likelihood of the dictionary data set in the subject detection unit 204 is high, the system control unit 201 sets the reliability threshold for subject detection high. This makes it easier for the subject detection unit 204 to detect a subject using dictionary data with high likelihood.

Furthermore, if the subject detection unit 204 fails to detect a subject despite having set dictionary data with a high estimated likelihood in the subject detection unit 204, the system control unit 201 may process the image to be input to the subject detection unit 204. As a processing method, for example, a trimming process is applied to make subject detection easier. The system control unit 201 may perform a trimming process on the input image data based on the center of the image. This makes it possible to increase the size of the subject in the image data, making it easier for the subject detection unit 204 to detect the subject.

As described above, the dictionary estimation unit 205 estimates the likelihood of the presence or absence of a subject for each dictionary data in order to estimate which of multiple types of subjects is present in the captured image data. Furthermore, the subject detection unit 204 estimates the position in the image data at which the subject is present. For this reason, the estimation results of the subject detection unit 204 are more likely to change over time than the estimation results of the dictionary estimation unit 205. Therefore, it is preferable that the processing cycle of the subject detection unit 204 is shorter than the processing cycle of the dictionary estimation unit 205.

As described above, the subject detection unit 204 and the dictionary estimation unit 205 are composed of a CNN trained by machine learning (trained CNN). The subject detection unit 204 takes image data as input, estimates the position, size, reliability, etc. of the subject, and outputs the estimated information. The dictionary estimation unit 205 takes image data as input, estimates the likelihood of each dictionary data, and outputs information on the estimated likelihood. The CNN may be, for example, a network in which a fully connected layer and an output layer are connected to a layer structure in which convolution layers and pooling layers are alternately stacked. In this case, for example, an error backpropagation method or the like may be applied as the learning of the CNN. The CNN may also be a neocognitron CNN that includes a feature detection layer (S layer) and a feature integration layer (C layer). In this case, a learning method called "Add-if Silent" may be applied as the learning of the CNN.

The CNN of the subject detection unit 204 and the CNN of the dictionary estimation unit 205 may have different learned coefficient parameters, etc. In this case, the learned coefficient parameters may be switched between the CNN of the subject detection unit 204 and the CNN of the dictionary estimation unit 205. In addition, the CNN of the subject detection unit 204 and the CNN of the dictionary estimation unit 205 may have different network configurations.

In addition, any trained model other than a trained CNN may be used for the subject detection unit 204 and the dictionary estimation unit 205. For example, a trained model generated by machine learning such as a support vector machine or a decision tree may be applied to the subject detection unit 204 and the dictionary estimation unit 205. In addition, the subject detection unit 204 and the dictionary estimation unit 205 do not have to be trained models generated by machine learning. For example, any subject detection method that does not use machine learning may be applied to the subject detection unit 204. A function that outputs a likelihood, etc. may be applied to the dictionary estimation unit 205.

As described above, the dictionary estimation unit 205 makes the estimation, and from multiple dictionary data, it is possible to set dictionary data suitable for the target subject in the subject detection unit 204. Then, the subject detection unit 204 performs subject detection using dictionary data suitable for the target subject. This shortens the time required to detect the target subject, and makes it possible to suppress erroneous detection of the subject. As a result, it is possible to efficiently select dictionary data when performing subject detection.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and variations are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

100 Imaging device 111 Imaging sensor 201 System control unit 204 Subject detection unit 205 Dictionary estimation unit 206 Dictionary data storage unit

Claims (15)

an estimation means for estimating a likelihood of the presence or absence of a subject for each of a plurality of dictionary data used for detecting the subject based on captured image data;
an object detection means for detecting an object from the image data by using dictionary data set according to the estimated likelihood;
A control means for switching between a plurality of dictionary data to be set in the object detection means;
Equipped with
The image processing device according to the present invention, wherein the control means switches between a plurality of dictionary data other than the dictionary data whose likelihood is less than a predetermined threshold value . an estimation means for estimating a likelihood of the presence or absence of a subject for each of a plurality of dictionary data used for detecting the subject based on captured image data;
an object detection means for detecting an object from the image data by using dictionary data set according to the estimated likelihood;
A control means for switching between a plurality of dictionary data to be set in the object detection means ;
Equipped with
The image processing device according to claim 1, wherein the control means changes either one or both of an order and a cycle for switching between a plurality of dictionary data sets set in the subject detection means in accordance with the likelihood . 2 . The image processing apparatus according to claim 1 , wherein the control means changes, in accordance with the likelihood, either one or both of an order and a cycle for switching between the plurality of dictionary data sets set in the subject detection means. 4. The image processing apparatus according to claim 1 , wherein the control means increases the priority of the dictionary data in the switching order as the likelihood increases. 4. The image processing apparatus according to claim 1 , wherein the control means shortens a switching period for the dictionary data as the likelihood increases. 4. The image processing apparatus according to claim 1 , wherein the control means increases the priority of the switching order of the dictionary data and shortens a switching cycle as the likelihood increases. 7. The image processing apparatus according to claim 1 , wherein the control means changes a threshold value of a reliability when the object detection means detects the object, in accordance with the likelihood. 8. The image processing apparatus according to claim 1, further comprising: a trimming process for the image data when the object is not detected by the object detection means. 9. The image processing apparatus according to claim 1, wherein a processing cycle of said subject detection means is shorter than a processing cycle of said estimation means. The estimation means estimates the likelihood using a trained model;
The image processing device according to claim 1 , wherein the object detection means detects the object using a trained model different from the trained model. 11. The image processing device according to claim 10 , wherein the trained model of the estimation means and the trained model of the object detection means are convolutional neural networks. An imaging unit;
An image processing device according to any one of claims 1 to 11 ,
An imaging device comprising: estimating a likelihood of the presence or absence of an object for each of a plurality of dictionary data used for detecting the object based on the captured image data;
detecting a subject from the image data by using dictionary data set according to the estimated likelihood;
a control step of switching between a plurality of dictionary data sets to be set in the step of detecting a subject from the image data;
Equipped with
The image processing method , wherein the control step switches between a plurality of dictionary data other than the dictionary data whose likelihood is less than a predetermined threshold value . estimating a likelihood of the presence or absence of an object for each of a plurality of dictionary data used for detecting the object based on the captured image data;
detecting a subject from the image data by using dictionary data set according to the estimated likelihood;
a control step of switching between a plurality of dictionary data sets to be set in the step of detecting a subject from the image data;
Equipped with
The image processing method is characterized in that the control step changes either one or both of the order and period of switching between multiple dictionary data set in a step of detecting a subject from the image data, depending on the likelihood. 12. A program for causing a computer to execute each unit of the image processing apparatus according to claim 1 .

Images