JP7217471B2 - Imaging device - Google Patents

Description

The present disclosure relates to an imaging device that captures an image while acquiring sound.

2. Description of the Related Art Techniques for clearly picking up the sound of a specific subject when shooting a moving image with an imaging device have been studied (for example, Patent Document 1).

Patent Literature 1 discloses a speech identification device comprising an imaging unit and a microphone array. This sound identification device detects feature information of a subject image from image data generated by an imaging unit, and detects sound feature information from sound data generated by a microphone array. This sound identification device adjusts the directional characteristics of the microphone array based on the distance of the subject calculated from the image data and the distance of the sound source calculated from the sound data, thereby intermittently generating sounds. We are also trying to obtain good sound for

JP 2010-154260 A

However, during shooting, the user changes the direction of the shooting device by following the moving subject with his or her eyes, and it is difficult to accurately follow the directivity of the microphone based on the sound detection result. In the prior art, there is a problem that it is difficult to clearly obtain the sound of a specific subject in an imaging device.

The present disclosure provides an imaging device that can facilitate obtaining the voice of a subject clearly according to the user's intention.

An imaging device according to an aspect of the present disclosure includes an imaging unit, an audio acquisition unit, a detection unit, an audio processing unit, and an operation unit. The imaging unit captures a subject image and generates image data. The audio acquisition unit acquires audio data representing audio being captured by the imaging unit. The detection unit detects a subject and its type based on the image data generated by the imaging unit. The audio processing unit processes the audio data acquired by the audio acquisition unit based on the type of subject detected by the detection unit. The operation unit selects a target type to be processed by the audio processing unit from among a plurality of types including a first type and a second type different from the first type, based on the user's operation of the device. set. The audio processing unit processes the audio data so that, when the subject of the target type is detected in the image data, the audio corresponding to the target type is emphasized or suppressed in the acquired audio data.

An imaging device according to another aspect of the present disclosure includes an imaging unit, an audio acquisition unit, a detection unit, a display unit, an operation unit, an audio processing unit, and a control unit. The imaging unit captures a subject image and generates image data. The audio acquisition unit acquires audio data representing audio being captured by the imaging unit. The detection unit detects a subject and its type based on the image data generated by the imaging unit. The display unit displays an image indicated by the image data. The operation unit selects a subject to be focused in the image from the subjects detected by the detection unit based on the user's operation of the device. The audio processing unit processes the audio data acquired by the audio acquiring unit based on the subject type selected by the operation unit. The control unit causes the display unit to display target type information indicating the type of a subject to be focused as a target type to be processed by the audio processing unit.

According to the imaging device according to the present disclosure, it is possible to easily obtain the voice of the subject clearly according to the user's intention.

FIG. 1 shows a configuration of a digital camera according to Embodiment 1 of the present disclosure; Block diagram showing configuration of audio processing engine in digital camera A diagram explaining an example of data of a specific type in the audio extraction unit of the audio processing engine FIG. 3 is a diagram explaining an example of data of a type different from that in FIG. 3 in the voice extraction unit; FIG. 2 is a diagram for explaining an outline of the human priority mode of the digital camera according to the first embodiment; FIG. 4 is a flowchart illustrating the operation of the digital camera according to the first embodiment; A diagram explaining an example of operation when a "person" moves in the human priority mode of a digital camera. FIG. 5 is a diagram for explaining an outline of a focus priority mode of a digital camera according to Embodiment 2; 4 is a flowchart illustrating the operation of the digital camera according to the second embodiment; Flowchart illustrating the operation of the digital camera continued from FIG. Diagram explaining an operation example of a digital camera when a subject to be focused moves A diagram explaining an example of how a digital camera operates in response to a user operation for changing the focus target. FIG. 12 shows a configuration of a digital camera according to Embodiment 3; 12 is a flowchart illustrating the operation of the digital camera according to the third embodiment;

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art. It is noted that the inventor(s) provide the accompanying drawings and the following description in order for those skilled in the art to fully understand the present disclosure, which are intended to limit the claimed subject matter. not something to do.

(Embodiment 1)
In Embodiment 1, as an example of an imaging device according to the present disclosure, a digital camera that clearly obtains the voice of a specific type of subject such as a person or an animal by linking image recognition technology and voice extraction technology will be described.

[1-1. composition〕
A configuration of a digital camera according to Embodiment 1 will be described with reference to FIG.

FIG. 1 is a diagram showing the configuration of a digital camera 100 according to this embodiment. The digital camera 100 of this embodiment includes an image sensor 115 , an image processing engine 120 , a display monitor 130 and a controller 135 . Digital camera 100 further includes buffer memory 125 , card slot 140 , flash memory 145 , operation unit 150 and communication module 155 . The digital camera 100 also includes a microphone 160 , an analog/digital (A/D) converter 165 for the microphone, and an audio processing engine 170 . The digital camera 100 also includes an optical system 110 and a lens driving section 112, for example.

The optical system 110 includes a focus lens, a zoom lens, an optical image stabilization lens (OIS), an aperture, a shutter, and the like. The focus lens is a lens for changing the focus state of the subject image formed on the image sensor 115 . A zoom lens is a lens for changing the magnification of a subject image formed by an optical system. Each focus lens or the like is composed of one or a plurality of lenses.

A lens driving unit 112 drives the focus lens and the like in the optical system 110 . The lens drive unit 112 includes a motor and moves the focus lens along the optical axis of the optical system 110 under the control of the controller 135 . A configuration for driving the focus lens in the lens driving section 112 can be realized by a DC motor, a stepping motor, a servo motor, an ultrasonic motor, or the like.

The image sensor 115 captures the subject image formed via the optical system 110 and generates captured data. The imaging data constitute image data representing an image captured by the image sensor 115 . Image sensor 115 generates new frames of image data at a predetermined frame rate (eg, 30 frames/second). A controller 135 controls the generation timing of imaging data and the electronic shutter operation in the image sensor 115 . The image sensor 115 can use various image sensors such as a CMOS image sensor, a CCD image sensor, or an NMOS image sensor.

The image sensor 115 performs a moving image, a still image imaging operation, a through image imaging operation, and the like. The through image is mainly a moving image, and is displayed on the display monitor 130 for the user to decide the composition for capturing a still image, for example.
A through image, a moving image, and a still image are examples of captured images in this embodiment. The image sensor 115 is an example of an imaging unit in this embodiment.

The image processing engine 120 generates image data by performing various types of processing on the imaging data output from the image sensor 115 , and performs various types of processing on the image data to generate an image to be displayed on the display monitor 130 . to generate. Various types of processing include, but are not limited to, white balance correction, gamma correction, YC conversion processing, electronic zoom processing, compression processing, and expansion processing. The image processing engine 120 may be configured by a hardwired electronic circuit, or may be configured by a microcomputer, processor, or the like using a program.

In this embodiment, the image processing engine 120 includes an image recognition unit 122 that realizes a function of detecting various types of subjects such as people and animals by image recognition of captured images. Details of the image recognition unit 122 will be described later.

The display monitor 130 is an example of a display that displays various information. For example, the display monitor 130 displays an image (through image) represented by image data captured by the image sensor 115 and image-processed by the image processing engine 120 . The display monitor 130 also displays menu screens and the like for the user to make various settings for the digital camera 100 . The display monitor 130 can be composed of, for example, a liquid crystal display device or an organic EL device.

The operation unit 150 is a general term for hard keys such as operation buttons and operation levers provided on the exterior of the digital camera 100, and receives operations by the user. Operation unit 150 includes, for example, a release button, a mode dial, and a touch panel. Upon receiving an operation by the user, the operation unit 150 transmits an operation signal corresponding to the user's operation to the controller 135 .

The controller 135 centrally controls the operation of the entire digital camera 100 . The controller 135 includes a CPU and the like, and the CPU executes a program (software) to realize a predetermined function. Instead of the CPU, the controller 135 may include a processor composed of dedicated electronic circuits designed to implement predetermined functions. That is, the controller 135 can be implemented with various processors such as CPU, MPU, GPU, DSU, FPGA, ASIC, and the like. Controller 135 may be comprised of one or more processors. Also, the controller 135 may be composed of one semiconductor chip together with the image processing engine 120 and the like.

A buffer memory 125 is a recording medium that functions as a work memory for the image processing engine 120 and the controller 135 . The buffer memory 125 is implemented by a DRAM (Dynamic Random Access Memory) or the like. Flash memory 145 is a non-volatile recording medium. Also, although not shown, the controller 135 may have various internal memories, such as a ROM. Various programs executed by the controller 135 are stored in the ROM. Also, the controller 135 may incorporate a RAM that functions as a work area for the CPU.

The card slot 140 is a means into which a removable memory card 142 is inserted. The card slot 140 can electrically and mechanically connect a memory card 142 . The memory card 142 is an external memory internally provided with a recording element such as a flash memory. The memory card 142 can store data such as image data generated by the image processing engine 120 .

The communication module 155 is a communication module (circuit) that performs communication conforming to the communication standard IEEE802.11, the Wi-Fi standard, or the like. Digital camera 100 can communicate with other devices via communication module 155 . The digital camera 100 may directly communicate with another device via the communication module 155, or may communicate via an access point. Communication module 155 may be connectable to a communication network such as the Internet.

Microphone 160 is an example of a sound pickup unit that picks up sound. The microphone 160 converts the collected sound into an analog signal, which is an electric signal, and outputs the analog signal. Microphone 160 may be composed of one or more microphone elements.

The microphone A/D converter 165 converts the analog signal from the microphone 160 into digital audio data. The A/D converter 165 for the microphone is an example of the voice acquisition section in this embodiment. It should be noted that microphone 160 may include a microphone element external to digital camera 100 . In this case, the digital camera 100 is provided with an interface circuit for the external microphone 160 as an audio acquisition section.

The audio processing engine 170 receives audio data output from an audio acquisition unit such as the A/D converter 165 for the microphone, and applies various audio processing to the received audio data. The audio processing engine 170 is an example of an audio processing unit in this embodiment. The audio processing engine 170 may be implemented integrally with the image processing engine 120 . The details of the configuration of the audio processing engine 170 will be described later.

[1-1-1. Image Recognition Unit]
Details of the image recognition unit 122 in this embodiment will be described below.

The image recognition unit 122 employs a trained model by a neural network such as a convolutional neural network. The image recognition unit 122 inputs the captured data from the image sensor 115 to the learned model and executes image recognition processing using the model. The image recognition unit 122 outputs detection information indicating the detection result of the subject type by image recognition processing. The image recognition unit 122 is an example of a detection unit in this embodiment. The image recognition unit 122 may be configured by cooperation between the image processing engine 120 and the controller 135 .

The image recognition processing of the image recognition unit 122 corresponds to position information indicating an area in which a subject classified into one of a plurality of preset categories is shown in an image indicated by data input to the trained model. Output as detection information by associating with categories. The multiple categories include types such as "people" and "animals." Also, each category may be further subdivided and may include, for example, human parts such as human body, face and eyes, and animal parts such as animal body, face and eyes. The position information is defined, for example, by horizontal coordinates and vertical coordinates on the image to be processed, and indicates, for example, a rectangular area surrounding the detected subject (see FIG. 5, etc.).

The image recognition unit 122 may simultaneously detect up to a preset maximum number of subjects for each category. In addition, the category (or type) of animals described above may be further classified according to the type of animal. For example, categories such as dogs, cats, and birds may be set separately, or a category combining dogs and cats may be set. A case will be described below in which a plurality of types preset in the digital camera 100 include the type "person" as an example of the first type and the type "cat" as an example of the second type.

The trained model of the image recognition unit 122 as described above can be obtained, for example, by supervised learning using, as teacher data, image data associated with a correct label indicating that an image of a subject in each category is correct. A trained model may generate a confidence or likelihood for each category of detection results.

The trained model of the image recognition unit 122 is not limited to a neural network, and may be machine learning models related to various image recognitions. Also, the image recognition unit 122 may adopt various image recognition algorithms without being limited to machine learning. Further, the image recognition unit 122 may be configured such that detection of some categories, such as human faces and eyes, is performed by rule-based image recognition processing.

[1-1-2. About the audio processing engine]
Details of the configuration of the audio processing engine 170 will be described with reference to FIGS. 2 to 4. FIG. FIG. 2 is a block diagram showing the configuration of the audio processing engine 170 in the digital camera 100. As shown in FIG.

The audio processing engine 170 includes, for example, a noise suppression unit 172, an audio extraction unit 174, and an enhancement processing unit 176 as shown in FIG. The audio processing engine 170 receives audio data Ain from the microphone A/D converter 165 and performs audio processing using various functions. The voice extraction unit 174 and the enhancement processing unit 176 are controlled by the controller 135, for example.

The noise suppression unit 172 performs noise suppression processing in the audio data Ain input to the audio processing engine 170 . The processing by the noise suppression unit 172 is performed to suppress predetermined noise such as wind noise, lens drive noise, and various handling noises generated when the user touches the digital camera 100, and is implemented by, for example, a rule-based algorithm. be done. The noise suppression unit 172 outputs the processed audio data A10 to the audio extraction unit 174 and the enhancement processing unit 176 . The audio data A10 after processing by the noise suppression unit 172 indicates, for example, video audio obtained when shooting a video without performing audio extraction.

The audio extracting unit 174 extracts audio of a specific type (hereinafter sometimes referred to as “target type”) from the audio data A10 of the video audio from the noise suppressing unit 172, and extracts audio data representing the extracted audio. Output A11. The processing of the speech extraction unit 174 is realized by a machine learning model such as a neural network, for example. An example using a convolutional neural network (CNN) is described below.

The CNN of the speech extraction unit 174 includes, for example, a convolution layer that receives image data as in the case of image recognition. In this example, the audio extraction unit 174 includes an audio/image conversion unit 174a that converts audio data of video audio into image data, and a CNN to identify a portion corresponding to a specific type on the converted image data. It comprises a CNN processing unit 175 that executes processing, and an image/audio conversion unit 174b that converts image data of the identified portion into audio data. The audio extractor 174 can operate periodically, for example, at a predetermined frame period.

A plurality of types that can be set as target types are set in the audio processing engine 170 in advance. A plurality of types in the audio processing engine 170 correspond to a plurality of types preset in the image recognition unit 122, for example. FIG. 3 is a diagram explaining an example of data of a specific type in the voice extraction unit 174. As shown in FIG.

FIG. 3A exemplifies the waveform of the voice of voice data A12 of a human voice as an example of data of the type "person". FIG. 3B illustrates image data B12 after conversion of FIG. 3A. The audio data A12 constitutes time-series data in which the amplitude of the audio waveform is defined along the time direction, as exemplified in FIG. 3(A). The audio/image conversion unit 174a calculates, for example, a short-time Fourier transform (STFT) or the like to generate image data B12 after conversion of the audio data A12.

As shown in FIG. 3B, the converted image data B12 represents the spectrogram or voiceprint image of the audio data A12, and has the frequency direction Y in addition to the time direction X. As shown in FIG. The pixel value of the image data B12 indicates the strength (amplitude) of the sound component defined by the (X, Y) coordinates. The area on the image of the image data B12 represents the corresponding time section and frequency band components in the audio data A12 before conversion.

FIG. 4 is a diagram explaining an example of data of a type different from that of FIG. FIG. 4A exemplifies the waveform of the voice by the voice data A13 of the meow of a cat as an example of the data of the type "cat". FIG. 4B illustrates image data B13 after conversion of FIG. 4A. Between the image data B12 and B13 shown in FIGS. 3(B) and 4(B), different feature amounts is included. According to the machine learning of the CNN processing unit 175, such a feature quantity identification method is obtained.

For example, image data B12 and B13 corresponding to various types of voices are labeled and image data serving as a teacher is stored in a teacher database (DB) 40 for machine learning of the CNN processing unit 175 . In the supervised learning using the teacher DB 40, the trained model of the CNN processing unit 175 converts the weight parameter group of the CNN into the input data by backpropagation so that when image data is input, identification information of a specific type is output. and by adjusting to reduce the error of the training data. Note that the teacher DB 40 may store audio data instead of image data. In this case, the conversion of the audio/image converter 174a can be applied to the audio data in the teacher DB 40 as well.

The identification information output by the CNN processing unit 175 includes, for example, image data indicating an area identified as corresponding to a specific type on the input image data, and may also include the reliability or likelihood of this identification. . The CNN processing unit 175 may include, for example, in addition to or instead of the above-described CNN, various generation models for generating image data or the like corresponding to each type of voice. The CNN processing unit 175 can use trained models that have undergone machine learning separately for each type. For example, each type of trained model or corresponding set of weighting parameters may be stored in a learning database (DB) 45 in flash memory 145 and used by controller 135 as setting information for performing a particular type of speech extraction. It is set in the CNN processing unit 175 . The CNN processing unit 175 may use a trained model that simultaneously identifies a plurality of types.

Returning to FIG. 2, the image/audio conversion unit 174b performs, for example, the inverse conversion of STFT by the audio/image conversion unit 174a on the image data identified by the CNN processing unit 175, and the extraction by the audio extraction unit 174 is performed. Speech data A11 of extracted speech indicating the result is generated.

The enhancement processing unit 176 includes an audio amplification unit 177 that receives the audio data A11 of the extracted audio from the audio extraction unit 174, an audio attenuation unit 178 that receives the audio data A10 of the video audio from the noise suppression unit 172, and an audio amplification unit. A voice combiner 179 that integrates the outputs of the unit 177 and the voice attenuator 178 is provided. The enhancement processing unit 176 processes the audio data A10 and A11 of the extracted audio and the video audio so that the audio extracted by the audio extraction unit 174 is emphasized from the video audio, and outputs the audio data as a result of processing by the audio processing engine 170. Output Aout.

The audio amplifier 177 multiplies the input audio data A11 by a gain G1 set by the controller 135, for example, to amplify the extracted audio. The audio attenuator 178 applies a gain G0 (< 1) is multiplied to suppress the moving image sound. The audio combiner 179 synchronously synthesizes the amplified extracted audio and the suppressed video audio to generate audio data Aout as a processing result.

Note that the gain G<b>0 of the sound attenuator 178 may be calculated by the enhancement processor 176 or may be set by the controller 135 . The gain G1 of the audio amplifier 177 may be 1 or less, for example. Even in this case, since the same audio as the extracted audio is included in the video audio, the extracted audio is amplified more than the video audio in the audio data Aout resulting from the processing. It will happen.

In the speech processing engine 170 as described above, the function of the speech extraction unit 174 is not limited to the CNN, and may be realized by other neural networks, or may be machine learning models related to various speech identifications other than neural networks. good. Further, the machine learning of the voice extraction unit 174 using the teacher DB 40 and the like may be performed in advance before being mounted on the digital camera 100 . In this case, if the learning result DB 45 is recorded in the flash memory 145 of the digital camera 100, the teacher DB 40 need not be recorded.

Further, in the speech processing engine 170, the extraction result of the speech extraction unit 174 may be corrected using a database such as the teacher DB 40 that includes speech data associated with various types. For example, the database may be stored in the flash memory 145, and the speech processing engine 170 may compare the extraction result of the speech extraction unit 174 with the data in the database. Also, the functions of the voice extraction unit 174 and the like may be realized by various voice recognition algorithms without being limited to machine learning, and searches in databases as described above may be used.

[1-2. motion〕
The operation of the digital camera 100 configured as above will be described. The operation of the digital camera 100 when shooting a moving image will be described below.

The digital camera 100 sequentially captures subject images formed via the optical system 110 with the image sensor 115 to generate captured data. The image processing engine 120 generates image data by performing various types of processing on the imaging data generated by the image sensor 115 , and records the image data in the buffer memory 125 . Also, the image recognition unit 122 of the image processing engine 120 detects the type and area of the subject based on the image represented by the image data, and outputs detection information D1 to the controller 135, for example.

Simultaneously with the above imaging operation, the digital camera 100 picks up sound with the microphone 160 . The voice processing engine 170 processes voice data as a result of sound pickup from the microphone A/D converter 165 . The audio processing engine 170 records the processed audio data Aout in the buffer memory 125 .

The controller 135 synchronizes the image data received from the image processing engine 120 and the audio data received from the audio processing engine 170 via the buffer memory 125 and records the moving image in the memory card 142 . Also, the controller 135 sequentially causes the display monitor 130 to display a through image. The user can check the shooting composition and the like at any time using the through image on the display monitor 130 . The motion picture shooting operation is started/finished according to the user's operation on the operation unit 150 .

Moving image shooting by the digital camera 100 as described above may be performed by focusing on a specific type of subject such as a "person" or an "animal". In this case, it is conceivable that there is a need to clearly collect utterances of the above types also for speech.

The digital camera 100 of this embodiment detects the type of subject by the detection information D1 of the image recognition unit 122 in the image processing engine 120, and when a specific type of subject is detected by image recognition, the sound processing engine 170 Perform voice extraction processing for the type. In this manner, the image recognition of the image processing engine 120 and the sound extraction of the sound processing engine 170 are interlocked, and the sound of a specific type of subject can be accurately extracted.

An example of operation of the digital camera 100 in an operation mode (hereinafter referred to as "person priority mode") in which the specific type is set to "person" will be described below.

[1-2-1. About people priority mode]
FIG. 5 is a diagram for explaining an outline of the human priority mode of the digital camera 100. As shown in FIG. The person-priority mode is an operation mode for performing moving image shooting or the like while focusing on a subject whose type is "person".

FIG. 5A shows an example of display on the display monitor 130 in the person priority mode. The controller 135 of the digital camera 100 displays the through image on the display monitor 130 together with the detection area R1 where the subject is detected by frame display or the like in the through image. Further, in the example of FIG. 5, the display monitor 130 displays the voice extraction icon 5. FIG. The audio extraction icon 5 includes a target type mark 5a indicating the type of audio extraction target, and an amplification level bar 5b indicating the level at which the extracted audio is amplified. The target type mark 5a (an example of target type information) and the amplification level bar 5b (an example of emphasis level information) are displayed under the control of the controller 135, respectively. In the voice extraction icon 5 in the person-priority mode, a "person" mark is displayed as the target type mark 5a.

In the digital camera 100 in human priority mode, the image recognition unit 122 of the image processing engine 120 detects, for example, various types of subjects. In the example of FIG. 5A, as subjects, persons 21 and 22 of the target type and a cat 20 of a type different from the target type are detected. At this time, the voice extracting unit 174 of the voice processing engine 170 operates according to the detection of the persons 21 and 22 by image recognition, and starts voice extraction processing for the target type "person".

FIG. 5B is a graph illustrating voice changes corresponding to FIG. 5A. In FIG. 5B, the horizontal axis indicates time and the vertical axis indicates amplification (or suppression) level. A curve C1 represents the extracted audio, and a curve C0 represents the video audio. When one of the persons 21 and 22 vocalizes and the speech of the type "person" is extracted, the enhancement processing unit 176 amplifies the extracted speech. On the other hand, the cry of "cat" is not subject to speech extraction. In this way, the voice of the target type "person" intended by the user is clearly obtained with priority over other voices.

Further, the enhancement processing unit 176 of the voice processing engine 170 gradually and gently increases the extracted voice as indicated by the curve C1 in FIG. 5(B). As a result, it is possible to avoid sudden voice changes that make it difficult for the user to hear the emphasized voice. Also, the audio processing engine 170 amplifies the extracted audio and suppresses the video audio so that the total volume is kept constant before and after the processing by the enhancement processing unit 176 . This makes it easier for the user to hear the emphasized voice. Also, the user can confirm the level of amplification of the extracted voice with the amplification level bar 5b in FIG. 5(A). Furthermore, the target type mark 5a allows the user to confirm the current target type, thereby facilitating realization of voice enhancement in accordance with the user's intention.

[1-2-2. Operation details]
Details of the operation of the digital camera 100 in the human priority mode as described above will be described with reference to FIGS. 6 and 7. FIG. For example, when the user intends to clearly obtain the voice of a subject of type "person", the user inputs a user operation to the operation unit 150 such as a touch panel and various keys in the setting menu of the digital camera 100, etc. 100 can be set to people-first mode.

FIG. 6 is a flow chart illustrating the operation of the digital camera 100 according to the first embodiment. The flowchart shown in FIG. 6 is executed, for example, while the digital camera 100 is set to the people-priority mode while shooting a moving image. In this state, the display monitor 130 is controlled by the controller 135 to display the target type mark 5a indicating the type "person" and the like. Each process according to this flowchart is executed by the controller 135 of the digital camera 100, for example. It should be noted that, instead of the controller 135, the audio processing engine 170 may be implemented with a function for executing each of the following processes.

First, the controller 135 acquires the detection information D1 from the image processing engine 120, and determines whether or not a subject whose type is "human" has been detected in the image recognition section 122 (S1). The controller 135 repeats the determination of step S1 at predetermined intervals, for example, until a subject of type "person" is detected (NO in S1). The cycle is, for example, the operation cycle of the image recognition section 122 in the image processing engine 120 .

In step S1, the audio processing engine 170 generates audio data A10 processed by the noise suppressor 172 without executing the processing of the audio extractor 174 (FIG. 2), and specifically suppresses the audio data A10 by the enhancement processor 176. output to the buffer memory 125 (G0=1).

When an object of type "person" is detected in image recognition (YES in S1), the controller 135 controls the audio processing engine 170 so as to start extracting audio with "person" as the target type (S2). . The controller 135 refers to the learning DB 45 and sets the setting information for extracting the voice of the target type “person” in the voice extraction unit 174 of the voice processing engine 170 . Also, the controller 135 sets, for example, the gain G1 of the audio amplification section 177 in the enhancement processing section 176 to an initial value. The initial value of the gain G1 is set to a value assumed that the user does not perceive a sudden volume change.

The controller 135 determines whether or not the sound of the target type has been extracted by the sound extractor 174 of the sound processing engine 170 (S3). The determination in step S3 is made based on the reliability of the identification information output from the CNN processing unit 175 of the voice extraction unit 174, for example. The controller 135 repeats the determination in step S1 at predetermined intervals, for example, until it determines that the target type "human" voice has been extracted (NO in S3). The cycle is, for example, the operating cycle of the voice extractor 174 in the voice processing engine 170 .

In the sound processing engine 170 after step S2, when the sound extraction unit 174 extracts the target type sound, the sound amplification unit 177 of the enhancement processing unit 176 amplifies the extracted sound. At this time, the audio amplification unit 177 sequentially uses the set gain G1. For example, the initial gain G1 is applied to the voice extracted in step S3. Also, the sound attenuation unit 178 of the enhancement processing unit 176 uses the gain G0 of a value that maintains the volume according to the gain G1 set in the sound amplification unit 177 .

When the controller 135 determines that the voice of the target type “human” has been extracted (YES in S3), it increases the gain G1 of the voice amplifying section 177 from the initial value (S4). This causes the increased gain G1 to be applied to the next extracted speech. In step S4, the gain G1 may be increased at a predetermined pitch, or may be increased continuously. Also, in step S4, the controller 135 controls the display monitor 130 so as to increase the level indicated by the amplification level bar 5b as the gain G1 increases (see FIGS. 5A and 5B).

Next, the controller 135 acquires the detection information D1 again from the image recognition unit 122, and determines whether or not a subject of the target type "person" is detected at this time (S5). The determination in step S5 is made in the same manner as in step S1.

When the controller 135 determines that a subject of the target type “person” has been detected (YES in S5), the controller 135 determines whether or not the voice of the target type has been extracted by the voice extraction unit 174 at the present time in the same manner as in step S3. (S6).

If the target type of sound has been extracted (YES in S6), the controller 135 determines whether or not the gain G1 set in the sound amplifying section 177 is the maximum value (S7). The maximum value is set, for example, to a value at which the user feels that the extracted speech is sufficiently emphasized. If the set gain G1 has not reached the maximum value (NO in S7), the controller 135 increases the gain G1 of the audio amplifier 177 again (S4), and repeats the processing from step S5. This applies a further increased gain G1 to the newly extracted speech.

On the other hand, if the gain G1 is the maximum value (YES in S7), the controller 135 does not perform the process of step S4, but performs the processes after step S5 again. Thereby, the amplification for emphasizing the extracted speech in the speech processing engine 170 can be maintained at an appropriate gain G1.

In addition, if the controller 135 does not detect a subject whose type is “human” at the present time (NO in S5) or does not extract the sound of the target type (NO in S6), the controller 135 sets the gain of the sound amplification unit 177 to G1 is decreased (S8). The processing of step S8 is performed at the same pitch as step S4, for example. Also, in step S8, the controller 135 controls the display monitor 130 so as to lower the level indicated by the amplification level bar 5b as the gain G1 decreases (see FIGS. 7A and 7B).

Also, the controller 135 determines whether the reduced gain G1 is the minimum value, for example (S9). The minimum value of gain G1 may be, for example, the same value as the initial value. If the gain G1 has not reached the minimum value (NO in S9), the controller 135 repeats the processing after step S5. As a result, when the sound extraction unit 174 extracts the sound of the target type in subsequent step S6, the sound is amplified by applying the reduced gain G1. On the other hand, when the gain G1 reaches the minimum value (YES in S9), the controller 135 stops the voice extraction processing by the voice extraction unit 174 (S10), and returns to step S1.

The above processing is repeatedly executed during video shooting in the human-priority mode of the digital camera 100, for example. Audio data Aout after audio processing is recorded as moving image recording.

According to the above processing, extraction of voice and amplification of the extracted voice are executed in conjunction with image recognition of a specific type such as "person".

For example, in the example of FIG. 5B, before time t1, no subject of type "person" has been detected (NO in S1), and no sound has been extracted and amplified/suppressed. In this way, even in human-priority mode, if a subject of type "human" is not detected by image recognition, audio processing for the target type is not performed, thereby avoiding unnecessary reduction of video audio. can.

When the image recognition unit 122 detects a subject of type “human” (YES in S1), and when the sound extraction unit 174 starts extracting elephant-type sound (YES in S3), the controller 135 controls the sound amplification unit. The gain G1 of 177 is gradually increased (S2-S7). As a result, the enhancement of the extracted voice gradually progresses from time t1 in FIG. 5B, and it is possible to obtain a voice that is easy for the user to listen to even before or after the timing of starting amplification.

7A and 7B are diagrams for explaining an operation example when a "person" moves in the person priority mode of the digital camera 100. FIG. In the digital camera 100 in the person-priority mode, voice extraction continues as long as even one person is recognized as a type "person".

FIG. 7A shows a display example after FIG. 5A. FIG. 7B illustrates voice changes corresponding to FIG. 7A. In this example, none of the persons 21 and 22 are present, and the type "person" is no longer detected in image recognition (NO in S5). At this time, the voice extraction is not stopped immediately, but the controller 135, for example, gradually decreases the gain G1 of the voice amplifier 177 (S7, S8).

FIG. 7C shows a display example after FIG. 7A. FIG. 7(D) illustrates voice changes corresponding to FIG. 7(C). In the example of FIG. 7C, the person 22 is detected again after FIG. 7A, and the gain G1 of the audio amplifier 177 is increased again (S4 to S8). As described above, even in a situation where the subject such as the person 22 is moving, it is possible to obtain clearer voice without abrupt changes in the extracted voice.

In the above description, an example of emphasizing the sound of the target type has been described, but instead of this, the sound of the target type may be suppressed. For example, if the user wishes to suppress human speech, the user selects an operating mode instead of the human priority mode described above. In this operation mode, for example, the audio processing engine 170 can suppress the audio of the target type by performing processing in which the audio amplification and audio suppression are interchanged in the flowchart of FIG. As a result, it is possible to realize voice clarification that meets the user's intention, such as suppressing a specific type of voice.

In the above description, an operation example in which the target type is the type "person" has been described, but the same operation is possible for other types as well. For example, the digital camera 100 may have an operation mode that employs each of a plurality of types that can be set for the image recognition unit 122 and the audio processing engine 170 as target types. For example, the operation mode corresponding to the target type desired by the user may be selected by inputting the user's operation from the operation unit 150 while displaying the options of each operation mode on the setting menu on the display monitor 130 .

[1-3. summary〕
As described above, the digital camera 100 of Embodiment 1 includes the image sensor 115 as an example of an imaging unit, the A/D converter 165 for a microphone as an example of a sound acquisition unit, and the image recognition unit 122 as an example of a detection unit. , an audio processing engine 170 as an example of an audio processing unit, and an operation unit 150 . The image sensor 115 captures a subject image and generates image data. The microphone A/D converter 165 acquires audio data Ain representing audio being captured by the image sensor 115 . The image recognition unit 122 detects the subject and its type based on the image data generated by the image sensor 115 . The audio processing engine 170 processes the acquired audio data Ain based on the subject type detected by the image recognition unit 122 . Based on various operations of the digital camera 100 by the user, the operation unit 150 selects, for example, a first type related to people and a second type different from the first type, from among a plurality of types, the audio processing engine 170 Set the target type to be processed. When a subject of the target type is detected in the image data (S1), the voice processing engine 170 controls the voice extraction unit 174 and the enhancement unit 174 so as to emphasize or suppress the voice corresponding to the target type in the acquired voice data Ain. The processing unit 176 processes the audio data Ain (S2 to S4).

According to the digital camera 100 described above, when a specific subject corresponding to the target type desired by the user is detected in the image recognition of the image data by the image sensor 115, the sound corresponding to the specific subject type is emphasized or suppressed. Voice data Aout is obtained. As a result, it is possible to easily obtain the voice of the specific subject clearly according to the user's intention.

In this embodiment, the digital camera 100 further includes a display monitor 130 as an example of a display that displays an image represented by image data. The display monitor 130 displays the target type mark 5a, which is an example of target type information indicating the target type. As a result, the user can shoot the action while confirming the current target type, and can easily acquire the voice of the subject in accordance with the user's intention. Furthermore, the display monitor 130 may display an amplification level bar 5b, which is an example of emphasis level information indicating the level at which the subject's voice is emphasized or suppressed.

In this embodiment, the digital camera 100 includes an operation unit 150 for inputting user's operations. The target type to be processed by the audio processing engine 170 is set based on the user's operation on the operation unit 150 . As a result, it is possible to easily obtain the voice of the type desired by the user.

In this embodiment, the digital camera 100 has a person priority mode according to the type "person" as an example of an operation mode according to the type of subject. The operation unit 150 sets the target type according to the user's operation for selecting the operation mode of the digital camera 100 . For example, when the person priority mode is selected, the target type is set to "person". Note that such an operation mode is not limited to the human-priority mode, and an operation mode that prioritizes various types of animals such as "cat" instead of the type "human" may be used.

In this embodiment, the sound processing engine 170 gradually increases the gain G1, which is an amplification factor for enhancing the sound according to the target type, from when the image recognition unit 122 detects the subject of the target type (YES in S2). (S3 to S7). This makes it easier for the user to listen to the emphasized extracted voice by avoiding sudden voice changes.

In this embodiment, the audio processing engine 170 gradually decreases the gain G1 (S8, S9 ). As a result, it is possible to avoid excessively changing the emphasis of the extracted voice depending on whether or not the subject is detected, and obtain a voice that is easier for the user to listen to.

In the present embodiment, the audio processing engine 170 adjusts the volume between the audio data A10 before the process of enhancing the audio corresponding to the target type and the audio data Aout after the process in the audio extraction unit 174 and the enhancement processing unit 176. The audio data A10 input to the audio attenuator 178 is processed so as to maintain the As a result, it is possible to obtain sound that is easier for the user to listen to without changing the volume before and after the sound processing.

In this embodiment, the digital camera 100 further includes a microphone 160 as an example of a sound pickup unit that picks up sound. A/D converter 165 for microphone acquires audio data Ain indicating the result of sound pickup by microphone 160 . Note that the microphone 160 is not limited to being built in the digital camera 100, and may be configured externally. Even when the external microphone 160 is used, the digital camera 100 acquires audio data as a sound collection result and performs audio processing of the audio processing engine 170 according to the detection result of the image recognition unit 122. It is possible to clearly obtain the voice of a specific type of subject.

The digital camera 100 of this embodiment includes an image sensor 115 (imaging unit), a microphone A/D converter 165 (sound acquisition unit), an image recognition unit 122 (detection unit), and a display monitor 130 (display unit). , an audio processing engine 170 (audio processing unit), an operation unit 150 (operation unit), and a controller 135 (control unit). The operation unit of the present embodiment sets a target type to be processed by the audio processing unit from among a plurality of types based on the user's operation of the setting menu of the device itself. The control unit causes the display unit to display the target type mark 5a as an example of target type information indicating the target type. This also allows the user to shoot the action while confirming the current target type, making it easier to obtain the voice of the subject clearly according to the user's intention.

(Embodiment 2)
Embodiment 2 will be described below with reference to FIGS. 8 to 12. FIG. Embodiment 1 describes an operation example of the digital camera 100 in the human priority mode, but Embodiment 2 describes an operation example of the focus priority mode. The focus priority mode is an operation mode in which the type of subject selected as a focus target in the digital camera 100 is prioritized and audio extraction is performed.

Hereinafter, the digital camera 100 according to the present embodiment will be described, omitting the description of the same configuration and operation as those of the digital camera 100 according to the first embodiment.

[2-1. About focus priority mode]
FIG. 8 is a diagram for explaining the outline of the focus priority mode of the digital camera 100. As shown in FIG. In the digital camera 100 of the present embodiment, the subject to be focused can be selected by user operation on the operation unit 150 such as a touch panel or keys while the subject is displayed in the through image on the display monitor 130, for example.

FIG. 8A shows a display example before focus selection. FIG. 8(B) illustrates voice changes corresponding to FIG. 8(A). FIG. 8C shows a display example after focus selection. FIG. 8(D) illustrates voice changes corresponding to FIG. 8(B).

In the display example of FIG. 8A, three subjects, a cat 20 and two people 21 and 22, are detected by the image recognition unit 122 similar to that of the first embodiment. For example, the user can visually recognize the display frame around each subject corresponding to the detection region R1 on the display monitor 130 to select the subject to be focused. Before the selection of the focus target, audio extraction is not performed in particular, and moving image audio is obtained as indicated by the curve C0 in FIG. 8(B).

The display example of FIG. 8(C) shows an example in which one person 21 is selected as a focus target from the state of FIG. 8(A). The display monitor 130 displays a focus target display frame F1 around the selected person 21 in a display mode different from the display frames of the other subjects 20 and 22 . Further, the lens driving section 112 drives the focus lens of the optical system 110 so as to focus on the subject within the display frame F1.

As shown in FIG. 8D, the audio processing engine 170 of the present embodiment performs audio processing for enhancing audio according to the type of the subject to be focused in conjunction with the above operation. Note that the speech processing engine 170 of the present embodiment is configured such that, for example, the speech extraction unit 174 and the enhancement processing unit 176 can process multiple types of extracted speech in parallel.

[2-2. Operation details]
Details of the operation of the digital camera 100 in the focus priority mode as described above will be described with reference to FIGS. 9 to 12. FIG. 9 and 10 are flowcharts illustrating the operation of the digital camera 100 according to this embodiment. In the following, explanations similar to those of the operation in the person priority mode (FIG. 6) will be omitted as appropriate.

In the digital camera 100 in the focus priority mode, the controller 135 determines whether or not there is a subject detected by image recognition based on the detection information D1 by the image recognition unit 122 instead of step S1 in FIG. S1A). If there is a subject detected (YES in S1A), the controller 135 determines whether or not the subject to be focused has been selected by the user's operation on the operation unit 150 (S1B).

When the subject to be focused is selected (YES in S1B), the controller 135 sets the type of the selected subject as the target type and causes the sound processing engine 170 to start sound extraction in the same manner as in step S2 of FIG. 6 (S2A). ). At this time, the controller 135 causes the display monitor 130 to display the voice extraction icon 5 so that the target type mark 5a indicates the type of the selected subject (see FIG. 8C). Further, the display of the amplification level bar 5b is controlled by the controller 135 in subsequent steps S4 and S8 so as to correspond to the gain G1, as in the first embodiment.

Further, after the audio processing engine 170 amplifies the extracted audio (S3, S4), the controller 135 determines whether or not a user operation to change the focus target subject has been performed on the operation unit 150 instead of step S5 in FIG. (S5A).

If there is no change in the focus target (NO in S5A), the controller 135 acquires the detection information D1 again from the image recognition unit 122, and determines whether the subject selected as the focus target is detected at this time. Judge (S5B). If the subject to be focused is detected in the current image recognition (YES in S5B), the controller 135 performs the processes after step S6 in the same manner as in the first embodiment. FIG. 11 illustrates an operation example when the subject to be focused moves.

FIG. 11A shows a display example after FIG. 8C. FIG. 11(B) illustrates voice changes corresponding to FIG. 11(A). In the example of FIG. 11A, the person 21 selected as the focus target in FIG. The image recognition unit 122 detects the other subjects 20 and 22, but does not detect the person 21 selected as the focus target. In this way, when the subject to be focused is no longer detected (NO in S5B), the controller 135 reduces the gain G1 of the extracted voice as shown by the curve C1 in FIG. 11B, for example (S8).

FIG. 12 illustrates an operation example when there is a user operation to change the focus target (YES in S5A). FIG. 12A shows a display example after FIG. 11A. FIG. 12(B) illustrates voice changes corresponding to FIG. 12(A). The graph of FIG. 12(B) further includes a curve C2 representing extracted speech of type "cat". In step S5A, the controller 135 may proceed to "YES" if, for example, the type of the subject to be focused has changed, and to "NO" if the type of the focused subject has not changed before and after the change.

In the example of FIG. 12A, the cat 20 is selected as a new focus target, and the target type mark 5a of type "cat" is displayed. If there is a change in the focus target type (YES in S5A), the controller 135 determines whether or not there is image recognition of the focus target after the change, as shown in FIG. 10, for example (S20). When there is image recognition of a focus target (YES in S20), the controller 135 sets the type of the focus target as the target type and causes the voice processing engine 170 to start voice extraction in the same manner as in step S2A (S21). At this time, the controller 135 updates, for example, the target type mark 5a displayed in FIG. 11(A) to indicate the new target type as shown in FIG. 12(A).

The controller 135 determines whether or not the voice of the target type after the change has been extracted in the voice extraction unit 174 in the same manner as in step S3 (S22). For example, as shown in FIG. 12B, when the cat 20 barks, an extracted voice of type "cat" is obtained and gradually increases. At this time, audio extraction for the focus target before the change is not stopped immediately.

Hereinafter, the gain of the extracted voice for the target type after the change is referred to as "G1a", and the gain of the extracted voice for the target type before the change is referred to as "G1b". When the voice of the target type after change is extracted (YES in S22), the gain G1a of the target type after change is increased (S23), and the gain G1b of the target type before change is decreased (S24). Also, the gain G0 of the moving image sound is appropriately set according to the respective gains G1a and G1b so that the volume before and after the processing is maintained. The amplification level bar 5b is controlled by the controller 135 so as to correspond to the changed gain G1b, for example.

The controller 135 repeats the processing of steps S22 to S25 (NO in S25) until the gain G1b of the target type before change reaches the minimum value (S25). When the gain G1b reaches the minimum value (YES in S25), the controller 135 stops voice extraction for the target type before change (S26), and returns to step S5A in FIG. 9, for example.

If the controller 135 does not recognize the image of the focus target (NO in S20), the controller 135 proceeds to step S8. This makes it possible to deal with the case where the user performs an operation to focus on an area that is not targeted for image recognition and voice extraction.

According to the above processing, in addition to image recognition, it is possible to realize voice processing that emphasizes the voice of a specific subject in conjunction with the user's selection of a focus target. FIG. 12 illustrates another operation example when there is a user operation to change the focus target (YES in S5A). FIG. 12(C) shows a display example after FIG. 12(A). FIG. 12(D) illustrates voice changes corresponding to FIG. 12(C).

In the example of FIG. 12C, the focus target is switched from cat 20 to person 22 . In this way, even if the cat 20, which was the target type for speech extraction, is continuously detected in image recognition (YES in S5B), if the focus target is switched to the person 22 by the user's operation (YES in S5A) ), voice extraction for the target type of “person” is started in conjunction with the focus (S21). In addition, the voice change at this time is also performed gently as shown in FIG.

In steps S1B and S5A described above, the user operation for selecting the subject to be focused includes, for example, a touch operation on a touch panel or a selection operation using various keys for the detection area R1 of each of the subjects 20 to 22 on the display monitor 130. . In addition, the user operation using the function of automatically selecting the default focus target of the digital camera 100 may be used.

For example, the controller 135 of the digital camera 100 automatically selects a subject located in the center of the entire image or appearing relatively large as a default focus target based on the detection information D1 of the image recognition unit 122. good too. Using such an automatic selection function, the user performs various operations such as changing the direction in which the digital camera 100 is pointed, changing the zoom value, etc., so that the digital camera 100 can focus on a desired subject. can be selected. The result of such selection can be confirmed, for example, by the display mode of the focus target display frame F1. Also in steps S1B and S5A in this case, in the digital camera 100, the type of the subject selected as the focus target can be set as the target type in the same manner as described above. Each unit of the digital camera 100 used for user operations as described above is an example of the operation unit in this embodiment.

[2-3. summary〕
As described above, in the digital camera 100 of the second embodiment, the display monitor 130 further displays information indicating the subject detection result by the image recognition section 122 . The operation unit in the digital camera 100 of the present embodiment sets the target type according to the user's operation of specifying the subject to be focused in the digital camera 100 based on the information displayed by the display monitor 130 . Accordingly, the target type of voice extraction can be dynamically set in accordance with the user's operation, and the voice of the type desired by the user can be clearly obtained.

In this embodiment, the digital camera 100 (imaging device) includes an image sensor 115 (imaging unit), an A/D converter 165 for a microphone (sound acquisition unit), an image recognition unit 122 (detection unit), and a display unit. It includes a monitor 130 (display unit), an audio processing engine 170 (audio processing unit), an operation unit 150 (operation unit), and a controller 135 (control unit). The operation unit of the present embodiment may select a subject to be focused in the image from among the subjects detected by the detection unit based on the user's operation of the own device (S1B). The audio processing unit processes the audio data acquired by the audio acquisition unit based on the subject type selected by the operation unit (S2A to S10). The control unit causes the display unit to display the target type mark 5a as an example of the target type information indicating the type of the subject to be focused as the target type to be processed by the audio processing unit (S2A, FIG. 8C, etc.). . As a result, the user can shoot the action while confirming the current target type, and can easily obtain the voice of the subject clearly according to the user's intention.

In this embodiment, the control unit further causes the display unit to display the amplification level bar 5b as an example of emphasis level information indicating the level of emphasis or suppression of the sound of the subject selected by the sound processing unit (S4, S8, FIG. 8(C), etc.). As a result, the user can confirm the degree to which the sound obtained during shooting of a moving image or the like is emphasized or suppressed, and can easily obtain the sound according to the user's intention.

In this embodiment, when the type of the subject before and after the change is different when the subject to be focused is changed (YES in S5B), the control unit stores the target type information so as to indicate the type after the change as the target type. It may be updated and displayed on the display unit (S21, FIGS. 12A and 12C, etc.). As a result, the user can confirm the subject type that dynamically changes during shooting, and can easily acquire the voice of the subject in accordance with the user's intention.

(Embodiment 3)
Embodiment 3 will be described below with reference to FIGS. 13 and 14. FIG. The digital camera 100 of Embodiments 1 and 2 extracts a specific type of voice in conjunction with image recognition. Embodiment 3 will further describe a digital camera that controls the directivity of sound pickup in conjunction with image recognition.

Hereinafter, the digital camera according to the present embodiment will be described, omitting the description of the same configuration and operation as those of the digital camera 100 according to the first and second embodiments.

[3-1. composition〕
FIG. 13 is a diagram showing the configuration of a digital camera 100A according to the third embodiment. The digital camera 100A of the present embodiment has a configuration similar to that of the digital camera 100 of the first and second embodiments, but includes a plurality of microphones 160A and a beam forming section 162 to generate the directivity of sound to be collected. do. The microphone 160A of this embodiment includes, for example, three or more microphone elements, positioned relative to each other between the elements.

The beam forming unit 162 is, for example, a circuit that adjusts the delay period of each element of the microphone 160A, and forms the sound picked up by the microphone 160A in a desired direction and width. The beam forming unit 162 can set the physical range in which the microphone 160A picks up sound. The beam forming section 162 may be configured integrally with the microphone 160A or the A/D converter 165, or the functions of the beam forming section 162 may be implemented in the audio processing engine 170.

[3-2. motion〕
FIG. 14 is a flow chart illustrating the operation of the digital camera 100A according to the third embodiment. In the digital camera 100A of the present embodiment, the controller 135 performs the same processing as in the first and second embodiments, and the beam forming unit 162 that varies the sound pickup range of the microphone 160A based on the detection information D1 by the image recognition unit 122. is controlled (S30, S31). FIG. 14 illustrates an operation example in which the sound pickup range is dynamically set in the focus priority mode (FIG. 9).

For example, when a subject to be focused is selected (YES in S1B), the controller 135 controls the microphone 160A to pick up sound from the direction of the subject based on the detection information D1 of the image recognition unit 122 at that time. , the beam forming unit 162 is controlled (S30). The beam forming section 162 forms the beam of the microphone 160A according to the position and size of the detection area R1 of the specific subject in the detection information D1. As a result, sound is picked up by the microphone 160A in the sound pickup range according to the image recognition, and target type sound extraction is applied to the sound data in the sound pickup range (S2A).

Also, when the image recognition of the focus target is continued (YES in S5B), the controller 135 sequentially controls the beam forming unit 162 to dynamically set the sound pickup range of the microphone 160A in the same manner as in step S30. (S31). As a result, the sound pickup range of the microphone 160A is changed, for example, when the subject to be focused moves or is changed to another subject.

According to the above processing, the sound pickup range of the microphone 160A is directed toward the subject to be focused according to the detection result of the image recognition unit 122, and the voice from the subject can be obtained more clearly. In the above description, an example in which the sound pickup range of the microphone 160A is controlled by the beam forming unit 162 in the focus priority mode has been described. good too.

[3-3. summary〕
As described above, digital camera 100A of the third embodiment further includes beam forming section 162 . The beam forming unit 162 changes the sound pickup range of the microphone 160A according to the detection result of the image recognition unit 122 . As a result, the voice from the subject detected by the image recognition unit 122 can be obtained more clearly.

(Other embodiments)
As described above, Embodiments 1 to 3 have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, replacements, additions, omissions, etc. are made as appropriate. Also, it is possible to combine the constituent elements described in the first embodiment to form a new embodiment.

In the above first and second embodiments, the human-priority mode and focus-priority mode of the digital camera 100 have been described. Such an operation mode can be set by a user's operation on the operation unit 150. For example, the digital camera 100 may display a menu screen on the display monitor 130 so that the operation mode can be selected.

In each of the above embodiments, the type "person" is used as an example of the first type, and the type "cat" is used as an example of the second type, but the first and second types are not limited to the above. It may be of various types. For example, the second type may be not only "cat" but also various animals such as "dog" or "bird", or may be the type "animal" including various animals other than humans. In addition, not only people or animals, but also objects having unique sounds such as trains or musical instruments may be appropriately adopted as types. Sounds from such objects may be targeted for enhancement/suppression, for example as background sounds. Furthermore, the first type is not limited to an unspecified "person", and may be, for example, a specific individual. In this case, the second type may be an individual different from the first type.

That is, in the present embodiment, the first and second types may be set to various types related to any one of humans, animals other than humans, and objects having background sounds. Also, the plurality of types set in the digital camera 100 may further include types other than the first and second types.

Even with various types as described above, for example, by preparing data sets for image and sound learning according to each type in machine learning, the same operation as in each of the above embodiments can be realized. . Moreover, even with such various types, by associating the types set in the image recognition unit 122 and the sound processing engine 170 with each other, the sound processing engine can be processed in conjunction with the image recognition unit 122 as in the above-described embodiments. At 170, desired types of speech can be enhanced/suppressed. Note that the types set in the image recognition unit 122 and the audio processing engine 170 may not necessarily be the same. For example, the type set in the image recognition unit 122 is more subdivided than the type set in the audio processing engine 170. may be modified. Further, the types set in the image recognition unit 122 may include types that are not set in the voice processing engine 170 because they are not particularly subject to voice processing.

In each of the above-described embodiments, the target type mark 5a is exemplified as an example of the target type information, and the amplification level bar 5b is exemplified as an example of the emphasis level information. In this embodiment, the target type information is not limited to the target type mark 5a, and may be character information such as the name of the target type, or may be an image such as a thumbnail. Further, the emphasis level information is not limited to the amplification level bar 5b, and may be character information such as numbers indicating the level of emphasis or suppression, or a graph such as a pie chart. Also, the target type information and the emphasis level information may be displayed as independent icons.

The digital camera 100 including the image recognition unit 122 has been described in each of the above embodiments. In this embodiment, the image recognition unit 122 may be provided in an external server. In this case, the digital camera 100 may transmit the image data of the captured image to the external server via the communication module 155 and receive the detection information D1 of the processing result by the image recognition section 122 from the external server. In such a digital camera 100, the communication module 155 functions as a detector. Also, the functions of the audio processing engine 170, such as the audio extraction unit 174, may be performed by an external server in the same manner as described above.

Further, in each of the above embodiments, the digital camera 100 including the optical system 110 and the lens driving section 112 has been exemplified. The imaging apparatus of this embodiment may not include the optical system 110 and the lens driving unit 112, and may be an interchangeable lens type camera, for example.

Also, in each of the above embodiments, a digital camera was described as an example of an imaging device, but the present invention is not limited to this. The imaging device of the present disclosure may be an electronic device (for example, a video camera, a smart phone, a tablet terminal, etc.) having an image capturing function.

As described above, the embodiment has been described as an example of the technique of the present disclosure. To that end, the accompanying drawings and detailed description have been provided.

Therefore, among the components described in the attached drawings and detailed description, there are not only components essential for solving the problem, but also components not essential for solving the problem in order to illustrate the above technology. can also be included. Therefore, it should not be immediately recognized that those non-essential components are essential just because they are described in the attached drawings and detailed description.

In addition, the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, replacements, additions, omissions, etc. can be made within the scope of the claims or equivalents thereof.

The present disclosure is applicable to an imaging device that captures an image while acquiring sound.

100, 100A Digital camera 115 Image sensor 120 Image processing engine 122 Image recognition unit 130 Display monitor 135 Controller 150 Operation unit 160, 160A Microphone 162 Beam forming unit 165 A/D converter for microphone 170 Audio processing engine 172 Noise suppression unit 174 Sound Extraction unit 176 Enhancement processing unit

Claims (11)

an imaging unit that captures an image of a subject and generates image data;
an audio acquisition unit that acquires audio data representing audio being captured by the imaging unit;
a detection unit that detects a subject and its type based on the image data generated by the imaging unit;
an audio processing unit that processes the audio data acquired by the audio acquisition unit based on the type of subject detected by the detection unit;
A target type to be processed by the audio processing unit from among a plurality of types including a first type indicating a person and a second type indicating a subject other than the person based on the user's operation of the device. an operation unit for setting the
a display unit for displaying an image indicated by the image data and information indicating a subject detection result by the detection unit;
The operation unit follows a user operation of designating a subject to be focused in the device based on information displayed by the display unit when shooting a moving image, and the type of the subject designated as the focus target by the user operation. is set as the target type,
The audio processing unit is an imaging device that processes audio data acquired when a subject of the target type is detected in the image data so as to emphasize or suppress audio corresponding to the target type in the audio data. . an imaging unit that captures an image of a subject and generates image data;
an audio acquisition unit that acquires audio data representing audio being captured by the imaging unit;
a detection unit that detects a subject and its type based on the image data generated by the imaging unit;
an audio processing unit that processes the audio data acquired by the audio acquisition unit based on the type of subject detected by the detection unit;
A target type to be processed by the audio processing unit is set from among a plurality of types including a first type and a second type different from the first type based on the operation of the device by the user. An imaging device comprising an operation unit for
The imaging device has an operation mode according to the type of the subject,
The operation unit sets the target type according to a user's operation for selecting an operation mode of the imaging device,
The sound processing unit does not emphasize the sound of the type when a subject of a type different from the target type is detected by the detection unit, and when the subject of the target type is detected in the image data, An imaging device that processes acquired audio data so as to emphasize or suppress the audio corresponding to the target type in the acquired audio data. The imaging apparatus according to claim 1, wherein the display unit displays target type information indicating the target type. 3. The imaging apparatus according to claim 1, wherein the first and second types are set to types related to any one of humans, animals other than humans, and objects having background sounds, respectively. 3. The imaging apparatus according to claim 1, wherein the sound processing section gradually increases an amplification factor for emphasizing the sound corresponding to the target type from when the detection section detects a subject of the target type. 6. The imaging apparatus according to claim 5, wherein the sound processing unit gradually decreases the amplification factor when the object of the target type is no longer detected after the detection unit detects the object of the target type. Equipped with a sound pickup part that collects sound,
3. The imaging apparatus according to claim 1, wherein the sound acquisition unit acquires sound data indicating a result of sound pickup by the sound pickup unit. 8. The imaging apparatus according to claim 7, further comprising a beam forming section that changes a range of sound picked up by said sound pickup section according to a detection result of said detection section. an imaging unit that captures an image of a subject and generates image data;
an audio acquisition unit that acquires audio data representing audio being captured by the imaging unit;
a detection unit that detects a subject and its type based on the image data generated by the imaging unit;
a display unit that displays an image indicated by the image data;
an operation unit that selects a subject to be focused in the image from subjects detected by the detection unit based on a user's operation of the device;
an audio processing unit that processes audio data so as to emphasize or suppress audio in the audio data acquired by the audio acquisition unit based on the type of subject selected by the operation unit;
a control unit that causes the display unit to display target type information indicating a target type to be processed by the audio processing unit;
The control unit controls the display unit such that the target type information indicates, as the target type, the type of the subject selected as the focus target by user operation on the operation unit. 10. The imaging apparatus according to claim 9, wherein the control section further causes the display section to display emphasis level information indicating a level at which the sound processing section emphasizes or suppresses the sound of the selected subject. When the type of the subject before and after the change is different when the subject to be focused is changed, the control unit updates the target type information so as to indicate the type after the change as the target type, and the display unit 11. The imaging device according to claim 9 or 10, wherein the image is displayed on the .

Images