JP4515005B2 - Electronic camera - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention generally relates to an electronic camera having a recording function, and more particularly to a technique for realizing a function of associating a subject with sound.
[0002]
[Prior art]
In recent years, not only a digital video camera (DVC) but also an electronic camera called a digital still camera (DSC) having a recording function for recording sound has been developed. With such an electronic camera, a captured image (still image or moving image) can be reproduced and recorded sound can be reproduced, so that the reproduction effect on the captured image can be enhanced. However, here, the electronic camera means not only a DSC but also a DVC having a still image shooting function.
[0003]
As a related prior art, a technique for presenting guidance information using at least one of screen display and audio output in an information processing apparatus having a dialog control technique has been proposed (see, for example, Japanese Patent Laid-Open No. 10-11248). ).
[0004]
Generally, an electronic camera can input a captured image to an information processing apparatus such as a personal computer. The personal computer can display the input captured image on the display screen and can reproduce the sound recorded from the attached speaker.
[0005]
[Problems to be solved by the invention]
In a personal computer, when a photographed image is displayed on the display screen, it is also possible to play back the recorded sound at the timing when the subject position is designated with a pointer (cursor). However, the conventional electronic camera cannot record the subject and the recorded voice in association with each other in the captured image. Although it is possible to add audio position information to an input still image with a personal computer or the like, it is usually necessary to perform advanced image processing.
[0006]
SUMMARY OF THE INVENTION An object of the present invention is to provide an electronic camera having a function capable of generating positional information that associates a subject of a captured image with sound in an electronic camera having a recording function.
[0007]
[Means for Solving the Problems]
An aspect of the present invention obtains (calculates or estimates) position information (pronunciation position information) that associates a subject (person or object) in a captured image with a recorded (detected) sound in an electronic camera having a recording function. The present invention relates to an electronic camera having a function to perform.
[0008]
An electronic camera according to the present invention includes an imaging unit that captures an image of a subject and converts it into image data, an audio detection unit that detects audio and converts it into audio data, and a position that acquires position information related to the position where the audio is generated An information acquisition means; an output means for outputting the position information in association with the image data or the audio data; and a dynamic detection means for detecting the dynamics of a subject using the image data. When it is determined that the subject position is detected from the dynamic detection by the dynamic detection unit , the acquisition unit temporarily stores position information corresponding to the subject position as a pronunciation position of the subject, and the dynamic detection by the dynamic detection unit If it is determined that the subject position cannot be detected from the subject position, the subject position at the substantially central position on the shooting screen is temporarily stored as the pronunciation candidate position .
[0009]
The position information acquisition unit is configured to detect, for example, dynamics of a subject (for example, movement of a person's mouth) and to acquire pronunciation position information in which the detected position is estimated as a voice generation position (sound generation position). The output unit outputs the position information to a recording unit that records the position information on a recording medium or a transmission unit that transmits data by wireless communication or the like.
[0010]
With an electronic camera having such a configuration, it is possible to obtain position information (pronunciation position information) that associates a subject with sound as well as a captured image and recorded sound. The process of performing the reproduction can be realized relatively easily. Thereby, not only simple audio reproduction during image reproduction, but also, for example, by specifying a subject in conversation, audio reproduction of the conversation can be realized, and an excellent reproduction effect can be obtained.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
[0012]
(Configuration of electronic camera)
FIG. 1 is a block diagram illustrating a main part of the electronic camera according to the embodiment. The electronic camera is assumed to be a digital still camera (DSC) having a still image and moving image shooting function, but is also applicable to a digital video camera (DVC) having a still image shooting function.
[0013]
The electronic camera is roughly divided into a photographing system, an image processing system, a control / operation system, a display / output system, and a recording system. The imaging system includes an optical lens unit 1, a lens driving circuit 2, an imaging device 3, an imaging system control circuit 4, and an A / D converter 5. The imaging system has a function of inputting subject image formation and converting it into digital image data (still image data and moving image data).
[0014]
The optical lens unit 1 includes a zoom and focus mechanism, and the lens drive circuit 2 performs zoom and focus control. The image sensor 3 usually has a charge coupled device (CCD) of several million pixels, and photoelectrically converts input light (subject image) input from the optical lens unit 1 into an image signal (including color components). The imaging system control circuit 4 executes an AGC (automatic gain adjustment) process, a CDS (correlated double sampling) process, and the like for controlling the output timing and output level of the imaging signal from the image sensor 11. The A / D converter 5 converts the imaging signal into digital image data.
[0015]
The image processing system includes a digital signal processor (DSP) circuit 6, a motion detection circuit 7, and an SDRAM (synchronous DRAM) 8. The DSP circuit 6 executes various types of image processing (white balance correction processing, image compression / decompression processing, etc.) for image data stored in the SDRAM 8 and processing processing for audio data. The SDRAM 8 is a work memory (buffer memory) that temporarily stores information (motion amount distribution, subject position information, etc.) related to the sound generation position information of the embodiment in addition to image data and audio data. As will be described later, the motion detection circuit 7 is a circuit that performs motion detection (moving object detection) related to the dynamics of the subject, and has a function of acquiring motion amount distribution and subject position information.
[0016]
The display / output system and the recording system include a DMA circuit 11, an image display unit 12 that displays a captured image on a display, and a card interface 13. The DMA circuit 11 executes data transfer between the SDRAM 8 and the card interface 13 in accordance with an instruction from the system controller 17. The image display unit 12 includes, for example, an LCD (liquid crystal display) monitor. The card interface 13 controls writing and reading of image data, audio data, sound generation position information, and the like with respect to the memory card 14 that is a recording medium. Note that the display / output system includes a device for performing data transmission in a wireless or wired communication system, a video output terminal, and the like (not shown).
[0017]
The electronic camera of the embodiment includes an audio interface 9 and a microphone (audio input device) 10 for realizing a recording function. The audio interface 9 generates sampling data of audio input from the microphone 10 and stores it as audio data in the SDRAM 8.
[0018]
The control / operation system includes a power supply unit 15, an operation unit 16, and a system controller 17. The power supply unit 15 controls power supply to each block of the electronic camera in accordance with an instruction from the system controller 17. The operation unit 16 includes a release button, switches necessary for various mode settings, and the like. The system controller 17 includes a microprocessor for controlling input / output and operation of each block of the electronic camera and its program. The system controller 17 executes sound generation position information generation processing related to the embodiment. The system controller 17 also executes control for displaying an operation screen such as mode setting on the image display unit 12 in accordance with an input operation of the operation unit 16.
[0019]
(Shooting operation)
Hereinafter, referring to the flowcharts of FIGS. 2 and 3 together with FIG. 1, and FIGS. 4 and 5, the photographing operation including the operation of generating position information (sound generation position information) for associating the subject and the sound according to the embodiment. Will be described.
[0020]
First, when the power of the electronic camera is turned on, the system controller 17 starts shooting processing and sound processing (YES in step S1, S2). In the photographing process, a series of operations until the captured image data obtained by the photographing system is stored in the SDRAM 8 is executed (see step S6). In the sound processing, when sound generated from the vicinity of the subject is detected by the microphone 10, the sound interface 9 converts the sound into sound data and stores it in the SDRAM 8.
[0021]
Here, in the same embodiment, a mode called a subject position detection mode is prepared, and when the mode is set in the system controller 17 via the operation unit 16, position information (sound generation position) is associated with the shooting operation. Information) is generated (YES in step S3). If the mode is not set, the system controller 17 executes the above-described photographing process according to the operation of the release button of the operation unit 16 (NO in steps S3, S5, S6). The system controller 17 transfers the captured image from the SDRAM 8 to the DSP circuit 6 and causes the DSP circuit 6 to execute image compression processing and the like. Then, the image data is recorded on the memory card 14 via the card interface 13 (NO in step S7, S11). If the audio recording mode is set, the system controller 17 reads the audio data from the SDRAM 8 and records it in the memory card 14 via the DSP circuit 6 and the card interface 13 (YES in step S12, S13). .
[0022]
On the other hand, when the subject position detection mode is set, the system controller 17 controls the motion detection circuit 7 to start the motion detection process (YES in step S3, S4). The system controller 17 instructs the motion detection circuit 7 at the timing of shooting in the shooting process (that is, the shooting frame) (YES in steps S6 and S7, S8). The motion detection circuit 7 discriminates the subject from the motion amount distribution of the frame, and acquires the position information of the subject (step S9). The system controller 17 records the position information acquired by the motion detection circuit 7 on the memory card 14 in association with image data and audio data (step S10).
[0023]
(Operation principle of motion detection circuit)
In this embodiment, when the subject position detection mode is set, the position of the subject at the time of shooting is assumed as the sound generation position that is the sound generation position, and the position information is recorded as image data and sound data at the time of shooting. And recorded in the memory card 14.
[0024]
In this case, in the embodiment, the motion detection circuit 7 detects the dynamics of the subject, specifically, the movement of the entire human body or the mouth (moving body), and the detection result (the position of the moving body) is used as the position information of the subject. Calculate as In short, it is assumed that sound is generated from a moving subject and the position of the subject is a sounding position.
[0025]
Hereinafter, the operation principle of the motion detection circuit 7 will be described with reference to FIGS.
[0026]
The motion detection circuit 7 starts operating in response to an instruction from the system controller 17 and acquires a frame from the SDRAM 8 (steps S20 and S21). Specifically, luminance data in the image frame is acquired. When the motion detection circuit 7 acquires the brightness data for at least two frames, the motion detection circuit 7 uses the brightness data of the latest frame (Fn) 40 and the immediately preceding frame (Fn-1) 41 as shown in FIG. Motion amount distribution (Mn) 42 is calculated (step S23). This motion amount distribution (Mn) 42 is a set of absolute differences between frames that exceed a predetermined threshold (Mf).
[0027]
The motion amount distribution (Mn) can be expressed by the following equation (1) as a set of coordinate values M (n, x, y) in the captured image.
[0028]
M (n, x, y) = MAX (| F (n, x, y) −F (n−1, x, y) |, Mf) (1)
Here, x and y mean coordinate positions in the arbitrary X-axis and Y-axis directions of the captured image. F (n, x, y) indicates the luminance at the coordinate (x, y) position at the n-th frame position. Mf represents a threshold value for binarizing the luminance. MAX (a, b) means a function that returns “0” when “b> a” and returns “1” otherwise.
[0029]
Next, when an imaging frame (the last frame in time series) is instructed from the system controller 17, the motion detection circuit 7 calculates the motion amount interval distribution (Sn) for m frames (YES in step S24). S25). This motion amount interval distribution (Sn) is a binarized distribution. As shown in FIG. 5A, the movement amount interval distribution (Sn) is divided by the X axis and the Y axis, and the number exceeding the threshold (Mf) is counted for each interval. It is a thing. The motion detection circuit 7 compares the motion amount interval distribution (Sn) from the designated shooting frame to m frames before, and acquires the subject position (step S26).
[0030]
Here, the motion amount interval distribution (Sn) will be described with reference to FIG. FIG. 5A shows the number obtained by adding M (n, x, y) in the (a, b) section in the area divided by [Xarea, Area] in the motion amount distribution (Mn) S (n, a, b)) is shown as a motion amount distribution (Mn). FIG. 5B shows a section where the value of S (n, a, b) exceeds the threshold value (Ks).
[0031]
The motion amount interval distribution (Sn) can be expressed by the following equation (2) as a set of S (n, a, b) in all areas.
[0032]
S (n, a, b) = Σ (x = a [0], x = a [Xe−1], Σ (y = b [0], y = b [Ye−1], M (n, x , Y))) ... (2)
Here, a and b indicate arbitrary areas [a, b] divided by “Xarea × Yarea”. “Σ (x = 0, x = n, X (x))” means a value obtained by integrating X (x) from x = 0 to x = n. Xe indicates the number of divisions in the X-axis direction section in the captured image area (Xe = number of X-axis pixels / Xarea). Ye indicates the number of divisions in the Y-axis direction section in the captured image area (Ye = number of Y-axis pixels / Yarea). a [0] indicates the 0th X coordinate in area a, and b [0] indicates the 0th Y coordinate in area b.
[0033]
When the subject's partial movement is intense, the motion detection circuit 7 has a case where the number of S (n, a, b) exceeding the threshold value (Ks) is less than N as shown in FIG. , It is estimated that the subject exists in the section where the value of S (n, a, b) is maximum. In addition, if there is a section that is adjacent to the section and exceeds the threshold (Ks), that section is also set as the subject position. On the other hand, when the overall movement is intense (when the number of sections S (n, a, b) counted per section is greater than the threshold (Ks) is N or more), it is assumed that the camera is moving. The subject position is not specified.
[0034]
When the motion detection circuit 7 determines that the subject position is finally detected from the image of the shooting frame, the motion detection circuit 7 temporarily stores the position information in the SDRAM 8 (YES in step S27, S28). On the other hand, when the subject position cannot be finally determined, a plurality of pieces of candidate position information are temporarily stored in the SDRAM 8 (NO in step S27, S29). Specifically, when the position of the subject between m frames is continuous in time series (that is, when the position of the subject is in the same position or adjacent section between m frames), the subject position is finally detected. It is judged. If the subject position is not continuous in time series between m frames (that is, if the subject position exists in one or more sections between m frames), the subject position cannot be finally detected. To be judged. In this case, a plurality of candidate positions of the subject are determined from time-series elements and positional elements such as close to the center of the captured image, and are temporarily stored in the SDRAM 8.
[0035]
In short, when the subject position detection mode is set, the motion detection circuit 7 detects the dynamics of the subject, specifically the movement of the entire human body or the mouth (moving object), and the detection result (moving object). Is acquired as the position information of the subject. That is, the position of a subject that moves at the time of shooting is assumed as the sound generation position that is the sound generation position, and the position information is recorded on the memory card 14 in association with the image data and sound data to be recorded at the time of shooting. Accordingly, it is possible to obtain position information (pronunciation position information) that associates the subject with the sound together with the photographed image and the recorded sound, so that it is relatively easy to perform sound reproduction in association with the subject during image reproduction. Can be realized. Thereby, not only simple audio reproduction during image reproduction, but also, for example, by specifying a subject in conversation, audio reproduction of the conversation can be realized, and an excellent reproduction effect can be obtained.
[0036]
(Modification 1)
According to the principle of the embodiment, when a person who does not move is photographed backward as a subject, since the movement of the mouth cannot be detected, the sound generation position cannot be specified. Therefore, as a modification of the embodiment, the system controller 17 assumes that the subject photographed at, for example, the center position on the photographing screen is the sound generation position without using the motion detection circuit 7. Accordingly, the system controller 17 records the position information of the assumed sound generation position in the memory card 14 in association with the image data and sound data recorded at the time of shooting.
[0037]
Usually, in the audio recording mode, the subject that is likely to generate audio is often photographed with the center approximately, so it can be estimated that the center position is appropriate as the sound generation position. In addition, if there is a function that allows the user to input subject position information (designated position) from the operation unit 16, the position information may be recorded as pronunciation position information.
[0038]
(Modification 2)
FIG. 6 is a diagram for explaining a method for realizing the pseudo stereo effect at the time of sound reproduction as a second modification of the embodiment.
[0039]
The motion detection circuit 7 of the same embodiment detects the position of the subject 60 between m frames with respect to the time axis T as shown in FIG. Therefore, the position information of the subject 60 is used to set the virtual stereo microphone positions XL and XR as shown in FIG. 5B and relate to the sound volume for converting from monaural sound to stereo sound. The coefficients (KL, KR) to be calculated can be calculated. The position XL means the left virtual microphone position, and the position XR means the right virtual microphone position.
[0040]
Specifically, the system controller 17 calculates the coefficients (KL, KR) by the following procedure. That is, as shown in FIG. 6B, the distance L to the subject 60 is calculated from the focal length obtained from the imaging system. Next, from the angles α and β between the center position 61 of the subject 60 and the virtual stereo microphone positions XL and XR, and the distance M (calculated from the distance to the subject 60 and the lens specifications), the following relational expression ( 3) and (4) hold.
[0041]
tan (α) = L / (XL + M) (3)
tan (β) = L / (XR−M) (4)
Using the relational expression (3), a distance (referred to as LL) from the left virtual microphone position XL to the subject 60 is calculated. Further, the distance (referred to as LR) from the right virtual microphone position XR to the subject 60 is calculated using the relational expression (4). Then, a table including virtual stereo microphone positions XL and XR and distances LL and LR based on the angles α and β is created. The system controller 17 finally calculates coefficients (KL, KR) related to the sound volume for converting monaural sound to stereo sound from the table.
[0042]
As described above, according to this modification, when reproducing sound in association with the position of the subject, the sound is recorded by a virtual stereo microphone, and a pseudo stereo reproduction effect is obtained as if the sound is reproduced in stereo. Can do.
[0043]
【The invention's effect】
As described above in detail, according to the present invention, in an electronic camera having a recording function, an electronic camera having a function capable of generating position information that associates a subject of a captured image with sound can be provided. Thereby, at the time of image reproduction, it is possible to relatively easily realize the process of performing audio reproduction in association with the subject.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a main part of an electronic camera related to an embodiment of the present invention.
FIG. 2 is a flowchart for explaining a photographing operation according to the embodiment.
FIG. 3 is a flowchart for explaining the operation of the motion detection circuit according to the embodiment;
FIG. 4 is a view for explaining the principle of motion detection processing according to the embodiment;
FIG. 5 is a view for explaining the principle of motion detection processing according to the embodiment;
FIG. 6 is a view for explaining pseudo stereo effect processing according to a modification of the embodiment;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Optical lens unit 2 ... Lens drive circuit 3 ... Imaging element 4 ... Imaging system control circuit 5 ... A / D converter 6 ... Digital signal processor (DSP) circuit 7 ... Motion detection circuit 8 ... SDRAM
DESCRIPTION OF SYMBOLS 9 ... Audio | voice interface 10 ... Microphone 11 ... DMA circuit 12 ... Image display part 13 ... Card interface 14 ... Memory card 15 ... Power supply part 16 ... Operation part 17 ... System controller

Claims (3)

Imaging means for imaging a subject and converting it into image data;
Voice detection means for detecting voice and converting it into voice data;
Position information acquisition means for acquiring position information related to the sound generation position;
An output means for outputting the position information in association with the image data or the audio data;
Dynamic detection means for detecting the dynamics of the subject using the image data;
The position information acquisition means includes
When it is determined that the subject position has been detected from the dynamic detection by the dynamic detection means , the position information corresponding to the subject position is temporarily stored as the pronunciation position of the subject,
An electronic camera characterized in that, when it is determined that the subject position cannot be detected from the dynamic detection by the dynamic detection means, the subject position at the substantially central position on the photographing screen is temporarily stored as a pronunciation candidate position .   The electronic camera according to claim 1, wherein the movement detection unit is a moving body detection unit that detects a moving body based on a movement amount of a subject. The electronic camera according to claim 1, further comprising means for generating audio data corresponding to a virtual audio generation position based on the audio data and the position information .

Images