JP2011035637A - Imaging device, imaging method, and imaging program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent reproduction of a sound which is difficult to listen when reproducing a moving picture which has been taken while changing a frame rate. <P>SOLUTION: A control unit (13) of a camera (100) includes a function for, when a frame rate during taking of a moving picture is determined not to be a predetermined frame rate, continuing sound collection by a sound collecting means (4) and recording the moving picture taken by an imaging means (2) with data as substitution for the sound added thereto into a recording means (7). Accordingly, an effect can be expected such that a sound which is difficult to listen is not reproduced when the frame rate is changed during taking of the moving picture. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a photographing apparatus, a photographing method, and a photographing program.

  2. Description of the Related Art Conventionally, as represented by a digital video camera or the like, a photographing apparatus capable of simultaneously recording a moving image and sound has been widely used. In such a photographing device, in order to match the reproduction timing of the moving image and the audio, the audio data that has been compression-encoded and packetized is multiplexed with the moving image, and the synchronization information is embedded in the header information of the audio data. Turn into. In addition, the frame rate can be changed arbitrarily during movie recording, and the sample rate of the recorded audio is changed according to the changed frame rate, so that there is no sense of incongruity in audio playback during playback. A technique for doing so has been proposed (see, for example, Patent Document 1).

JP 2004-180890 A

  However, in the above-described conventional technology, in order to change the audio sample rate in accordance with the changed frame rate, a moving image with audio recorded at a high frame rate (for example, 60 fps) is reproduced at a normal frame rate (for example, 30 fps). In such a case, there is a problem that the normal playback speed is ½ times faster and it becomes difficult to hear the voice. In addition, it is possible to switch between a high frame rate and a normal frame rate arbitrarily during video recording, and when playing back only at the normal frame rate during playback, before and after the frame rate switching timing during video recording A problem that the voice becomes difficult to hear suddenly is also expected.

  An object of the present invention is to provide a photographing apparatus, a photographing method, and a photographing program in which a hard-to-hear sound is not reproduced when a moving image photographed while changing a frame rate is reproduced.

  An image pickup apparatus according to the first aspect of the present invention includes a photographing unit that photographs a moving image, a sound collecting unit that collects sound during moving image shooting by the photographing unit, and Change means for changing the frame rate; determination means for determining whether or not the frame rate changed by the change means is a predetermined frame rate; a moving image shot by the shooting means; and sound collection by the sound collection means If the recording means for recording the sound and the determination means determine that the frame rate is not the predetermined frame rate, the sound collection by the sound collection means is continued and the sound captured by the photographing means is recorded on the moving image. Recording control means for adding data to be replaced and recording the data in the recording means.

  The image pickup apparatus according to a second aspect of the invention is characterized in that the recording control unit adds a dummy sound output as an unvoiced sound as data instead of the sound.

  According to a third aspect of the present invention, the recording control unit adds the sound collected by the sound collecting unit to the moving image when the determining unit determines that the frame rate is not a predetermined frame rate. Without deleting.

  An image pickup apparatus according to a fourth aspect of the present invention includes a reproducing unit that reproduces a moving image recorded in the recording unit, and the predetermined frame rate is normally set when the reproducing unit reproduces the moving image. A playback frame rate.

  According to a fifth aspect of the present invention, there is provided an imaging method for photographing a moving image, a sound collecting step for collecting sound during moving image photographing in the photographing step, and a recording step for collecting the moving image during moving image photographing in the photographing step. A change step for changing the frame rate; a determination step for determining whether or not the frame rate changed by the change step is a predetermined frame rate; and a moving image shot by the shooting step and a sound collection step. When the recording step for recording the recorded audio and the determination step determines that the frame rate is not the predetermined frame rate, the sound collection by the sound collection step is continued and the moving image captured by the shooting step is recorded. A recording control step of recording by the recording step by adding alternative data; Obtain.

  According to a sixth aspect of the present invention, there is provided an image pickup program comprising: a computer including a photographing unit that photographs a moving image; a sound collecting unit that collects sound during the moving image photographing by the photographing unit; and a moving image photographing by the photographing unit Changing means for changing the frame rate of the moving image, determining means for determining whether or not the frame rate changed by the changing means is a predetermined frame rate, the moving image shot by the shooting means and the sound collecting means Recording means for recording the sound collected by the recording means, and when the judgment means determines that the frame rate is not a predetermined frame rate, the sound collection by the sound collection means is continued and the image pickup means Data serving as a substitute for sound is added to the moving image, and the recording means is made to function as recording control means.

  According to the present invention, it is possible to provide a photographing apparatus, a photographing method, and a photographing program in which a sound that is difficult to hear is not reproduced when a moving image photographed while changing a frame rate is reproduced.

1 is a block diagram illustrating a circuit configuration of a digital camera having a function of an imaging apparatus according to an embodiment of the present invention. It is explanatory drawing which shows the relationship between video data and audio | voice data when a frame rate is changed during video recording. 2 is a flowchart showing processing contents of a control unit when creating a moving image file in the digital camera of FIG. 1.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a circuit configuration of a digital camera 100 that functions as an image processing apparatus according to an embodiment of the present invention.

  The digital camera 100 according to the present embodiment includes a shooting mode for shooting a moving image and a playback mode for playing back a captured image as basic operation modes. Among these, a shooting mode for shooting a moving image includes a high-speed moving image shooting mode in addition to a normal moving image shooting mode. As one of the high speed moving image shooting modes, a moving image shooting mode is set in which the frame rate can be changed between 30 fps (normal frame rate) and 210 fps (high frame rate) during moving image shooting. In the present embodiment, during moving image shooting in this moving image shooting mode, when the frame rate is changed between 30 fps and 210 fps, sound that is difficult to hear is not reproduced.

  As shown in FIG. 1, the digital camera 100 includes an optical system 1, an imaging unit 2, a recording / playback data creation unit 3, an audio input / output unit 4, a moving image temporary memory 5, an audio temporary memory 6, A memory card 7, a liquid crystal display device 8, a shooting start / stop button 9, a frame rate switching button 10, a playback button 11, various operation buttons 12, and a control unit 13 are provided.

  The optical system 1 is an optical lens for forming an incident subject light as an optical image on an imaging unit 2 described later, and includes a focus lens, a zoom lens, and the like.

  The imaging unit 2 includes an image sensor such as a complementary metal oxide semiconductor (CMOS) and is disposed on the optical axis of the optical system 1. The imaging unit 2 receives the imaging light converged by the optical system 1, photoelectrically converts the optical image of the subject formed on the light receiving surface, and outputs it as an analog imaging signal. The imaging unit 2 captures an image at a predetermined frame rate during moving image shooting. As will be described later, in the moving image shooting mode of the present embodiment, an image is captured at a frame rate of 30 fps or 210 fps.

  In addition, the imaging unit 2 performs image processing such as A / D conversion, gamma correction, white balance correction, and contour enhancement that converts an analog imaging signal output from the image sensor into digital image data, and performs digital image processing. A pre-processing unit (not shown) for outputting as data is provided. The image data processed by the preprocessing unit is temporarily stored in the moving image temporary memory 5 via the recording / playback data creation unit 3 and the control unit 13. The optical system 1 and the imaging unit 2 in the present embodiment function as a photographing unit of the present invention.

  The recording / playback data creation unit 3 is a circuit that encodes image data output from the imaging unit 2 and audio data collected by the audio input / output unit 4 described later. Specifically, for example, encoding is performed by a compression encoding process using a predictive encoding method adopted in the MPEG (Motion Picture Expert Group) method. Image data stored in the moving image temporary memory 5 and audio data stored in the audio temporary memory 6, which will be described later, are compressed and encoded in the order of image capture, for example, as normal frame rate data of 30 fps, and sequentially stored in the memory card 7. Remembered.

  The audio input / output unit 4 includes a microphone that collects surrounding audio during moving image shooting, a speaker that outputs audio data as audio, and the like. The sound collected at the time of moving image shooting is subjected to predetermined signal processing in the control unit 13 and stored in the sound temporary memory 6 as sound data. In this embodiment, since the audio data is continuously stored in the audio temporary memory 6 even during the period when the real audio data is not added to the moving image data, the microphone is always on during moving image shooting.

  The moving image temporary memory 5 is an image recording unit that sequentially stores the moving image data output from the imaging unit 2 for each frame. The temporary audio memory 6 is an audio recording means for sequentially storing audio data collected by the audio input / output unit 4.

  In the present embodiment, first, moving image data is stored for each frame in the moving image temporary memory 5, and audio data is stored in the audio temporary memory 6. Then, while the frame rate at the time of moving image shooting is 30 fps, the audio data stored in the temporary audio memory 6 is added to the moving image data stored in the temporary video memory 5 and stored in the memory card 7.

  However, when the frame rate at the time of moving image shooting is 210 fps, dummy sound (dummy data) output as unvoiced sound is added to the moving image data stored in the moving image temporary memory 5 and stored in the memory card 7. At this time, the audio data collected by the audio input / output unit 4 is continuously stored in the audio temporary memory 6, but this audio data is sequentially deleted at a predetermined timing without being added to the image data. . That is, the actual voice data stored in the voice temporary memory 6 is sequentially deleted when a predetermined time elapses, and the voice temporary memory 6 always stores only a certain amount of the latest voice data.

  The memory card 7 is a storage medium that stores the above-described moving image data and audio data. When moving image data and audio data stored in the memory card 7 are instructed to end shooting, predetermined header information is added and the moving image data and audio data are filed as moving image files. The moving image temporary memory 5, the audio temporary memory 6 and the memory card 7 in the present embodiment function as recording means of the present invention.

  The liquid crystal display device 8 includes a VRAM (not shown), a liquid crystal monitor, and a drive circuit thereof. The liquid crystal display device 8 displays a through image during shooting standby or during shooting, a moving image stored in the memory card 7, and the like. The liquid crystal display device 8 in the present embodiment functions as the reproducing means of the present invention. The frame rate of 30 fps used at the time of capturing a moving image in the imaging unit 2 is also a reproduction frame rate normally set in the liquid crystal display device 8.

  The shooting start / stop button 9 is a part for inputting a shooting start and shooting end instruction from the user. The frame rate switching button 10 is a part for obtaining an instruction to change the frame rate during moving image shooting from the user. The play button 11 is a part for inputting an instruction to play a moving image or the like from the user. The various operation buttons 12 are parts for inputting various operation instructions to the digital camera 100 from the user. The various operation buttons 12 are operated by the user when setting / changing various functions and operation contents of the digital camera 100 in addition to, for example, a mode setting button for setting a shooting mode and a playback mode, a power switch, and a zoom key. An operating member may be used. Further, the frame rate switching button 10 in the present embodiment functions as a changing unit of the present invention.

  The control unit 13 includes a CPU, ROM, RAM, and peripheral circuits (not shown), and controls the overall operation of the digital camera 100 including moving image shooting and playback display. The ROM stores a program for performing control including moving image shooting and playback display. The RAM is used as a temporary storage area when performing various controls and arithmetic processes. The control unit 13 in the present embodiment functions as a determination unit and a recording control unit of the present invention. The processing contents of the determination unit and the recording control unit of the control unit 13 will be described later with reference to a flowchart.

  FIG. 2 is an explanatory diagram showing the relationship between moving image data and audio data when the frame rate is changed during moving image shooting. FIG. 2 shows an example in which the frame rate is changed to 210 fps and then changed again to 30 fps during moving image shooting at a frame rate of 30 fps in time series. The flag shown in FIG. 2 is written into the status register of the internal memory by the control unit 13 during processing.

  In FIG. 2, the recording suppression flag is a flag for determining whether or not to add real audio data to moving image data in accordance with a change in the frame rate. It is “TRUE” when it is suppressed, and “FALSE” when it is not suppressed. The flag is “FALSE” when the frame rate is set to 30 fps, and the flag is “TRUE” when the frame rate is set to 210 fps. The initial recording start flag is a flag for determining the timing at which the audio data to be added is switched from the dummy audio to the actual audio in the process of adding the audio data to the moving image data. Specifically, the timing at which the frame rate is switched from 210 fps to 30 fps is determined as the timing for switching from the dummy sound to the actual sound. When switching from the dummy voice to the real voice, “TRUE”, otherwise “FALSE”. The dummy sound insertion flag is a flag for determining whether or not to add dummy sound to the moving image data. The dummy voice insertion flag is determined by the recording suppression flag and the recording start initial flag. It is “TRUE” when the recording suppression flag is “TRUE” and the recording start initial flag is “FALSE”, and “FALSE” when the recording suppression flag is “FALSE” and the recording start initial flag is “TRUE”.

  When the frame rate is set to 30 fps, the recording start initial flag is “FALSE” and the recording suppression flag is “FALSE”. At this time, real audio data is added to the moving image data. When the frame rate is switched from 30 fps to 210 fps, the initial recording start flag is “FALSE” and the recording suppression flag is “TRUE”. The dummy voice insertion flag is “TRUE”. As a result, dummy audio data is added to the moving image data instead of the actual audio data.

  The actual voice data is continuously stored in the voice temporary memory 6. This is to prevent real audio data from being added to the moving image data immediately after the frame rate returns to 30 fps again. The audio data is sequentially deleted when a predetermined time elapses, and the audio temporary memory 6 always stores only a certain amount of the latest audio data. This is to prevent the temporary audio memory 6 from overflowing with the actual audio data stored during the process of adding only the dummy audio data to the moving image data. Therefore, while adding the voice data of the dummy voice to the moving picture data, the process of storing the voice data of the real voice in the temporary voice memory 6 is continued, and the stored voice data of the real voice is old when a predetermined time elapses. Are sequentially deleted (however, FIG. 2 shows only the manner in which real voice data is continuously stored). Also, by performing such processing, it is possible to maintain the consistency of audio data immediately after the frame rate switching.

  When the frame rate is switched from 210 fps to 30 fps, the recording start initial flag is “TRUE” and the recording suppression flag is “FALSE”. The dummy voice insertion flag is “FALSE”. As a result, real voice data is added to the moving image data instead of dummy voice data. It should be noted that immediately after the frame rate is switched from 210 fps to 30 fps, when the frame rate is switched from 210 fps to 30 fps, the actual audio data is replaced with dummy audio data. Before adding the audio data to the moving image data, the real audio data is added so as to be synchronized with the moving image frame. This synchronization shift between the audio data and the moving image data causes a so-called lip sync shift. By performing such processing, it is possible to prevent a lip sync shift from occurring between the audio data and the moving image data immediately after the frame rate is switched from 210 fps to 30 fps.

  Next, in the digital camera 100 of the present embodiment, the processing content of the control unit 13 when creating a moving image file will be described with reference to the flowchart of FIG. The following processing is executed by the control unit 13 shown in FIG. 1 controlling each unit. Further, the processing of this flowchart is performed by the user operating the mode setting buttons of the various operation buttons 12 to set the moving image shooting mode in which the frame rate can be changed between 30 fps and 210 fps during moving image shooting, and then start shooting / The operation is started by operating the stop button 9 to instruct the start of shooting.

  First, in step S1, the control unit 13 starts a process of sequentially storing the moving image data output from the imaging unit 2 in the moving image temporary memory 5, and the audio data output from the audio input / output unit 4 is stored in the temporary audio memory. 6 is started. In step S2, the control unit 13 determines whether or not the currently set frame rate is 30 fps. If the frame rate switching button 10 is not operated during shooting, the frame rate is not changed. If the frame rate is 30 fps, the process proceeds to step S3 (step S2: YES). If the frame rate has been changed from 30 fps by operating the frame rate switching button 10 or the like, the process proceeds to step S8 (step S2: NO).

  In step S3, the control unit 13 controls the recording / playback data creation unit 3 to convert the moving image data and the real sound audio data sequentially stored in the moving image temporary memory 5 and the audio temporary memory 6 to a data with a frame rate of 30 fps. Are sequentially encoded, and real audio data is added to the moving image data, which are sequentially stored in the memory card 7.

  In step S4, the control unit 13 determines whether or not an instruction to end shooting has been detected. Here, if an instruction to end shooting is not detected by operating the shooting start / stop button 9, the process proceeds to step S5 (step S4: NO), and if an instruction to end shooting is detected, the process proceeds to step S7 (step S7). Step S4: YES).

  In step S5, the control unit 13 determines whether a frame rate change instruction has been detected. Here, if it is determined that an instruction to change the frame rate by operating the frame rate switching button 10 has not been detected, the process returns to step S3 (step S5: NO), and the audio data of the actual audio is added to the moving image data. The process of sequentially storing in 7 is repeated. On the other hand, if it is determined that an instruction to change the frame rate is detected, the process proceeds to step S6 (step S5: YES). In step S6, the control unit 13 changes the frame rate to 210 fps, and returns to step S2.

  If YES in step S4, the process proceeds to step S7, and the control unit 13 determines the file name, shooting date and time, frame rate for the moving image data and audio data stored in the memory card 7 so far. The header information including such information is added and filed as a moving image file in the memory card 7.

  On the other hand, if the determination in step S2 is NO, in step S8, the control unit 13 controls the recording / playback data creation unit 3 to convert the moving image data sequentially stored in the moving image temporary memory 5 to a frame rate of 210 fps. The data is sequentially compressed and encoded as data, and dummy audio data is added to the moving image data and sequentially stored in the memory card 7.

  In step S9, the control unit 13 determines whether an instruction to end shooting is detected. In this case, if an instruction to end shooting is not detected by operating the shooting start / stop button 9, the process proceeds to step S10 (step S9: NO). If an instruction to end shooting is detected, the process proceeds to step S7. Proceed (step S9: YES). When the audio data of dummy audio is added to the video data and sequentially stored in the memory card 7 in the previous step S8, the video data and the audio data of dummy audio stored in the memory card 7 until then are stored in step S7. Is added to the header information including information such as the file name, shooting date and time, frame rate, etc., and filed as a moving image file in the memory card 7.

  In step S10, the control unit 13 determines whether a frame rate change instruction has been detected. Here, if an instruction to change the frame rate by operating the frame rate switching button 10 is not detected, the process returns to step S8 (step S10: NO), and the audio data of dummy audio is added to the moving image data and the memory card 7 is restored. Repeat the process of sequentially storing. On the other hand, if an instruction to change the frame rate is detected, the process proceeds to step S11 (step S10: YES). In step S11, the control unit 13 changes the frame rate to 30 fps and returns to step S2.

The digital camera 100 of the present embodiment has at least the following effects.
(1) When shooting at a playback frame rate (30 fps), real audio data is added to the moving image data, and when shooting at a high frame rate (210 fps), dummy audio data is added to the moving image data. Real video is played in a moving image shot at a frame rate of 30 fps during playback, and no sound is played when the moving image is shot at a frame rate of 210 fps. Therefore, when playing back a moving image shot while changing the frame rate, it is possible to prevent playback of sound that is difficult to hear.
(2) In the related art, when the file is formed by embedding the synchronization information in the header information of the audio data, since it cannot be synchronized depending on the reproduction apparatus, it is difficult to reproduce the moving image synchronized with the audio. there were. However, according to the present embodiment, when capturing at the playback frame rate, the real audio data synchronized with the moving image data is added, so that the audio can be synchronized regardless of the playback device. It is possible to play the video. For example, a moving image reproduction application or the like installed in a general personal computer can reproduce a moving image synchronized with sound without performing special processing.
(3) Audio encoding processing during moving image shooting need not be stopped or restarted at the frame rate switching timing, and may be the same as during normal moving image shooting, thus reducing the processing burden on the CPU. be able to.
(4) When shooting at a high frame rate, the dummy sound output as no sound is added to the moving image data. Therefore, the actual sound added at the time of shooting at the normal playback frame rate is less uncomfortable. can do.
(5) When shooting at the normal playback frame rate, the actual audio data is added to the moving image data, so that both the playback frame rate and the high frame rate are more realistic than when no audio is used. It is possible to play video with
(6) When shooting at a high frame rate, the actual audio data stored in the temporary audio memory 6 that is not added to the moving image data is sequentially deleted at a predetermined timing. Immediately after switching to the frame rate, it becomes easy to add real audio data to the moving image data. In addition, it is possible to maintain the consistency of audio data immediately after the frame rate is switched. Furthermore, since it is not necessary to increase the storage capacity of the temporary audio memory 6, there is no cost increase.
(7) Since the frame rate of 30 fps used at the time of moving image capturing is set as a reproduction frame rate normally set in the liquid crystal display device 8, by reproducing a moving image shot at a high frame rate at a normal reproduction frame rate. In addition, it is possible to more efficiently slow-play moving images that have been shot at high speed.

As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment. That is, the following modifications and improvements as long as the object of the present invention can be achieved are included in the present invention.
(1) As an example of a moving image shooting mode in which the frame rate can be changed during moving image shooting, the example in which the frame rate can be changed between 30 fps and 210 fps has been described. However, the present invention is not limited to this and is changed to a higher frame rate. It can also be applied to possible movie shooting modes.
(2) Although an example of adding dummy sound output as no sound as dummy data to be added to moving image data has been described, a plurality of types of sound effects and BGM (music) are prepared as sound data as dummy data. These may be appropriately selected and added to the moving image data.
(3) Some or all of the functions of the control unit 13 described above may be realized by operating other control units operating in parallel with the control unit 13 in accordance with the photographing program according to the present invention.
(4) The present invention can be applied not only to a digital video camera or a digital camera but also to a digital still camera or a camera-equipped mobile phone having a moving image shooting and playback function. In addition, the present invention can also be applied to other apparatuses capable of only reproducing captured moving images, for example, general-purpose personal computers having a general configuration by installing the imaging program according to the present invention.

DESCRIPTION OF SYMBOLS 1 Optical system 2 Image pick-up part 3 Recording / reproduction | regeneration data creation part 4 Audio | voice input / output part 5 Movie temporary memory 6 Voice temporary memory 7 Memory card 8 Liquid crystal display device 9 Shooting start / stop button 10 Frame rate switching button 11 Playback button 12 Various operations Button 13 Control unit 100 Digital camera

Claims (6)

Shooting means for shooting videos,
Sound collecting means for collecting sound during video shooting by the photographing means;
Changing means for changing a frame rate of the moving image during moving image shooting by the shooting means;
Determining means for determining whether or not the frame rate changed by the changing means is a predetermined frame rate;
Recording means for recording the moving image photographed by the photographing means and the sound collected by the sound collecting means;
When the determination means determines that the frame rate is not the predetermined frame rate, the sound collection by the sound collection means is continued, and data that replaces the sound is added to the moving image shot by the shooting means. Recording control means for recording in the recording means;
An imaging apparatus comprising: The recording control means includes
The photographing apparatus according to claim 1, wherein a dummy sound output as an unvoiced sound is added as data that replaces the sound. The recording control means includes
The photographing according to claim 1 or 2, wherein when the determination unit determines that the frame rate is not a predetermined frame rate, the sound collected by the sound collection unit is deleted without being added to the moving image. apparatus. Reproducing means for reproducing the moving image recorded in the recording means,
4. The photographing apparatus according to claim 1, wherein the predetermined frame rate is a reproduction frame rate that is normally set when the reproduction unit reproduces the moving image. 5. Shooting steps to shoot videos,
A sound collecting step for collecting sound during video recording by the photographing step;
A changing step of changing a frame rate of the moving image during moving image shooting by the shooting step;
A determining step of determining whether or not the frame rate changed by the changing step is a predetermined frame rate;
A recording step for recording the moving image photographed by the photographing step and the sound collected by the sound collecting step;
If it is determined by the determination step that the frame rate is not the predetermined frame rate, the sound collection by the sound collection step is continued, and data that replaces the sound is added to the moving image shot by the shooting step. A recording control step for recording by a recording step;
A photographing method characterized by comprising: A computer equipped with a shooting means for shooting videos
Sound collecting means for collecting sound during video shooting by the photographing means;
Changing means for changing a frame rate of the moving image during moving image shooting by the shooting means;
Determining means for determining whether or not the frame rate changed by the changing means is a predetermined frame rate;
Recording means for recording the moving image photographed by the photographing means and the sound collected by the sound collecting means;
When the determination means determines that the frame rate is not the predetermined frame rate, the sound collection by the sound collection means is continued, and data that replaces the sound is added to the moving image shot by the shooting means. Recording control means for recording in the recording means,
Shooting program to function as.

Images