JP4164696B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging device and an imaging method for compressing captured image information obtained by imaging and recording it on a recording medium such as a disk.

  For example, when recording time-series information such as image information and audio information on a recording medium such as a disk, tape, or semiconductor memory, it is possible to write time-series information for a longer time in a limited writing area. In order to be able to do this, time-series information is generally compressed.

  As a data compression method, the MPEG (Moving Picture Experts Group) method has been widely used.

  In the MPEG system, for image information, data using a predictive coding using correlation of information of an image frame (or an image field) which is a unit of decoding and reproducing image information and data using DCT (Discrete Cosine Transform). Perform compression.

  In the MPEG system, an image of each frame (or field. The same applies to the following, but for the sake of simplicity, only the word “frame” is used in the following description.) Depending on the difference, there are three types: I picture, P picture, and B picture.

  An I picture is an image (intra-frame encoded image) that is subjected to compression encoding that is closed only within one frame. The P picture is an image (forward prediction encoded image) that is subjected to compression encoding using the correlation with the image frame before the image frame. A B picture is an image (bidirectional predictive encoded image) subjected to compression encoding using the correlation between image frames before and after the image frame.

  In the MPEG system, a unit of writing / reading (read / write unit) with respect to a recording medium is configured by a so-called GOP (Group of Pictures) composed of a plurality of frames. FIG. 9 shows an example of the GOP.

  As shown in FIG. 9, in the MPEG system, the GOP always includes at least one I picture. This is because an image cannot be decoded using only a P picture and a B picture. Thus, by including an I picture in a GOP, random access can be performed in GOP units.

  The I picture and the P picture are encoded in the same order as the original picture, but the B picture is encoded after the I picture and the P picture are encoded first. In consideration of this encoding order, in the MPEG encoder, the order of images is rearranged from the original order in the state of FIG. 9A as shown in FIG. 9B, The rearranged image information is subjected to compression encoding processing (see Japanese Patent Application Laid-Open No. 8-214265).

  That is, as indicated by an arrow in FIG. 9A, the P picture is subjected to predictive coding using the preceding I picture or P picture. In addition, the B picture is subjected to predictive coding using the front and rear I pictures or P pictures for the two pictures sandwiched between the front and rear I pictures or P pictures.

  In this way, since the B picture uses the backward I picture or P picture at the time of encoding, as shown in FIG. 9B, the B picture predictively encodes it. The rearrangement is performed so that the order is later than the rear I picture or P picture used for.

  Then, as will be described later, management information for decoding / reproduction processing is added to the decoded and reproduced image data, that is, picture-by-picture data, and recorded on the recording medium. When reproducing the image data compressed by the MPEG method, management information for decoding / reproducing processing is extracted from the recording data, and based on the management information, the output of the reproduced picture unit image is controlled. , The order of the pictures is restored.

The above-mentioned patent documents are as follows.
See JP-A-8-214265

  By the way, for example, in a digital camera capable of shooting not only still images but also moving images, the MPEG method is adopted as a data compression method. In a digital camera, it is convenient if shooting conditions such as aperture, exposure value, and white balance can be added to image information and recorded on a recording medium in addition to the shooting time.

  In particular, if additional information such as the shooting time and shooting conditions for each frame of the image is recorded corresponding to the image information of each frame, it is possible to know how the image of the frame was shot. Convenient. Further, by referring to the additional information about the image in units of frames at the time of image reproduction processing or editing processing, the reproduction image quality can be improved, or fine editing processing can be performed. The application to various uses is expected.

  However, in the MPEG format recording data transmission format, a method for recording (transmitting) additional information about image data and audio data has not been proposed, and how the additional information can be used conveniently. Whether it is added is a technical issue.

  In this case, in the case where a plurality of units of image frames as decoding / playback units are included in a GOP which is a read / write unit for the recording medium, as in the MPEG system described above, the information of the decoding / playback unit information When recording additional information for each, it is important how the additional information for each decoding / playback unit is recorded in association with the information of the decoding / playback unit.

  In particular, as described above, when the image data is recorded with the frame order rearranged as shown in FIG. 9B during the encoding process, as in the GOP which is a read / write unit in the MPEG system. However, there is a problem that it is difficult to associate the information of each decoding reproduction unit with the additional information.

  In view of the above, the present invention, when the information in the read / write unit for the recording medium includes information on a plurality of decoding / playback units, the additional information on the information on the decoding / playback units, and the recorded decoding / playback It is an object of the present invention to provide an imaging apparatus, an imaging method, and an apparatus that can reliably correspond to unit information.

In order to solve the above problems, an imaging apparatus according to the present invention provides:
An image sensor;
An imaging optical system for forming a subject image on the imaging element;
Data compression means for data compression processing of image information from the image sensor;
Recording means for recording the compressed image information on a recording medium;
Generates additional information indicating an environment at the time of imaging of the image information corresponding to one or a plurality of the decoding reproduction units among the information of the predetermined decoding reproduction units of the compressed image information included within a predetermined time. And generating means for
The recording means includes
The first pack to which the additional information is stored, and, the one or more of the decoding of said predetermined plurality of said decoded reproduction unit including a reproduction unit information and the predetermined plurality of said decoded reproduction unit corresponding to the additional information As a unit, the second pack is arranged before or after the corresponding first pack as a logical arrangement of recording data. As described above, the image information is recorded on the recording medium.

  According to the present invention, for example, an environment at the time of imaging of image information corresponding to one or a plurality of decoding reproduction units among information of a plurality of decoding reproduction units corresponding to a GOP and information of a plurality of predetermined decoding reproduction units is shown. Since the data including the additional information is recorded on the recording medium in units, it is possible to reproduce the information of a predetermined plurality of decoding reproduction units and the additional information in association with each other.

  Therefore, it is possible to easily improve the reproduction output quality of the captured image information in the decoding reproduction unit or perform the editing process using the additional information.

  According to the present invention, it is possible to reproduce information associated with a predetermined plurality of decoding reproduction units and the additional information, and to improve the reproduction output quality of the captured image information of the decoding reproduction unit using the additional information. Or editing processing can be easily performed.

  Hereinafter, an embodiment of an image pickup apparatus and an image pickup method according to the present invention will be described with reference to the drawings, taking as an example a case where the present invention is applied to a digital camera using, for example, an optical disk as a recording medium.

  In this example, a writable optical disc such as a DVD-R (Digital Versatile Disc Recordable) or DVD-RW (Digital Versatile Disc Re-Recordable) is used as the optical disc. As the data compression method, the MPEG2 method is used, and the recording data has a structure of MPEG2-PS (PS; Program Stream).

  FIG. 1 is a block diagram illustrating a configuration example of a digital camera according to an embodiment. The digital camera in this example includes a camera unit 10, a recording / playback processing unit 20, and a control unit 30.

  The control unit 30 has a program ROM (Read Only Memory) 303, a work area RAM (Random Access Memory) 304, and an operation input unit 31 via a system bus 302 for a CPU (Central Processing Unit) 301. An operation input interface 305 for connection, an LCD controller 306 for connecting an LCD (Liquid Crystal Display) 32 as a display device, a memory card interface 307 for loading a memory card 50, and a USB (Universal) Serial Bus) communication interface 308, clock circuit 309, I / O ports 310, 311, 312, 313, 314, 315, 3 6 and is configured by connecting.

  The ROM 303 stores a program for controlling the camera unit 10, recording control and reproduction control for captured image information and audio information from the microphone, and other programs.

  The operation input unit 31 includes a mode switching key for switching between a shooting mode and another mode such as a playback mode, a zoom adjustment key, an exposure adjustment key, a shutter key, a moving image shooting key, a white balance adjustment key, and the like. A plurality of keys are provided.

  The CPU 301 analyzes an operation signal from the operation input unit 31 input through the operation input interface 305, determines which key of the operation input unit 31 is operated, and performs control processing according to the determination result. .

  Under the control of the CPU 301, the image at the time of shooting and shooting conditions are displayed on the LCD 32, and a playback image and additional information played back from the recording medium are displayed.

  The clock circuit 309 measures a clock signal from a reference clock signal generation circuit (not shown) and generates time information representing year, month, day, hour, minute, and second. As the second, the clock circuit 309 is configured so as to generate information with precision below the decimal point so that the frame unit of the image can be counted.

  The digital camera of this example is configured so that recording data can be recorded on the optical disc 40 (in this example, DVD 40) and also on the memory card 50. In that case, compressed data is recorded in the memory card 50 through the memory card interface 307.

  The communication interface 308 is used, for example, when the digital camera of this example is connected to a personal computer to exchange recording data and other data.

  The camera unit 10 includes a camera optical system 11, a camera control unit 12, an imaging element 13, an imaging signal processing unit 14, a microphone 15, and an input audio signal processing unit 16.

  Although not shown, the camera optical system 11 includes a lens group for imaging a subject, an aperture adjustment mechanism, a focus adjustment mechanism, a zoom mechanism, a shutter mechanism, a flash (strobe) mechanism, and a camera shake correction mechanism. ing.

  The camera control unit 12 receives a control signal through the I / O port 310 of the control unit 30 and generates a control signal to be supplied to the camera optical system 11. Then, the generated control signal is supplied to the camera optical system 11 to perform controls such as zoom control, shutter control, and exposure control. The signal through the I / O port 310 of the control unit 30 is set in advance according to the operation input from the operation input unit 31 as well as the control signal corresponding to the operation input from the operation input unit 31 at that time. A control signal based on the photographing conditions is included.

  In this example, the imaging device 13 is configured by a CCD (Charge Coupled Device), and an image through the camera optical system 11 is formed on the imaging plane. In response to an image capture timing signal generated from the I / O port 311 of the control unit 30 according to the shutter operation, the subject image formed on the imaging plane is converted into an imaging signal, and the imaging signal processing unit 14 To supply.

  The imaging signal processing unit 14 performs processing such as gamma correction and AGC (Auto Gain Control) on the imaging signal based on control through the I / O port 312 of the control unit 30 and converts the imaging signal into a digital signal. The process to perform is also performed.

  The microphone 15 collects sound around the subject at the time of shooting. The sound signal from the microphone 15 is supplied to the input audio signal processing unit 16.

  The input audio signal processing unit 16 performs processing such as correction on the input audio signal and AGC (Auto Gain Control) based on control through the I / O port 312 of the control unit 30, and converts the audio signal into a digital signal. A conversion process is also performed.

  Although not shown in FIG. 1, the digital camera of this example includes an external microphone terminal, and when an external microphone is connected to the external microphone terminal, the input audio signal processing unit 16 includes a built-in microphone. Instead of 15, it is configured to be automatically switched so that an audio signal is supplied from the external microphone.

  In this example, the AGC processing in the input audio signal processing unit 16 has the characteristics when the audio signal from the built-in microphone 15 is input and when the audio signal from the external microphone is input. Processing that takes into account the difference is made. The switching of the AGC process is performed in conjunction with the switching of the audio signal input to the input audio signal processing unit 16.

  The recording / playback processing unit 20 includes a DSP (Digital Signal Processor) 21, a DVD signal processing unit 22, a writing / reading unit 23, and a video / audio output processing unit 24. A buffer memory 25 made of, for example, SDRAM is connected to the DSP 21.

  The DSP 21 receives a digital image signal from the imaging signal processing unit 14 and a digital audio signal from the input audio signal processing unit 16, and information at the time of shooting through the I / O port 313 of the control unit 30. The The information at the time of shooting input through the I / O port 313 includes shooting time in frame units and shooting condition information.

  In this example, the time information from the clock circuit 309 is read and used as shooting time information. However, the time information (shooting start time) of the clock circuit 309 is taken in at the time of starting the shooting of the moving image, and thereafter, the relative time in frame units (may be a frame number) based on the shooting start time is used as the shooting time It may be used.

  The shooting condition information includes both automatically set information and information set by the user. For example, exposure control information, automatic white balance adjustment information, digital zoom magnification information, camera shake correction information, F It includes setting information such as value (F number), exposure time, AGC gain value, and lens focal length.

  Further, in this example, information on whether the built-in microphone 15 is used as a voice information capturing unit or an external microphone is used is also included in the information at the time of photographing.

  In this embodiment, the information at the time of shooting described above is generated and recorded as additional information for each frame of an image, as will be described later.

  The DSP 21 performs automatic white balance control, control for exposure correction, control according to the digital zoom magnification, and the like on the digital image signal from the imaging signal processing unit 14 based on camera information through the I / O port 313. Then, the DSP 21 compresses the digital image signal to which such control is applied and the digital audio signal from the input audio signal processing unit 16 by the MPEG2 method.

  Further, the DSP 21 performs additional information processing for recording the input photographing time information as additional information together with the compressed image data and the compressed audio data.

  The DSP 21 also performs a process of multiplexing the compressed image data, the compressed audio data, and the above-described additional information to form a data string according to the MPEG-PS format.

  Then, the MPEG-PS format data from the DSP 21 is supplied to the writing / reading unit 23 through the recording signal processing unit of the DVD signal processing unit 22 that is controlled by the control unit 30 through the I / O port 314, and this writing / reading out. The data is written on the DVD 40 by the unit 23. The writing / reading unit 23 is controlled by the control unit 30 through the I / O port 316.

  Then, the data read from the DVD 40 by the writing / reading unit 23 is supplied to the DSP 21 through the reproduction signal processing unit of the DVD signal processing unit 22.

  The DSP 21 separates the compressed image data, the compressed audio data, and the additional information that have been multiplexed. Then, the DSP 21 decompresses the compressed image data that has been separated, reproduces the image data in units of frames in the original arrangement order, outputs the image data to the video / audio output processing unit 24, and outputs the separated compressed audio data. The compression is decompressed, the audio data is reproduced, and output to the video / audio output processing unit 24.

  The video / audio output processing unit 24 receives control of the control unit 30 through the I / O port 315 and processes image data and audio data to be output. The image data and audio data as digital data are The image signal and the audio signal which are supplied to the control unit 30 through the / O port 315 and converted into analog signals by D / A conversion are output to the output terminals 26V and 26A. The reproduced image based on the image data input to the control unit 30 through the I / O port 315 is reproduced and displayed on the screen of the LCD 32 based on the control of the LCD controller 306.

[About data compression and multiplexing]
Next, data compression and multiplexing processing in the DSP 21 will be described in more detail with reference to FIG.

  That is, in the DSP 21, as described above, the image data from the imaging signal processing unit 14 is subjected to control processing according to setting conditions and control conditions at the time of imaging, and then supplied to the video encoding unit 202 through the buffer 201.

  The video encoding unit 202 configures a GOP composed of at least one I picture and a plurality of P pictures and B pictures, and compresses this GOP as a writing unit for a DVD as a recording medium.

  That is, in the video encoding unit 202, first, the picture order rearrangement unit 2021 rearranges the picture order in the GOP as described with reference to FIG.

  Then, in compression encoding section 2022, for I picture, intra-frame encoding processing is performed to perform data compression, and for P picture, between frames using the correlation with the preceding I picture or P picture. Data compression is performed by performing predictive coding processing. For B pictures, data compression is performed by performing interframe predictive coding processing using correlation with front and rear I pictures or P pictures. For P pictures, there is also an image portion for which intraframe coding processing is performed.

  As described above, the image data subjected to the compression encoding process in the compression encoding unit 2022 is subjected to the variable length encoding process using the Huffman code or the like in the variable length encoding unit 2023, and then the buffer 203. To the multiplex unit (multiplexing unit) 211.

  In the DSP 21, the audio data from the input audio signal processing unit 16 is captured in synchronization with the imaging timing, supplied to the audio encoding unit 205 through the buffer 204, and a predetermined amount of audio data (referred to as an audio frame) is received. It is compression-encoded as a decoding reproduction unit. The compressed and encoded audio data is supplied to the multiplex unit 211 through the buffer 206.

  Further, in the DSP 21, the shooting time information input from the I / O port 313 is supplied to the additional information processing unit 208 through the buffer 207, and additional information to be added to each frame of the image is generated. Then, the generated additional information is supplied to the multiplex unit 211 through the buffer 209.

  The multiplex unit 211 packetizes image data, audio data, and additional information, and multiplexes the image data, audio data, and additional information. In this case, a pack is formed by adding a pack header to one or a plurality of packet data and grouping them. The multiplexed output data from the multiplex unit 211 is output to the DVD signal processing unit 22 and written to the DVD 40 by the writing / reading unit 23 as described above.

  In order to synchronize the image data and the audio, in the MPEG system, a time stamp is provided for each unit of image data or audio data decoding / playback (image data is one frame unit, and audio data is an audio frame unit). Management information called is added.

  This time stamp information is supplied from the time management information generation unit 210 to the multiplex unit 211. This time stamp is given a time base by an SCR (System Clock Reference; system time base reference value).

  In the multiplex unit 211, the time stamp is added to the packet header when there is a head of decoding / reproduction unit information in the packet, and is added when there is no head of decoding / reproduction unit information in the packet. Not.

  There are two types of time stamps: PTS (Presentation Time Stamp) and DTS (Decoding Time Stamp).

  The PTS is time management information for reproduction output of information of a decoding reproduction unit. That is, the PTS is time management information for when to reproduce and output the information of the decoding reproduction unit.

  Further, the time stamp DTS is time management information for decoding the information of the decoding reproduction unit. As described above, this is provided in correspondence with the fact that the reproduction order and the decoding order are different in the MPEG system by rearranging the order of pictures during encoding.

  If the time stamp PTS and the DTS are different, both time stamps are added to the packet header, and if they match, generally only the time stamp PTS is added to the packet header.

  As described above, the time stamp PTS corresponds to the original arrangement order of the frames of the image data, and is time management information corresponding to each frame on a one-to-one basis. For audio data, the time stamp DTS is not added, and only the time stamp PTS is added.

  By the way, in this embodiment, as described above, the additional information is generated corresponding to each frame of the image data. However, as described above, in the MPEG system, the frame unit image information, that is, In combination with the fact that the order of pictures is changed during the encoding process, it is necessary to devise a method for associating additional information for each frame with image information of the corresponding frame.

  In this embodiment, it is used that the PTS of the above-described time management information corresponds to the original arrangement order of the frames of the image data and is one-to-one time management information for each frame. Thus, the PTS is used as a medium to associate the image data in units of frames as the decoding / reproduction unit with the additional information about the image data. This association process is performed in the multiplex unit 211.

  Hereinafter, some examples of processing for associating image frames with additional information in the multiplex unit 211 will be described.

[First example of association between image frame and additional information]
The first example is a case of recording in the DVD-Video format. The logical arrangement of recording data on the optical disc in the DVD-Video format is as shown in FIG.

  That is, as shown in FIG. 3A, the recording area of the optical disc is divided into a lead-in area, a data zone area, and a lead-out area in order from the innermost side when viewed in the radial direction of the optical disc. Data such as management information and image data is recorded in the data zone area.

  The data zone area is divided into a UDF (Universal Disc Format) area, a VMG (Video ManaGer) area, and a real-time data recording area in order from the lead-in area side. The UDF area and the VMG area are management information recording areas for managing files such as image data recorded on the optical disc. The UDF area is for supporting the UDF standard and the ISO9660 standard so that an optical disk can be read by a computer. The VMG area is a recording area for DVD management information.

  The real-time data recording area is an area for recording image data and audio data. As shown in FIG. 3B, data is recorded in units of data groups called VTS (Video Title Set).

  As shown in FIG. 3C, the VTS includes VTSI (Video Title Set Information), VTS Menu_VOBS (Video Object Set For the VTS Menu), and VTSTT_VOBS (Video Object Set TV). VTSI).

  In VTSTT_VOBS, compressed data in the MPEG2-PS format is recorded with an information block called VOBU (Video Object Unit) as a read / write unit, as shown in FIG. In VOBU, image data includes 1 GOP, and audio data includes information corresponding to the 1 GOP. That is, it is equivalent to reading / writing from / to the optical disc in units of 1 GOP. In this embodiment, as will be described later, the VOBU includes additional information for each piece of frame-unit image data constituting the GOP.

  In the VTSI, management information such as recording position information such as image data is recorded. Also, a root menu such as image data is recorded in VTSM_VOBS. This VTSM_VOBS is optional. Furthermore, VTSI_BUP is VTSI backup information.

  As shown in FIG. 3E, the read / write unit data VOBU is composed of a plurality of packs PCK. The pack PCK is composed of one or a plurality of packets. In this example, the pack is composed of one packet. In the DVD-Video format, one pack is defined as 2048 bytes (corresponding to one sector).

  One pack in this example is as shown in FIG. That is, the head of the pack PCK is the pack header. A packet header is located next to the pack header, and packet data including image data and audio data is located following the packet header. The packet format of the image data and audio data is a private 1 packet.

  The pack header includes the above-described time-based information SCR. The head of the packet header is a packet start code, and the packet header includes flag information such as a PTS flag and a DTS flag, and time stamp information PTS and DTS. The PTS flag and the DTS flag indicate whether or not the time stamp information PTS and DTS is included in the packet. As described above, the time stamp information PTS and DTS is added to the packet header when the head of the picture is included in the packet.

  In the first embodiment, the additional information constitutes one pack (one packet) for each frame. As described above, in this example, the additional information packet includes the time stamp PTS of the corresponding image data. Therefore, in this example, the packet of the additional information is in a private 1 packet format. In the first embodiment, the pack of additional information is multiplexed before the pack of image data to which the time stamp PTS is added.

  The multiplexing process in the first embodiment will be described with reference to FIG. FIG. 4 is a diagram for explaining the multiplexing process of the additional information and the image data for the image data for 1 GOP, and the multiplexing of the audio data is omitted for simplicity of explanation. In this example, 1 GOP is composed of 15 frames (15 pictures).

  FIG. 4A shows image data and additional information input to the buffers 201 and 207. B0, B1, I2, B3, B4, P5... P14 indicate image data in units of frames in consideration of the picture type. B is a B picture, I is an I picture, P is P pictures are shown respectively. Each suffix number is assigned a consecutive number in units of frames regardless of the picture type.

  A0 to A14 indicate additional information about the image frames having the suffix numbers. That is, each of the additional information A0 to A14 is additional information for image frames having the same suffix number.

  The image data shown in FIG. 4A is rearranged in the picture order in the picture order rearrangement unit 2021 of the video encoding unit 202 to be as shown in FIG. 4B. On the other hand, in this example, since the arrangement order of the additional information is not changed, the additional information shown in FIG. 4B has the same arrangement order as the additional information shown in FIG. . Therefore, as the arrangement order, it is difficult to associate each picture with additional information about it.

  As described above, in the multiplex unit 211, the image data of each picture includes a time stamp PTS, which is time management information for reproduction output, included in the header of a packet including the head of the picture. . A time stamp PTS # (# is a suffix number, and the earlier the time is, the lower the number) shown in the lower part of each picture in FIG. 4B, is the time stamp PTS added to each picture. Yes. Since the reproduction output is in the original frame order, the suffix number of each picture and the suffix number of the time stamp PTS are the same.

  Similarly, in this embodiment, in the multiplex unit 211, a time stamp PTS having the same suffix number is assigned to additional information for each picture. Therefore, even if the picture order is rearranged, it is possible to associate which additional information is associated with which picture by referring to the time stamp PTS.

  The compressed image data of each picture is recorded as one or a plurality of packs as shown in FIG. In FIG. 4C, V_PCK indicates a video pack (pack of image data), and I #, P #, and B # at the end thereof are the pictures of FIGS. 4A and 4B. It shows that it is compressed image data.

  As described above, the time stamp PTS is included in the packet header of a packet including the head of each picture. In this example, since the pack includes one packet, the time stamp PTS is included in the pack including the packet.

  In this embodiment, each additional information pack ARI_PCK # (# is a suffix number) is multiplexed immediately before the video pack including the time stamp PTS, as shown in FIG.

  Note that the first pack NV_PCK of the VOBU containing the GOP data is called a navigation pack, and this includes reproduction management information such as data search information (for example, control information for instructing which sector to jump to when jumping). Is included.

  The additional information includes shooting time information (recording time information) for each frame and camera setting information. As described above, the camera setting information includes information such as the F number, exposure time, exposure program, exposure correction value, AGC gain value, lens focal length, flash, white balance adjustment value, and digital zoom magnification. .

  In the first example described above, the time stamp PTS in the video pack V_PCK is multiplexed one pack at a time just before the pack in which the time stamp PTS is added to the packet header. Immediately after, one pack may be multiplexed.

[Second example of association between image frame and additional information]
In the first example described above, the additional information pack ARI_PCK # for each of the 15 pictures constituting the GOP is immediately before or immediately after the pack in which the time stamp PTS of the video pack V_PCK is added to the packet header. However, the additional information pack ARI_PCK # for each of the 15 pictures may be collectively arranged at a specific position in the VOBU.

  This second example is an example in that case. As shown in FIG. 5B, in each of the VOBUs shown in FIG. 5A, for each of the 15 pictures immediately after the navigation pack NV_PCK. Additional information packs ARI_PCK0 to ARI_PCK14 are arranged together.

  In the additional information packs ARI_PCK <b> 0 to ARI_PCK <b> 14, the same time stamp PTS as the time stamp PTS added to the picture corresponding to each additional information is added, as in the first example.

  In the above description, the additional information packs ARI_PCK0 to ARI_PCK14 are arranged together immediately after the navigation pack NV_PCK. However, they can be arranged together at the end of the VOBU. You may make it arrange | position collectively in the intermediate | middle predetermined position.

[Data separation and data compression / decompression processing]
Next, image data, audio data, and additional information separation processing and image data and audio data compression / decompression processing in the DSP 21 will be described with reference to FIG.

  Data read from the DVD 40 in units of VOBU by the writing / reading unit 23 is separated into an image data pack, an audio data pack, and an additional information pack in the demultiplexer 221.

  The demultiplexer 221 separates the pack header and packet header from each pack and supplies the header information to the control unit 220. The control unit 220 analyzes header information, extracts time stamps PTS and DTS, and controls decoding processing and reproduction output of image data, audio data, and additional information.

  The demultiplexer 221 supplies the compressed image data extracted from the separated image data pack to the video decoder unit 225 through the buffer 222, and the compressed audio data extracted from the separated audio data pack passes through the buffer 223 to the audio decoder. The additional information supplied to the unit 226 and extracted from the separated additional information pack is supplied to the additional information decoder unit 227 through the buffer 224.

  Then, the video decoder unit 225 performs compression / decompression processing of the image data in the order according to the time stamp DTS according to the control from the control unit 220, restores the image data in units of frames, and outputs it to the buffer 228. Then, the control unit 220 controls to read out the image data in units of frames from the buffer 228 in the order according to the time stamp PTS included in the image data pack and supply the read image data to the video signal processing unit 231. .

  Also, the audio decoder unit 226 performs audio data compression / decompression processing sequentially in accordance with the control from the control unit 220, restores the audio data, and outputs it to the buffer 229. Then, the control unit 220 controls to read out the audio data from the buffer 229 in the order according to the time stamp PTS included in the audio data pack and supply the audio data to the audio signal processing unit 232.

  Further, the additional information decoder unit 227 reproduces additional information about the image data in units of frames in accordance with the control from the control unit 220 and outputs it to the buffer 230. In this case, in the additional information decoder unit 227, the additional information is generated as image data control information Av, audio data control information Aa, and display information Ad such as shooting date and time, respectively. Output to the buffer 230.

  Then, the control unit 220 reads additional information about the image data in units of frames from the buffer 230 in the order according to the time stamp PTS included in the additional information pack, but the control information Av for the image data is The audio signal control information Aa is supplied to the video signal processing unit 231, and the display information Ad is supplied to the OSD (On Screen Display) circuit 233.

  In this case, since the image data control information Av and the audio data control information Aa are supplied from the buffer 230 to the video signal processing unit 231 and the audio signal processing unit 232 according to the time stamp PTS, the control information Av and Aa is image data and audio data of a corresponding frame, and is automatically associated. Therefore, the image data and the audio data are appropriately controlled by the corresponding control information Av and Aa.

  The video signal processing unit 231 performs image data processing corresponding to the image data control information Av. For example, image data control corresponding to auto white balance adjustment at the time of shooting is performed.

  The audio signal processing unit 232 performs audio data control processing according to the audio data control information Aa. For example, since the AGC characteristics are changed depending on whether the sound is picked up by the built-in microphone or the sound picked up by the external microphone, control processing corresponding to that is performed.

  The display information Ad is display information such as shooting date, shooting time, and zoom magnification, and is supplied to the combining unit 234 through the OSD circuit 233 and combined with image data from the video signal processing unit 231. The image data from the synthesis unit 234 is supplied to the control unit 30 through the video / audio output processing unit 24 and displayed on the screen of the LCD 32.

  As described above, the image data and the additional information display information Ad are read from the buffers 228 and 230 based on the same time stamp PTS for the corresponding frames, and are therefore combined. The shooting time for each frame is correctly displayed on the screen.

[Other Embodiments]
[First other example]
In the above-described embodiment, the additional information is recorded as one pack for each of the image frames. However, for one VOBU, one pack is set as the additional information pack ARI_PCK, and the VOBU is included in the one pack. In this example, additional information for 15 pictures (frames) may be recorded together.

  FIG. 7 shows an example of the data format in that case. That is, in this example, for each VOBU shown in FIG. 7A, as shown in FIG. 7B, a pack ARI_PCK of additional information for one pack is arranged immediately after the navigation pack NV_PCK. .

  The additional information pack ARI_PCK includes a set of additional information and PTS for each image frame for 15 image frames constituting the GOP.

  That is, as shown in FIG. 7C, a set of data in which additional information data ARI_DT # (# is a frame number) for each image frame and time stamp PTS # added to the image frame are associated with each other. ARI_SET # is generated. Then, 15 sets of data ARI_SET0 to ARI_SET14 for 15 frames constituting the GOP are sequentially included in the additional information pack ARI_PCK.

  In this example, as the packet ARI_PCK for additional information, the private 1 packet can be used as in the above example, but the private 2 packet shown in FIG. 7D can also be used. It is.

  In the case of this example, at the time of data reproduction, PTS is detected from each set of data ARI_SET, and using this PTS, time adjustment is performed on the image data and audio data in the same manner as in the above example, and additional information Can be used for controlling image data and audio data, and can also be used as display data for a screen.

[Second other example]
In the example shown in FIG. 4, the additional information is a video in which the additional information pack with the time stamp PTS added is left in the original time series order regardless of the rearrangement of the picture order and the PTS is included in the packet header. Although the information is added before (or after) the pack V_PCK, the additional information is also rearranged in accordance with the rearrangement of the picture order of the image data, and the PTS before the video pack V_PCK included in the packet header ( Or you may make it add to after).

  In the case of the second other example, the video pack V_PCK and the additional information pack ARI_PCK have the same PTS included in the packet header, and are easily associated with each other.

[Third other example]
According to the second other example described above, the additional information pack ARI_PCK is a video pack of an image frame corresponding to the additional information, and the PTS is added before or after the pack included in the packet header. The correspondence between the additional information and the image frame to which the additional information corresponds can be easily related even if there is no PTS in the additional information.

  Therefore, in the third other example, as in the second other example, the additional information pack ARI_PCK is a video pack of an image frame corresponding to the additional information, and the PTS is included in the packet header. It is added before or after the pack, but the PTS is not included in the additional information pack. Therefore, in the third other example, the private 2 packet shown in FIG. 7D can be used as the packet of the additional information pack.

  In the case of this third other example, at the time of reproduction, additional information corresponding to each image frame is extracted from the correspondence relationship by the recording position, and is added to, for example, a video pack. Based on the PTS, the reproduction output is managed, and the order of the corresponding additional information is rearranged to be reproduced and output, so that the reproduction output image data and audio data are associated with each other in the same manner as described above. It is used as control information and display information.

[Fourth other example]
The fourth other example is a modification of the example shown in FIG. 5 and is a case where the additional information pack ARI_PCK does not include the time stamp PTS. That is, in the fourth other example, when the image data is restored, the image data is reproduced by being returned to the original picture order (frame order) by the PTS, and therefore the order in which the image frames are reproduced based on the PTS. The additional information pack ARI_PCK is also arranged. Therefore, the additional information pack ARI_PCK is arranged in the arrangement order as shown in FIG.

  In the case of this fourth other example, at the time of reproduction, the additional information corresponding to each image frame is sequentially added to the image frames that are reproduced and output with the arrangement order restored. By outputting, it can be used as control information and display information in correspondence with the restored reproduction output image data and audio data.

[Fifth other example]
The above example is a case where the present invention is applied when recording in the DVD-Video format. However, the VOBU data in the DVD-Video Recording (DVD-VR) format is processed in the same manner as described above. Can be applied.

  Further, in the case of the DVD-VR format, as shown in FIG. 8, the RDI data area of the RDI packet in the pack RDI_PCK arranged at the head of the VOBU has the same format as the example of FIG. Additional information can also be arranged.

[Sixth other example]
In the above examples, additional information is generated and added for each image frame. However, additional information need not be generated for all image frames.

  The sixth other example is an example in that case. In this example, for example, one frame every two image frames, one frame every three image frames,... As described above, with respect to an image frame as a decoding / playback unit for each predetermined time, additional information about the image frame is generated and recorded in association with management information for decoding / playback processing for the image frame. To do.

  If the above-described time stamp PTS is used as the management information for the decoding / playback processing, the additional information about the image frame is synchronized with the playback of the information of the image frame as the decoding / playback unit for each predetermined time. It is easy to reproduce and use.

  For example, when the additional information is the recording time, the recording time for the image frame in which the additional information is not recorded can be easily calculated from the recording time as the recorded additional information. Further, for image frames in the vicinity of image frames in which additional information is recorded, the additional information may be used as additional information for image frames in the vicinity thereof.

[Other variations]
In the above example, the case where the additional information is added to one frame unit of the image data has been mainly described. However, the above example is also applicable to the case where the additional information is added using the frame of the audio data as a decoding reproduction unit. .

It is a figure which shows the structural example of embodiment of the imaging device by this invention. It is a figure for demonstrating the structural example of the data encoding and multiplexing process part in FIG. It is a figure which shows the logical arrangement | positioning on the optical disk by a DVD-Video format. It is a figure for demonstrating an example of the principal part of embodiment of the imaging method by this invention. It is a figure for demonstrating the other example of the principal part of embodiment of the imaging method by this invention. It is a figure for demonstrating the structural example of the data separation process in FIG. 1, and a data decoder part. It is a figure for demonstrating the other example of the principal part of embodiment of the imaging method by this invention. It is a figure for demonstrating the other example of the principal part of embodiment of the imaging method by this invention. It is a figure for demonstrating rearrangement of the picture in GOP in an MPEG system.

Explanation of symbols

  202 ... Video encoding unit, 2021 ... Picture order rearrangement unit, 2022 ... Compression encoding unit, 2023 ... Variable length encoding unit, 208 ... Additional information processing unit, 211 ... Multiplex unit, 220 ... Control unit, 221 ... Decoding Multiplexer, 225: Video decoder unit, 226 ... Audio decoder unit, 227 ... Additional information decoder unit, 231 ... Video signal processing unit, 232 ... Audio signal processing unit, 233 ... OSD circuit, 234 ... Synthesis unit

Claims (16)

An image sensor;
An imaging optical system for forming a subject image on the imaging element;
Data compression means for data compression processing of image information from the image sensor;
Recording means for recording the compressed image information on a recording medium;
Generates additional information indicating an environment at the time of imaging of the image information corresponding to one or a plurality of the decoding reproduction units among the information of the predetermined decoding reproduction units of the compressed image information included within a predetermined time. And generating means for
The recording means includes
The first pack to which the additional information is stored, and, the one or more of the decoding of said predetermined plurality of said decoded reproduction unit including a reproduction unit information and the predetermined plurality of said decoded reproduction unit corresponding to the additional information As a unit, the second pack is arranged before or after the corresponding first pack as a logical arrangement of recording data. so that the imaging device for recording the image information on the recording medium. The imaging device according to claim 1,
The additional information, including an imaging device imaging time of the image information acquired by the imaging device. The imaging device according to claim 1,
The additional information, including an imaging device imaging condition information when the image information is acquired by the imaging device. The imaging device according to claim 1,
The additional information, the exposure control information, F number, the focal length including imaging device. The imaging device according to claim 1,
The additional information, the predetermined plurality of decoding the plurality of additional information der Ru imaging device extracted in correspondence with the information of the decoding and reproducing unit for each reproduction unit. The imaging device according to claim 1,
The additional information, the predetermined plurality of decoding one said decoding corresponding to the extracted additional information der Ru imaging device in the reproduction units included in the reproduction unit. The imaging device according to claim 1,
The information of the predetermined plurality of decoding reproduction units of the compressed image information included within the predetermined time corresponds to 1 GOP, and the additional information extracted corresponding to one picture included in the 1 GOP information that is recorded imaging device. The imaging device according to claim 1,
The unit of the data information and said additional information recorded on a recording medium said predetermined plurality of said decoded reproduction unit, the recording medium Ru write unit der is a unit of writing / reading of data in the imaging device . A data compression process for compressing image information from the image sensor;
A recording step of recording the image information compressed in the data compression step on a recording medium;
Prior to the recording step, an environment at the time of imaging of the image information corresponding to one or a plurality of the decoded reproduction units among the information of the predetermined plurality of decoded reproduction units of the compressed image information included within a predetermined time. A generating step for generating additional information indicating
In the recording step,
The first pack to which the additional information is stored, and, the one or more of the decoding of said predetermined plurality of said decoded reproduction unit including a reproduction unit information and the predetermined plurality of said decoded reproduction unit corresponding to the additional information As a unit, the second pack is arranged before or after the corresponding first pack as a logical arrangement of recording data. so that the imaging method of recording the image information on the recording medium. The imaging method according to claim 9,
The additional information, including IMAGING METHOD the shooting time of the image information acquired by the imaging device. The imaging method according to claim 9,
The additional information, including IMAGING METHOD shooting condition information when the image information is acquired by the imaging device. The imaging method according to claim 9,
The additional information, the exposure control information, F number, including IMAGING METHOD focal length. The imaging method according to claim 9,
The additional information, the predetermined plurality of decoding the plurality of additional information der Ru IMAGING METHOD extracted in correspondence with the information of the decoding and reproducing unit for each reproduction unit. The imaging method according to claim 9,
The additional information, the predetermined plurality of decoding one said decoding corresponding side information der Ru IMAGING METHOD extracted playback units included in the reproduction unit. The imaging method according to claim 9,
The information of the predetermined plurality of decoding reproduction units of the compressed image information included within the predetermined time corresponds to 1 GOP, and the additional information extracted corresponding to one picture included in the 1 GOP IMAGING mETHOD information that is recorded. The imaging method according to claim 9,
The unit of the data information and said additional information recorded on a recording medium said predetermined plurality of said decoded reproduction unit, the recording Ru write unit der is a unit of writing / reading of data in the media IMAGING METHOD .

Images