JP5247346B2 - Imaging apparatus, control method and program thereof, image processing apparatus, and image processing method - Google Patents

Description

  The present invention relates to a technique for suppressing image quality deterioration due to foreign matter adhering to the surface of an optical low-pass filter or the like disposed in front of an image pickup device in an image pickup apparatus using an image pickup device such as a CCD or a CMOS sensor, and particularly at the time of moving image shooting. The present invention relates to a technology for suppressing image quality deterioration due to foreign matter.

  In an interchangeable lens digital camera, dust or the like floating in the air may enter the camera body when the lens is removed from the camera body. In addition, various mechanical units such as a shutter mechanism are disposed inside the camera. By operating these mechanical units, dust such as metal pieces is generated in the camera body. In some cases.

  When foreign matter such as dust or dust adheres to the surface of an optical low-pass filter disposed in front of an image sensor that is an optical element that constitutes an image capturing unit of a digital camera, the foreign matter causes shadows on the captured image. And the quality of the photographed image is deteriorated.

  In a camera using a silver salt film, since the film is sent each time an image is taken, it is very rare that the same foreign matter is continuously reflected at the same position in the image. However, in a digital camera, there is no problem that a film frame is sent every time shooting is performed, and therefore, there is a problem in that the same foreign matter is continuously captured at the same position of a captured image.

  In order to solve such a problem, a method of correcting a pixel in which a foreign object is reflected by using a signal of a pixel around the pixel can be considered. As a technique for correcting such pixels, for example, Patent Document 1 (Japanese Patent Laid-Open No. 6-105241) proposes an image defect correction method for correcting pixel defects of an image sensor. In Patent Document 2 (Japanese Patent Laid-Open No. 2004-242158), in order to simplify the setting of pixel defect position information, an extension of an image file shot in the dust acquisition mode may be different from that of a normal image. Proposed. In this way, the dust information image is automatically determined on the PC (personal computer) side, and the correction target image is corrected using the information.

  In recent years, in order to handle moving image information as digital data and use it for storage and transmission, a technique for encoding with high compression rate and high image quality has been proposed and widely used.

  The Motion JPEG system performs encoding by adapting still image encoding (for example, JPEG (Joint Photographic Coding Experts Group) encoding) to each frame. JPEG encoding is basically an encoding method for still images, but there are products that can handle moving images by high-speed processing.

  An outline of the JPEG method will be briefly described. Image data is divided into blocks of a predetermined size (for example, blocks of 8 × 8 pixels), two-dimensional discrete cosine transform is performed for each block, and the transform coefficient is quantized linearly or nonlinearly. The quantized transform coefficient is Huffman encoded (variable length encoding) as follows. That is, for the DC component of the transform coefficient, the difference value from the DC component of the neighboring block is Huffman coded, and the AC component is serialized from the low frequency component to the high frequency component by zigzag scanning. Then, Huffman coding is performed on a set of consecutive invalid component values having a value of 0 and subsequent valid component values.

  On the other hand, H.264 is an encoding method aiming at higher compression ratio and higher image quality. H.264 (MPEG4-Part10 AVC). This H. H.264 is known to achieve higher encoding efficiency compared to conventional encoding schemes such as MPEG2 and MPEG4, although a larger amount of computation is required for encoding and decoding (Non-patent) Reference 1).

  FIG. 1 is a diagram illustrating a configuration of an image processing apparatus that compresses image data using an H.264 system.

  In FIG. 16, input image data is divided into macro blocks and sent to the subtraction unit 401. The subtraction unit 401 calculates a difference between the image data and the predicted value and outputs the difference to an integer DCT (Discrete Cosine Transform) conversion unit 402. The integer DCT transform unit 402 performs integer DCT transform on the input data and outputs the result to the quantization unit 403. The quantization unit 403 quantizes the input data. One of the quantized data is sent to the entropy encoding unit 415 as difference image data. The other is inversely quantized by the inverse quantization unit 404 and then subjected to inverse integer DCT transform by the inverse integer DCT transform unit 405. The predicted value is added to the data subjected to the inverse integer DCT conversion by the adding unit 406. Thereby, the image is restored.

  One of the restored images is sent to the frame memory 407 for intra (intraframe) prediction. The other is subjected to deblocking filter processing by the deblocking filter 409 and then sent to the frame memory 410 for inter (interframe) prediction. The image in the frame memory 407 for intra prediction is used for intra prediction in the intra prediction unit 408. In this intra prediction, the value of an adjacent pixel of an already encoded block is used as a prediction value in the same picture.

  An image in the frame memory 410 for inter prediction is composed of a plurality of pictures as will be described later. The plurality of pictures are divided into two lists, “List0” and “List1”. A plurality of pictures divided into two lists are used for inter prediction in the inter prediction unit 411. After the inter prediction is performed, the internal image is updated by the memory controller 413. In inter prediction performed by the inter prediction unit 411, a prediction image is determined using an optimal motion vector based on the result of motion detection performed by the motion detection unit 412 on image data of different frames.

  As a result of intra prediction and inter prediction, an optimal prediction result is selected by the selection unit 414. The motion vector is sent to the entropy encoding unit 415 and encoded together with the difference image data. In this way, an output bit stream is formed.

  Here, H. H.264 inter prediction will be described in detail with reference to FIGS.

  H. In the H.264 inter prediction, a plurality of pictures can be used for prediction. For this reason, two lists (“List0” and “List1”) are prepared to specify the reference picture. Each list is assigned a maximum of five reference pictures.

  In the P picture, only “List0” is used and forward prediction is mainly performed. In B picture, “List0” and “List1” are used to perform bi-directional prediction (or prediction only forward or backward). That is, a picture for forward prediction is mainly assigned to “List0”, and a picture for backward prediction is mainly contained in “List1”.

  FIG. 17 shows an example of a reference list used for encoding. Here, a case where the ratio of I picture, P picture, and B picture is standard will be described as an example. That is, a case where an I picture has an interval of 15 frames, a P picture has an interval of 3 frames, and a B picture between them has an interval of 2 frames will be described. In FIG. 17, reference numeral 1001 denotes image data arranged in the display order. In the square of the image data 1001, a number indicating the type of picture and the display order is entered. For example, picture I15 is the 15th I-picture in display order, and only intra prediction is performed. The picture P18 is the 18th P picture in the display order, and only forward prediction is performed. Picture B16 is the 16th B picture in display order, and performs bi-directional prediction.

  The order of encoding is different from the display order and is the order of prediction. That is, in FIG. 17, the encoding order is “I15, P18, B16, B17, P21, B19, B20,.

  In FIG. 17, reference numeral 1002 denotes a reference list (List0). The reference list (List0) 1002 stores pictures that have been once encoded and decoded. For example, when inter prediction is performed on the picture P21 (the 21st P picture in display order), a picture that has already been encoded and decoded in the reference list (List0) 1002 is referred to. In the example illustrated in FIG. 17, pictures P06, P09, P12, I15, and P18 are stored in the reference list 1002.

  In inter prediction, for each macroblock, a motion vector having an optimal prediction value is obtained from the reference picture in the reference list (List0) 1002 and encoded. The pictures in the reference list (List0) 1002 are given reference picture numbers in order and are distinguished (given separately from the numbers shown).

  When the encoding of the picture P21 is completed, the picture P21 is newly decoded and added to the reference list (List0) 1002. The oldest reference picture (here, picture P06) is removed from the reference list (List0) 1002. After this, encoding is performed with pictures B19 and B20, and continues to picture P24. The state of the reference list (List0) 1002 at this time is shown in FIG.

  FIG. 19 shows how the reference list changes for each picture.

  In FIG. 19, pictures are encoded in order from the top. Also, the picture being encoded and the contents of the reference lists (List0 and List1) for the picture being encoded are shown. As shown in FIG. 19, when a P picture (or I picture) is encoded, the reference list (List0 and List1) is updated, and the oldest picture in the reference list (List0 and List1) is removed. In this example, the reference list (List1) has only one picture. This is because if the number of pictures referred for backward prediction is increased, the amount of buffer until decoding increases. In other words, this is because a reference to a backward picture that is too far away from the picture being encoded is avoided.

  In the example given here, pictures used for reference are an I picture and a P picture, and all of the I picture and P picture are sequentially added to the reference list (List0 and List1). Also, only the P picture is used in the reference list (List1) for backward prediction. The reason described above is that it is a picture structure that would normally be used most often. However, the configuration of pictures in such a reference list is just one example that would be most often used. H.264 itself has a higher degree of freedom in the configuration of the reference list.

  For example, it is not necessary to add all of the I picture and P picture to the reference list, and it is also possible to add a B picture to the reference list. Also defined are long-term reference lists that remain in the reference list until explicitly specified. FIG. 20 shows how the reference list changes when a B picture is added to the reference list. When a B picture is added to the reference list, it is usually considered that the encoded picture is added to the reference list every time all the B pictures are encoded.

  Compact digital cameras capable of recording moving images using these encoding methods have also been developed and commercialized, and users can easily view images using these devices, personal computers, DVD players, etc. Is possible.

  Conventionally, an image pickup apparatus such as a digital video camera is generally equipped with an electronic zoom. Electronic zooming thins out the image signal read from the image sensor or intermittently reads the image signal from the image sensor, then generates an interpolation signal from the image signal, and inserts the interpolation signal into the image signal. This is a function that electronically enlarges the image magnification of the subject. As a result, an image enlarged at an arbitrary magnification can be obtained.

  For example, in FIG. 6A, reference numeral 61 denotes a photographing screen of the image sensor, and the image signal of the shaded portion 62 therein is enlarged to the same size as the photographing screen 61 as shown in FIG. 6B. Since an enlarged image is obtained by electronic processing, it is called electronic zoom. In the electronic zoom, an arbitrary enlargement ratio can be selected, and it is easy to continuously change the enlargement ratio.

  At an arbitrary magnification, the output scanning line is positioned between the scanning line obtained from the output of the image sensor and the scanning line. Interpolate. As a result, image quality deterioration due to enlargement can be suppressed to some extent, but it is difficult to achieve a general zoom ratio of 6 times, 8 times, or 10 times while securing a certain level of image quality using only the electronic zoom. Therefore, at present, optical zoom and electronic zoom are used together.

  By using the optical zoom and the electronic zoom together, it is possible to realize a zoom lens system that is small and light and has a large zoom magnification. In such a system, when performing so-called zoom-up to change from a small zoom magnification to a large zoom magnification, first, the enlargement ratio in the electronic zoom is increased by a factor of 1, that is, without performing the enlargement in the electronic zoom. Change the zoom magnification only with optical zoom. Then, after the optical zoom reaches the tele end, the zoom magnification is changed using the electronic zoom.

  In addition, when performing so-called zoom-down in which the zoom magnification is changed from a large zoom magnification to a small zoom magnification, the zoom magnification is first changed only by the electronic zoom, and after the enlargement ratio in the electronic zoom reaches 1 time, the optical zoom is performed. This is used to change the zoom magnification.

  Furthermore, in recent years, due to the improvement of semiconductor technology, An image sensor having a much larger number of pixels than the number of pixels defined in a format such as H.264 (for example, HD resolution) is adopted, and the image signal is intermittently read out from such a multi-pixel image sensor. An apparatus that generates an image signal with a prescribed number of pixels and records a higher quality image has also been proposed.

As a technique of electronic zoom in this type of image pickup apparatus, for example, Japanese Patent Application Laid-Open No. 2007-212273 proposes a method for suppressing deterioration in image quality during an electronic zoom operation.
JP-A-6-105241 JP 2004-242158 A JP 2007-212273 A ISO / IEC 14496-10, "Advanced Video Coding"

  Under such circumstances, in recent years, there is an increasing need for recording a moving image with higher pixels and higher definition not only in a compact type digital camera but also in an interchangeable lens type digital camera. However, as already mentioned, dust is attached to the surface of the image sensor due to various factors in an interchangeable lens digital camera, so if you record a movie as it is, it will always appear at the same position during movie playback. There was a possibility.

  In a conventional dust removal method in an interchangeable lens digital camera, information necessary for dust removal (for example, dust position and size information) and image data are recorded, and the image is later imported into a personal computer or the like. Garbage was removed by processing. That is, dust is reflected in the recorded image data. In the still image, the dust removal operation is performed one by one, but in the moving image, the dust removal must be performed over the entire recorded time, which is a very time-consuming operation.

  In addition, when performing moving image shooting, it is common to enlarge an image using an electronic zoom. However, when enlargement is performed by electronic zoom, the angle of view (effective pixel area of the image sensor) differs at each magnification of electronic zoom (the magnification of electronic zoom when electronic zoom is OFF is regarded as 1). For this reason, the position and size information necessary for dust removal does not coincide with the position and size of the dust actually reflected in the image, and there is a problem that dust correction processing cannot be performed successfully.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to enable dust removal processing to be performed on a moving image shot using an electronic zoom during moving image recording. .

In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes an imaging device that photoelectrically converts a subject image and generates an image signal based on an output signal from the imaging device. And an electronic zoom processing means for performing an electronic zoom process for cutting out and enlarging a part of the image signal, and a foreign object detection image including an image of the foreign object attached to the surface of the optical element disposed in front of the imaging element. A foreign object detection unit for detecting foreign object information, which is information including at least the position and size information of the foreign object, a foreign object information recording unit for recording the foreign object information in an image file, and an electronic zoom zoom in the electronic zoom processing unit. a zoom magnification recording means for recording magnification to the image file, and the foreign substance information, by using the zoom magnification of the electronic zoom, the position and size of the foreign substance before Converting the position and size of the foreign substance in an image after the electronic zoom process, characterized in that it and a foreign substance removing means for removing by image image processing foreign matter in the image after the electronic zoom processing.

The image pickup apparatus control method according to the present invention is a method for controlling an image pickup apparatus having an image pickup device that photoelectrically converts a subject image and generating an image signal based on an output signal from the image pickup device. A foreign object detection image including an electronic zoom process for performing an electronic zoom process for cutting out and enlarging a part of the image signal, and an image of a foreign object attached to the surface of an optical element disposed in front of the image sensor A foreign substance detection step for detecting foreign substance information that is information including at least the position and size information of the foreign substance, a foreign substance information recording step for recording the foreign substance information in a moving image file, and an electronic zoom in the electronic zoom processing step. and the zoom magnification recording process of the zoom magnification recorded in the moving image file, and the foreign substance information, by using the zoom magnification of the electronic zoom, the position of the foreign substance及Convert the size on the position and size of the foreign substance in an image after the electronic zoom processing, characterized in that it and a foreign matter removing step of removing the image the image processing of the foreign matter in the image after the electronic zoom processing .

The image processing apparatus according to the present invention also includes foreign object information, which is information including information on the position and size of a foreign object attached to the surface of an optical element disposed in front of an image sensor at the time of photographing, and electronic zoom at the time of photographing. An image processing apparatus for processing an image file in which a zoom magnification is recorded, wherein the position and size of the foreign matter in the image after the electronic zoom processing are determined using the foreign matter information and the zoom magnification of the electronic zoom. Foreign matter removing means for converting to the position and size of the foreign matter and removing the foreign matter image in the image after the electronic zoom processing by image processing based on the converted position and size of the foreign matter is provided.

The image processing method according to the present invention also includes foreign object information, which is information including information on the position and size of a foreign object attached to the surface of an optical element disposed in front of an image sensor at the time of photographing, and electronic zoom at the time of photographing. An image processing method for processing an image file in which a zoom magnification is recorded, wherein the position and size of the foreign matter are determined in the image after the electronic zoom processing using the foreign matter information and the zoom magnification of the electronic zoom. A foreign matter removing step of converting the position and size of the foreign matter into a foreign matter and removing the image of the foreign matter in the image after the electronic zoom processing based on the converted position and size of the foreign matter is provided.

  According to the present invention, it is possible to perform dust removal processing on a moving image shot using an electronic zoom during moving image recording.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus having an image processing function according to the first embodiment of the present invention. In the present embodiment, a single-lens reflex digital still camera with interchangeable lenses will be described as an example of an imaging apparatus. As an imaging apparatus, the present invention can be applied to a digital video camera or the like in which lenses can be exchanged.

  As shown in FIG. 1, the imaging apparatus of the present embodiment mainly includes a camera body 100 and an interchangeable lens type lens unit 300.

  In the lens unit 300, 310 is an imaging lens composed of a plurality of lenses, 312 is a diaphragm, and 306 is a lens mount that mechanically couples the lens unit 300 to the camera body 100. The lens mount 306 includes various functions for electrically connecting the lens unit 300 to the camera body 100. Reference numeral 320 denotes an interface for connecting the lens unit 300 to the camera body 100 in the lens mount 306, and 322 denotes a connector for electrically connecting the lens unit 300 to the camera body 100.

  The connector 322 has a function of transmitting control signals, status signals, data signals, and the like between the camera body 100 and the lens unit 300 and supplying currents of various voltages. The connector 322 may be configured to perform communication using not only electrical communication but also optical communication, voice communication, and the like.

  Reference numeral 340 denotes an aperture control unit that controls the aperture 312 in cooperation with a shutter control unit 40 that controls the sitter 12 of the camera body 100, which will be described later, based on photometric information from the photometry control unit 46. Reference numeral 342 denotes a focus control unit that controls focusing of the imaging lens 310, and 344 denotes a zoom control unit that controls zooming of the imaging lens 310.

  A lens system control circuit 350 controls the entire lens unit 300. The lens system control circuit 350 includes a memory for storing operation constants, variables, programs, and the like. Further, a non-volatile memory that holds identification information such as a unique number of the lens unit 300, management information, function information such as an open aperture value, a minimum aperture value, and a focal length, and current and past setting values is also provided.

  Next, the configuration of the camera body 100 will be described.

  Reference numeral 106 denotes a lens mount that mechanically couples the camera body 100 and the lens unit 300, and reference numerals 130 and 132 denote mirrors, which guide light beams incident on the imaging lens 310 to the optical viewfinder 104 by a single-lens reflex system. The mirror 130 may be either a quick return mirror or a half mirror. Reference numeral 12 denotes a focal plane shutter. Reference numeral 14 denotes a CCD, a CMOS sensor, or the like, which is an image sensor that photoelectrically converts a subject image. The imaging device 14 has not only a thinning-out readout mode in which analog signals are output from all pixels arranged in the imaging device but also a thinning-out readout mode in which analog signals are output by thinning out pixels in the horizontal direction and the vertical direction. Yes. By utilizing this thinning readout mode, an analog signal having the optimum number of pixels for display resolution and recording resolution can be obtained.

  An optical element 14 a such as an optical low-pass filter is disposed in front of the image sensor 14, and foreign matters such as dust attached to the surface of the optical element 14 are reflected in the image generated by the image sensor 14. Deteriorate image quality. The present embodiment relates to a technique for suppressing the image quality deterioration.

  The light beam incident on the imaging lens 310 is guided by the single-lens reflex system through the aperture 312, the lens mounts 306 and 106, the mirror 130, and the shutter 12, which are light amount limiting means, and is formed on the image sensor 14 as an optical image. .

  Reference numeral 16 denotes an A / D converter that converts an analog signal (output signal) output from the image sensor 14 into a digital signal. A timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14, the A / D converter 16, and the D / A converter 26, respectively, and is controlled by the memory control circuit 22 and the system control circuit 50.

  An image processing circuit 20 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 16 or the data from the memory control circuit 22. Further, the image processing circuit 20 performs predetermined arithmetic processing using the image data output from the A / D converter 16 as necessary. TTL (through-the-lens) type autofocus (AF) processing and automatic exposure (AE) for the system control circuit 50 to control the shutter control unit 40 and the focus adjustment unit 42 based on the obtained calculation result. Processing and flash pre-emission (EF) processing can be performed. Further, the image processing circuit 20 performs predetermined arithmetic processing using the image data output from the A / D converter 16, and also performs TTL auto white balance (AWB) processing based on the obtained arithmetic result. ing. The image processing circuit 20 performs image enlargement processing using pixel interpolation processing during electronic zoom.

  In the example shown in FIG. 1 in the present embodiment, the focus adjustment unit 42 and the photometry control unit 46 are provided exclusively. Therefore, the AF adjustment process, the AE process, and the EF process are performed using the focus adjustment unit 42 and the photometry control unit 46, and the AF process, the AE process, and the EF process using the image processing circuit 20 are not performed. It does not matter. Further, AF processing, AE processing, and EF processing are performed using the focus adjustment unit 42 and the photometry control unit 46, and further, AF processing, AE processing, and EF processing using the image processing circuit 20 are performed. It is good also as a structure.

  A memory control circuit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the image display memory 24, the D / A converter 26, the memory 30, and the compression / decompression circuit 32. Image data output from the A / D converter 16 is written into the image display memory 24 or the memory 30 via the image processing circuit 20, the memory control circuit 22, or only through the memory control circuit 22.

  Reference numeral 24 denotes an image display memory, 26 denotes a D / A converter, and 28 denotes an image display unit made up of a TFT type LCD or the like. Display image data written in the image display memory 24 is stored in the D / A converter 26. Via the image display unit 28. An electronic viewfinder (EVF) function can be realized by sequentially displaying captured image data using the image display unit 28. Further, the image display unit 28 can arbitrarily turn on / off the display according to an instruction from the system control circuit 50, and when the display is turned off, the power consumption of the camera body 100 can be significantly reduced. it can.

  Reference numeral 30 denotes a memory for storing captured still images or moving images, and has a storage capacity sufficient to store a predetermined number of still images or a predetermined amount of moving images. This makes it possible to write a large amount of images to the memory 30 at high speed even in continuous shooting or panoramic shooting in which a plurality of still images are continuously shot. In moving image shooting, it is used as a frame buffer for images written continuously at a predetermined rate. Further, the memory 30 can be used as a work area for the system control circuit 50.

  31 is a dust removal circuit (foreign matter removal circuit) for removing dust contained in image data by image processing using dust information stored in a nonvolatile memory 56 described later and optical information obtained from the lens unit 300. ).

  A compression / decompression circuit 32 compresses / decompresses image data using a known compression method. The compression / decompression circuit 32 reads an image stored in the memory 30, performs a compression process or an expansion process, and writes the processed data to the memory 30 again. It also has a function of compressing and encoding moving image data into a predetermined format, or expanding a moving image signal from predetermined compression-encoded data.

  An audio signal processing circuit 33 encodes an audio signal input from a microphone (not shown) into a predetermined code format, or has a function of decoding an audio signal from predetermined encoded data. Note that the digital camera of the present embodiment has a function of outputting audio data decoded by the audio signal processing circuit 33 via a speaker (not shown).

  Reference numeral 40 denotes a shutter control unit that controls the shutter 12 in cooperation with an aperture control unit 340 that controls the aperture 312 based on photometric information from the photometry control unit 46. A focus adjusting unit 42 performs AF (autofocus) processing. A light beam incident on the imaging lens 310 in the lens unit 300 is made incident as a single lens reflex system through a diaphragm 312, lens mounts 306 and 106, a mirror 130, and a focus adjustment submirror (not shown), thereby forming an optical image. The in-focus state of the captured image is measured.

  A photometry control unit 46 performs AE (automatic exposure) processing. A light beam incident on the image pickup lens 310 in the lens unit 300 is made incident as an optical image by being incident by a single lens reflex system through a diaphragm 312, lens mounts 306 and 106, a mirror 130, and a photometric sub mirror (not shown). Measure the exposure of the image. A flash 48 has an AF auxiliary light projecting function and a flash light control function. The photometry control unit 46 also has an EF (flash dimming) processing function in cooperation with the flash 48.

  Further, the AF control may be performed using the measurement result by the focus adjustment unit 42 and the calculation result obtained by calculating the image data from the A / D converter 16 by the image processing circuit 20. Furthermore, exposure control may be performed using the measurement result obtained by the photometry control unit 46 and the calculation result obtained by calculating the image data from the A / D converter 16 by the image processing circuit 20.

  Reference numeral 50 denotes a system control circuit that controls the entire camera body 100, and incorporates a known CPU. A memory 52 stores constants, variables, programs, and the like for operating the system control circuit 50.

  Reference numeral 54 denotes a notification unit for notifying the outside of an operation state, a message, and the like using characters, images, sounds, and the like in accordance with execution of a program in the system control circuit 50. As the notification unit 54, for example, a display unit that performs visual display using an LCD, an LED, or the like, or a sound generating element that performs voice notification, etc. are used. The notification unit 54 is configured by a combination of one or more of these. . In particular, in the case of a display unit, it is installed at a single or a plurality of locations near the operation unit 70 of the camera body 100 that are easy to see. In addition, the notification unit 54 has a part of its functions installed in the optical viewfinder 104.

  Among the display contents of the notification unit 54, the following are displayed on the image display unit 28 such as an LCD. First, there are displays relating to shooting modes such as single-shot / continuous-shot shooting display, self-timer display, and the like. In addition, there are displays relating to recording such as compression rate display, recording pixel number display, recording number display, and remaining image number display. In addition, there are displays relating to photographing conditions such as shutter speed display, aperture value display, exposure correction display, dimming correction display, external flash emission amount display, and red-eye reduction display. In addition, there are a macro shooting display, a buzzer setting display, a battery remaining amount display, an error display, an information display with a multi-digit number, and a recording medium 200 and PC 210 attachment / detachment state display. Furthermore, the attachment / detachment state display of the lens unit 300, the communication I / F operation display, the date / time display, the display indicating the connection state with the external computer, and the like are performed.

  In addition, among the display contents of the notification unit 54, examples of what is displayed in the optical finder 104 include the following. In-focus display, shooting preparation completion display, camera shake warning display, flash charge display, flash charge completion display, shutter speed display, aperture value display, exposure correction display, recording medium writing operation display, and the like.

  Reference numeral 56 denotes an electrically erasable / recordable non-volatile memory in which programs and the like to be described later are stored. For example, an EEPROM or the like is used.

  An optical information storage memory 58 stores various lens information described later obtained from the lens unit 300 via the connector 122.

  Reference numerals 60, 62, 64, 66, 68, and 70 denote operation means for inputting various operation instructions of the system control circuit 50. A single unit such as a switch, a dial, a touch panel, pointing by line-of-sight detection, a voice recognition device, or the like Consists of multiple combinations.

  Here, a specific description of these operating means will be given.

  Reference numeral 60 denotes a mode dial switch which can be used to switch and set each function shooting mode such as an automatic shooting mode, a program shooting mode, a shutter speed priority shooting mode, an aperture priority shooting mode, a manual shooting mode, and a depth of focus priority (depth) shooting mode. it can. In addition, each function shooting mode such as portrait shooting mode, landscape shooting mode, close-up shooting mode, sports shooting mode, night view shooting mode, panoramic shooting mode, and the like can be switched and set.

  Reference numeral 62 denotes a shutter switch SW1, which is turned on while a shutter button (not shown) is being operated (for example, half-pressed), and instructs to start operations such as AF processing, AE processing, AWB processing, and EF processing.

  A shutter switch SW2 64 is turned on when a shutter button (not shown) is completed (for example, fully pressed), and instructs the start of a series of processes including exposure processing, development processing, and recording processing. First, in the exposure process, a signal read from the image sensor 14 is written to the memory 30 via the A / D converter 16 and the memory control circuit 22, and further, an operation in the image processing circuit 20 and the memory control circuit 22 is used. Development processing is performed. Further, in the recording process, the image data is read from the memory 30, compressed by the compression / decompression circuit 32, and written or transmitted to the recording medium 200 or the PC 210.

  Reference numeral 66 denotes a playback switch, which instructs to start a playback operation for reading an image shot in the shooting mode state from the memory 30, the recording medium 200, or the PC 210 and displaying it on the image display unit 28. In addition, the playback switch 66 can set various function modes such as a playback mode, a multi-screen playback / erase mode, and a PC connection mode.

  68 is a single-shot / continuous-shot switch. When the shutter switch SW2 (64) is pressed, a single-shot mode in which one frame is shot and the camera is in a standby state, and while the shutter switch SW2 (64) is being pressed, continuous shooting is performed. The continuous shooting mode can be set.

  An operation unit 70 includes various buttons and a touch panel. For example, live view start / stop button, video recording start / stop button, zoom switch for changing the magnification of electronic zoom, menu button, set button, multi-screen playback page break button, flash setting button, single shooting / continuous shooting / self Includes a timer switching button, menu movement + (plus) button, menu movement-(minus) button. Further, a playback image movement + (plus button, playback image movement- (minus) button, shooting image quality selection button, exposure correction button, dimming correction button, external flash emission amount setting button, date / time setting button, and the like are also included. The functions of the plus button and the minus button can be selected more easily with numerical values and functions by providing a rotary dial switch.

  In addition, there is an image display ON / OFF switch for setting ON / OFF of the image display unit 28, and a quick review ON / OFF switch for setting a quick review function for automatically reproducing image data taken immediately after shooting. Further, there is a compression mode switch that is a switch for selecting a compression rate for JPEG compression or for selecting a RAW mode in which a signal from an image sensor is directly digitized and recorded on a recording medium. In addition, there is an AF mode setting switch that can set a one-shot AF mode and a servo AF mode. In the one-shot AF mode, an autofocus operation is started when the shutter switch SW1 (62) is pressed, and once focused, the focused state is maintained. In the servo AF mode, the autofocus operation is continued while the shutter switch SW1 (62) is being pressed. Further, as will be described later, it includes a setting switch that can set a dust information acquisition mode for capturing dust information by capturing dust detection images.

  Reference numeral 72 denotes a power switch, which can switch and set the power-on and power-off modes of the camera body 100. Also, the power on and power off settings of various accessory devices such as the lens unit 300, the external flash 112, the recording medium 200, and the PC 210 connected to the camera body 100 can be switched.

  A power control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like. The power supply control unit 80 detects the presence / absence of a battery, the type of battery, and the remaining battery level, controls the DC-DC converter based on the detection result and the instruction of the system control circuit 50, and requires the necessary voltage. It is supplied to each part including the recording medium for a period.

  Reference numerals 82 and 84 denote connectors, and 86 denotes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, NiMH battery, Li-ion battery or Li polymer battery, an AC adapter, or the like.

  Reference numerals 90 and 94 denote recording media such as a memory card and a hard disk, and an interface with a PC. A recording medium attachment / detachment detection circuit 98 detects whether the recording medium 200 or the PC 210 is attached to the connectors 92 and / or 96.

  Although the present embodiment has been described as having two systems of interfaces and connectors for attaching the recording medium, the interface and connectors for attaching the recording medium may have a single or a plurality of systems. Further, a configuration including a combination of interfaces and connectors of different standards may be used.

  As the interface and the connector, it is possible to configure using interfaces that comply with various storage medium standards. For example, a PCMCIA (Personal Computer Memory Card International Association) card, a CF (Compact Flash (registered trademark)) card, an SD card, or the like. When the interfaces 90 and 94 and the connectors 92 and 96 are configured using a standard conforming to a PCMCIA card or a CF card, various communication cards can be connected. Communication cards include LAN cards, modem cards, USB (Universal Serial Bus) cards, and IEEE (Institute of Electrical and Electronic Engineers) 1394 cards. In addition, there are a P1284 card, a SCSI (Small Computer System Interface) card, a PHS, and the like. By connecting these various communication cards, image data and management information attached to the image data can be transferred to and from other computers and peripheral devices such as a printer.

  Reference numeral 104 denotes an optical viewfinder, which guides a light beam incident on the imaging lens 310 through an aperture 312, lens mounts 306 and 106, and mirrors 130 and 132 by a single-lens reflex system, and forms an optical image for display. it can. Thereby, it is possible to perform imaging using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28. Further, in the optical viewfinder 104, some functions of the notification unit 54, for example, in-focus state, camera shake warning, flash charging, shutter speed, aperture value, exposure correction, and the like are displayed.

  Reference numeral 112 denotes an external flash device mounted via the accessory shoe 110.

  Reference numeral 120 denotes an interface for connecting the camera body 100 to the lens unit 300 in the lens mount 106.

  A connector 122 electrically connects the camera body 100 to the lens unit 300. Further, whether or not the lens unit 300 is attached to the lens mount 106 and the connector 122 is detected by a lens attachment / detachment detection unit (not shown). The connector 122 transmits a control signal, a status signal, a data signal, and the like between the camera body 100 and the lens unit 300, and has a function of supplying currents of various voltages.

  Various optical information (aperture, zoom position, pupil position, focal length, etc.) of the lens unit 300 communicated via the connector 122 is stored in the optical information storage memory 58 of the camera body 100. The communication request may be made from the camera side, or may be communicated from the lens side every time information is updated.

  Further, the connector 122 may be configured to perform communication by optical communication and voice communication as well as electrical communication.

  Reference numeral 200 denotes a recording medium such as a memory card or a hard disk. The recording medium 200 includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, and the like, an interface 204 with the camera body 100, and a connector 206 for connecting with the camera body 100.

  As the recording medium 200, a memory card such as a PCMCIA card or compact flash (registered trademark), a hard disk, or the like can be used. Of course, it may be composed of a micro DAT, a magneto-optical disk, an optical disk such as a CD-R or CD-RW, a phase change optical disk such as a DVD.

  Reference numeral 210 denotes a PC, which includes a recording unit 212 composed of a magnetic disk (HD) or the like, an interface 214 with the camera body 100, and a connector 216 for connecting with the camera body 100. Examples of the interface 94 include USB and IEEE1394, but are not particularly limited.

  Next, a process for removing the influence of dust on the optical element 14a such as a low-pass filter and a cover glass arranged in front of the image sensor in the image pickup apparatus having the above configuration by image processing will be described.

  In this embodiment, first, a dust detection image (foreign matter detection image) for obtaining dust information (foreign matter information), which is information such as the position and size of dust (foreign matter), is photographed, and the dust is collected. Extract data and generate garbage data. Here, the dust detection image is preferably an image obtained by photographing a luminance surface that is as uniform as possible. However, since it is desirable that the dust detection image can be easily photographed in a familiar place, strict uniformity is not required. For example, it is assumed that a blue sky or a white wall is photographed.

  FIG. 2 is a flowchart showing processing in the imaging apparatus (digital camera in the present embodiment) when acquiring dust information in the present embodiment.

  First, in step S201, it is determined whether or not the dust information acquisition mode is selected by the operation unit 70. The determination in step S201 is repeated until the dust information acquisition mode is selected. If the dust information acquisition mode is selected, the process proceeds to step S202, and it is determined whether the shutter switch SW1 (62) is turned on. If it is OFF, the process returns to step S201 and the above process is repeated.

  On the other hand, if it is ON, in step S203, the aperture, ISO value, shutter speed, and other shooting-related parameters are set. The parameters set here are shown in FIG. The aperture is set to a narrowed aperture such as F22. The image may be taken with the lens unit 300 connected to the lens mount 106 in the most narrowed range within a settable range. The aperture is reduced in this way because dust does not usually adhere to the surface of the image sensor 14 but to an optical element 14a such as a protective glass for protecting the image sensor 14 or an optical filter disposed on the subject side of the image sensor. This is because the imaging state differs depending on the aperture value of the lens unit 300. Therefore, if the aperture is close to the open value, the dust image will be blurred, and an appropriate image for dust detection cannot be acquired. Therefore, it is preferable to shoot with the aperture as narrow as possible.

  Returning to the description of the flowchart of FIG. 2, by this time, the photographer points the imaging device to a uniform luminance surface such as a white wall as much as possible and operates the shutter switch SW2 (64).

  In step S204, it is determined whether or not the shutter switch SW2 (64) is turned on. If it is OFF, the process returns to step S202 to determine the shutter switch SW1 (62). If it is ON, the process proceeds to step S205. In step S <b> 205, a dust detection image is captured (uniform brightness surface is captured), and the image data is captured in the memory 30. In step S206, dust information is acquired from the image data stored in the memory 30.

  Here, acquisition of dust information will be described. Specifically, the position (coordinates) and size of the dust region are obtained from the photographed dust detection image. First, the captured dust detection image region is divided into a plurality of blocks, the maximum luminance Lmax and average luminance Lave in the block are calculated, and the threshold value T1 in the block is calculated using the following equations.

T1 = Lave × 0.6 + Lmax × 0.4
Next, since the luminance of the pixel to which dust is attached is lower than the luminance of the surrounding pixels, the pixel not exceeding the threshold value T1 is defined as the dust pixel, and each isolated region constituted by the dust pixel is defined as one dust. Let it be a region di (i = 0, 1,..., N).

  FIG. 4 is a diagram showing an outline of dust region size calculation. As shown in FIG. 4, for each dust region, the maximum value Xmax and minimum value Xmin of the horizontal coordinate of the pixels constituting the dust region and the maximum value Ymax and minimum value Ymin of the vertical coordinate are obtained, and the dust region di is obtained. The radius ri representing the size of is calculated by the following equation.

ri = [√ {(Xmax−Xmin) ² + (Ymax−Ymin) ² }] / 2
The center coordinates (Xdi, Ydi) at this time are approximately:
Xdi = (Xmax + Xmin) / 2
Ydi = (Ymax + Ymin) / 2
Sought in The position (coordinates) and radius determined in this way are recorded as a dust information profile.

  In some cases, the data size of the dust correction data (dust information profile) is limited due to limitations due to the size of the nonvolatile memory 56. In order to cope with such a case, the dust position information is sorted according to the size and the average luminance value of the dust region. In this embodiment, sorting is performed in descending order of ri. When ri is equal, the sort is performed in ascending order of the average luminance value. In this way, it is possible to preferentially register noticeable dust in the dust correction data. The sorted dust area is Di, and the radius of the dust area Di is Ri.

  If there is a dust area larger than a predetermined size, it may be excluded from sorting and placed at the end of the sorted dust area list. This is because a large dust region may lower the image quality if interpolation processing is performed later, and is preferably treated as the lowest priority for correction.

  This dust information profile has a structure as shown in FIG. As shown in FIG. 5, the dust information profile stores lens information and dust position and size information when the dust detection image is captured. More specifically, the actual aperture value (F value) at the time of dust detection image capture and the lens pupil position at that time are stored as lens information at the time of dust detection image capture. Subsequently, the number of detected dust areas (integer value) is stored in the storage area, and subsequently, individual specific dust area parameters are repeatedly stored by the number of dust areas. The parameter of the dust area is a set of three numerical values: a dust radius (for example, 2 bytes), a center x coordinate (for example, 2 bytes) in the effective image area, and a similar y coordinate (for example, 2 bytes).

  The acquired dust information is stored in the nonvolatile memory 56 in step S207, and the processing for acquiring dust information is terminated.

  Note that since the shooting operation in the dust information acquisition mode is intended to acquire dust information, in this embodiment, the captured image itself is not compressed and recorded on the recording medium 200. This is to prevent unnecessary consumption of the capacity in the recording medium 200 with image data unnecessary for the photographer, but it may be stored in the recording medium 200 after compression, as in the case of normal images. In addition, you may make some changes such as changing the extension.

  Here, the present embodiment mainly relates to a method of correcting image quality degradation due to dust by image processing when shooting a moving image. Before describing the processing of a moving image, the processing in the case of a still image will be described below. To do.

  In the case of a still image, when normal shooting is performed instead of shooting a dust detection image, the dust correction data (dust information profile) shown in FIG. 5 is associated with the image data together with the camera setting value during normal shooting. To the recording medium 200.

  Specifically, for example, association can be realized by adding dust correction data to the Exif area, which is the header area of an image file in which camera setting values at the time of shooting are recorded. Alternatively, the dust correction data can be independently recorded as a file, and the association can be realized by recording only the link information to the dust correction data file in the image data. However, if the image file and the dust correction data file are recorded separately, the link relationship may be lost when the image file is moved. Therefore, it is desirable to hold the dust correction data integrally with the image data.

  As described above, the dust correction data is recorded in association with the image data because the image data recorded with the dust correction data is transferred to an external image processing apparatus, and the dust removal processing is performed by the external image processing apparatus. This is because it is assumed that (foreign matter removal processing) is performed.

  Next, dust removal processing during normal shooting using dust information stored in the nonvolatile memory 56 as described above will be described with reference to the flowcharts of FIGS. Note that the description here also relates to dust removal processing for still images, but in the case of moving images, dust removal processing is similarly performed by applying dust removal processing similar to that for still images to images for each frame. be able to. This dust removal process is performed using the dust removal circuit 31 in FIG.

  FIG. 7 shows still image shooting processing during normal shooting in the present embodiment.

  In step S501, the process waits until the shutter switch SW1 (62) is turned on. When the shutter switch SW1 (62) is turned on, the process proceeds to step S502 to perform photometry and focus adjustment processing, and then in step S503, it is determined whether or not the shutter switch SW2 (64) is turned on. If the shutter switch SW2 (64) is OFF, the process returns to step S501 and the above processing is repeated. If it is detected that the shutter switch SW2 (64) is turned ON, the process proceeds to step S504 to perform photographing. When shooting is completed, the process proceeds to step S505, where it is determined whether valid dust information exists in the nonvolatile memory 56 or not. If dust information exists, the process proceeds to step S506, and if not, the process proceeds to step S507 to store the captured image data in the recording medium 200.

  In the present embodiment, it is determined whether or not dust information exists in the nonvolatile memory 56, but originally, it is a necessary condition whether or not shooting in the dust information acquisition mode described above is performed. The determination method is not particularly limited. For example, a method may be used in which a certain flag is set at the time of shooting in the dust information acquisition mode and the flag is evaluated.

  In step S506, the acquired dust information is embedded in the header area such as the Exif area for the captured image data, and in step S507, the image data in which the dust information is embedded is stored in the recording medium 200.

  Next, the operation of dust removal processing will be described with reference to FIG.

  In step S601, it is determined whether dust information is embedded in the selected image. If it is embedded, the process advances to step S602 to capture dust information. In step S603, a correction process such as a pixel interpolation process using peripheral pixels of dust is performed in order to remove the influence of dust in the image data from the captured dust information.

  A coordinate row Di (i = 1, 2,... N), a radius row Ri (i = 1, 2,..., N), an aperture value f1 and a lens pupil position L1 are obtained from the extracted dust correction data. Here, Ri is the size of the dust at the coordinate Di previously obtained when sorting the dust correction data. Further, f1 is a lens aperture value when the dust detection image is captured, and L1 is a lens pupil position when the dust detection image is captured. An aperture value f2 and a lens pupil position L2 at the time of shooting a normally shot image are acquired, and Di is converted by the following equation. Here, d is a distance from the image center to the coordinate Di, and H is a distance between the surface of the image sensor 14 and dust. The coordinate Di ′ after conversion and the radius Ri ′ after conversion are defined by the following equation, for example.

Di ′ (x, y) = (L2 × (L1−H) × d / ((L2−H) × L1)) × Di (x, y)
Ri ′ = (Ri × f1 / f2 + 3) (1)
The unit here is a pixel, and “+3” for Ri ′ is a margin amount.

  Dust in the area indicated by the coordinates Di ′ and the radius Ri ′ is detected, and interpolation processing is applied as necessary. Details of the interpolation processing will be described later. If dust removal processing is applied to all coordinates and processing is completed for all coordinates, the process proceeds to step S604.

  In step S604, a corrected image obtained by removing the influence of dust from the captured image is newly recorded.

  This completes the dust removal process.

  In the present embodiment, the camera body 100 has a configuration in which dust information is recorded in a form embedded in captured image data, and a correction process for removing the influence of dust is performed later. In contrast, when an image is captured and recorded by the camera body 100, correction processing is performed to remove the influence of dust without embedding dust information, and the image after correction processing is recorded on the recording medium 200. Also good.

(Interpolation routine)
Next, details of dust region interpolation processing will be described.

  FIG. 9 is a flowchart showing the flow of the interpolation routine.

First, in step S701, dust region determination is performed. The dust area is an area that satisfies all of the following conditions.
(1) Center coordinate Di ′, radius Ri ′ (Di ′, Ri ′ obtained by equation (1))
A region that is darker than the threshold value T2 obtained by the following equation using the average luminance Yave and the maximum luminance Ymax of the pixels included in.

T2 = Yave × 0.6 + Ymax × 0.4
(2) A region not in contact with the circle having the center coordinate Di ′ and the radius Ri ′.
(3) For an isolated region composed of low-luminance pixels selected in (1),
A region where the radius value calculated by the above method is not less than X1 pixels and less than X2 pixels.
(4) A region including the center coordinates Di of the circle.

  In the present embodiment, X1 is 3 pixels and X2 is 30 pixels. In this way, only an isolated small area can be handled as a dust area. If the lens pupil position cannot be obtained accurately, the condition (4) may have a width.

  For example, if the region of interest includes coordinates within a range of ± 3 pixels in the X direction and the Y direction from the coordinate Di, a condition such as determining as a dust region can be considered.

  If there is such an area (part) in the image signal in step S702, the process proceeds to step S703, and dust area interpolation is performed. If there is no such area (part), the process ends. The dust region interpolation processing executed in step S703 is performed by a known missing region interpolation method. Known defect area interpolation methods include, for example, pattern replacement disclosed in JP-A-2001-223894. In Japanese Patent Laid-Open No. 2001-223894, a defective region is specified using infrared light. In this embodiment, the dust region detected in step S701 is treated as a defective region, and the dust region is treated as a normal pixel by pattern replacement. Interpolate with. For pixels that are not filled by pattern replacement, p normal pixels are selected in the order closest to the interpolation target pixel from the image data after pattern interpolation, and interpolation is performed using the average color.

  Still image dust removal processing has the advantage that it is not necessary to be aware of the correspondence between dust correction image data and captured image data by attaching dust correction data to an image in this way. In addition, since the dust correction data is compact data composed of position, size, and conversion data (aperture value, distance information on the lens pupil position), the captured image data size does not become extremely large. Further, by performing interpolation processing only on the region including the pixel specified by the dust correction data, the probability of erroneous detection can be greatly reduced.

  Next, MP4, which is a moving image file format used in recent years for recording moving image data in digital cameras and digital video cameras, will be described.

  MP4 file format (ISO / IEC 14496-14; “Information technology—Coding of audio-visual objects—Part 14: MP4 file format”; ISO / IEC; see 2003-11-24) In order to record video / audio content data such as MPEG standardized by JTC1 / SC29 / WG11 (International Organization for Standardization / International Engineering Consortium) in a file, “ISO Base Media File 49 EC / F14” 12; “Inform tion technology - Coding of audio-visual objects - Part 12: ISO base media file format "; is a reference to 2004-01-23) file format, which has been extended on the basis of the file format of the general purpose of; ISO / IEC. Note that the present invention is not limited to MP4 and can be applied to cases using similar file formats. For example, ISO standard file formats such as “Motion JPEG 2000 file format” (ISO / IEC 15444-3) and “AVC file format” (ISO / IEC 14496-15) are standard file formats having the same basic structure as MP4. Has been enacted.

  FIG. 10 is a conceptual diagram for explaining the data structure in the MP4 file format.

  The MP4 file 1001 includes metadata (header information) 1002 indicating the physical position, temporal position, and characteristic information of the video / audio data, and media data 1003 indicating the actual state of the encoded video / audio data. . In the MP4 format, the presentation of the entire content is called “movie”, and the presentation of the media stream that constitutes the content is called “track”. Typically, the metadata 1002 logically represents the entire moving image data. The video track 1004 and the audio track 1005 that logically handles the entire audio data are included, and the basic configuration contents of the video track 1004 and the audio track 1005 are almost the same. That is, each track records various metadata information of actual media data, and the contents thereof are only slightly different depending on the characteristics of the media data.

  The data included in the video track 1004 includes, for example, so-called decoder configuration information for decoding encoded data and information such as a moving image rectangular size, and in addition, media data is actually recorded. An offset 1006 indicating the position on the file, a sample size 1007 indicating the size of each frame data (also called a picture) of media data, a time stamp 1008 indicating the decoding time of each frame data, and the like are recorded. Yes.

  On the other hand, the media data 1003 includes moving picture data and audio data by means of a data structure called “chunk” in which one or more “samples” indicating the basic unit of encoded data are continuously recorded. It is recorded. This chunk is composed of a video chunk 1009 including moving image media data and an audio chunk 1010 including audio media data in accordance with the track of the metadata 1002.

  The configuration shown in FIG. 10 shows that video chunks 1009 and audio chunks 1010 are recorded alternately, but the recording position and order are not necessarily the same. This example is merely an example of a format that is generally recorded. However, such alternate arrangement (interleaving) has the effect of improving the accessibility of the data recorded in the file by arranging the video and audio data to be played back at approximately the same time at close positions. This is a very common method.

  A chunk contains one or more samples of individual media data. For example, as shown in FIG. 10, video samples (frames) 1011 are continuously recorded in the video chunk 1009. In general, this video sample (frame) 1011 corresponds to one frame data (picture) of video. Each track and chunk is related as follows.

  For example, in the case of moving image data, the information included in the video track 1004 includes information on each video chunk 1009 included in the media data 1003. The offset 1006 is composed of a table of information indicating the relative position of each video chunk 1009 on the file, and by referring to the individual entries in the table, the actual video chunk is recorded at any position. You can know its position. The sample size 1007 describes a plurality of samples included in a plurality of chunks, that is, a size of each video frame as a table. More precisely, information describing the number of samples included in each individual chunk is also described in the video track 1004. From this information, the samples included in each individual video chunk 1009 are accurately determined. It is possible to get to. The time stamp 1008 records the decoding time of each sample in a table as a difference between samples. By referring to this table, the so-called time stamp of each sample can be obtained by calculating the accumulated time. Such a relationship between the track and the chunk is defined so that the audio track 1005 and the audio chunk 1010 are similarly established. As a result, in the MP4 file and the ISO Base Media File Format, it is possible to acquire encoded data from an arbitrary position with additional information such as a time stamp in a necessary unit using the metadata 1002 and the media data 1003. It has become. It should be noted that, for simplicity of explanation, not all the standardized recording information is described here. Details of the standardized definition contents can be known by referring to the corresponding part of ISO / IEC 14496.

  In the MP4 file format, data recorded in the file is described in a data structure called “BOX”, and is recorded in the file in units of BOX. The BOX is composed of the following fields.

    Size: The size of the entire BOX, including the size field itself.

    Type: 4-byte type identifier representing the type of BOX. Usually expressed in 4 alphanumeric characters.

  Since the other fields are optional depending on the BOX, the description is omitted here.

  Data recorded in the file is held in different types of BOX depending on the type. For example, the media data 1003 is a Media Data BOX for storing encoded data (the content of the type field is “mdat”. When an identifier indicating the BOX type is used in the following description, the BOX indicated by the type is expressed. The metadata 1002 is recorded as a Movie BOX ('moov') that stores metadata information of the entire content. Information on the above chunks and samples is also recorded for each track in the moov as a BOX having a unique identifier.

  In addition, the MP4 file format allows not only a form in which all metadata is recorded in moov but also a form in which metadata is divided and recorded in a plurality of areas in time series. This format is called “Fragmented Movie”.

  FIG. 11 shows the structure of a fragment movie file. In the fragment movie format, the media data and metadata of content can be divided in arbitrary time units, and the divided “fragments” are recorded in chronological order from the beginning of the file. For example, in FIG. 11, moov 1101 indicates the metadata of the first fragment, and holds information regarding data included in mdat 1102. The next appearing moof 1103 is the metadata of the second fragment, and is recorded in the same manner so as to hold the information of mdat 1104. When the fragment movie format is adopted, it is necessary to add a Movie Extends Box ('mvex') 1104 indicating that a fragment exists in the moov 1101. Information included in the mvex 1104 includes a duration (time length) of the entire content including all fragments. As described above, in the MP4 file format file, various attributes relating to the media data are stored as metadata areas separately from the media data, so that the desired data can be obtained regardless of how the media data is physically stored. It is possible to easily access the sample data.

  Hereinafter, the moving image file format used for recording moving images and audio data in the present embodiment is the fragment movie format shown in FIG. 11 of the MP4 format, and the dust correction data and video samples (frames) described above at the time of moving image recording A method of associating with 1011 will be described.

  Standards such as the aforementioned “Motion JPEG 2000 file format” (ISO / IEC 15444-3) and “AVC file format” (ISO / IEC 14496-15), and wireless terminals centering on third-generation mobile phones 3GPP file format (Through Generation Partnership Project) (3GPP TS 26.244 “Technical Special Group Transport Services and Systems Specified Spectral Specs Specimen Spect s Spec s Spec s s pe s s s s s s s t s e m s s e s s s s s s s s s s s s s s s s s s t s s s s s s s s s s t s e m e s s s s As s s s e s s s s s s s s s s t s s s s s s s s s s ed e s s t service (PSS); 3GPP file format (3G ) (Release 6) "3rd Generation Partnership Project (see 2003-02-28), etc., and the present invention is also applied to standards that employ a file format and architecture similar to those defined in MP4. Is possible.

  Next, a file recording operation and dust correction processing at the time of moving image shooting in the imaging apparatus having the electronic zoom function according to the present embodiment will be described.

  FIG. 12 is a diagram showing an outline of the operation of electronic zoom processing (zoom magnification of 2) performed during live view and moving image recording in the present embodiment.

  In FIG. 12A, reference numeral 121 denotes a shooting screen corresponding to an image signal read from the image sensor, and a hatched portion 123 included therein is a zoom target area. In the present embodiment, it is assumed that the center portion (center portion) of the image is enlarged. Accordingly, since the center of the image is fixed regardless of the zoom magnification, the zoom target area (coordinates) for the entire screen can be determined only by the zoom magnification. By generating an interpolation signal from the image signal of the hatched portion 123 and inserting this interpolation signal into the image signal, the image is doubled to have the same size as the shooting screen 121 as shown in FIG. Generate an enlarged image. The electronic zoom is realized as described above. Reference numeral 122 denotes a dust area reflected on the shooting screen, and the dust area 122 is doubled by electronic zoom as in the case of a normal subject. A dust region enlarged by the electronic zoom is indicated by 124 in FIG.

  FIG. 13 is a flowchart for explaining the operation at the time of moving image shooting.

  First, in step S1401, it is determined whether or not live view start is selected by the operation unit 70. The determination in step S1401 is repeated until live view start is selected. When live view start is selected, the mirror 130 is mirrored up, the shutter 12 is opened, and an optical image is formed on the image sensor 14.

  An analog signal output from the image sensor 14 at a predetermined frame rate is converted into a digital signal by the A / D converter 16 and subjected to predetermined pixel interpolation processing and color conversion processing by the image processing circuit 20. Stored in the frame memory buffer.

  At this time, various optical information (aperture, zoom position, pupil position, focal length, etc.) is requested from the camera body 100 to the lens unit 300 in accordance with the frame rate. Various optical information sent from the lens unit 300 via the connector 122 is stored in the optical information storage memory 58 in association with each image data in the frame memory buffer.

  The image data stored in the frame memory buffer is read again, converted into image data for display in the image processing circuit 20, and stored in the image display memory 24. The display image data written in the image display memory 24 is displayed by the image display unit 28 via the D / A converter 26. As a result, the image display unit 28 enters a live view display state (electronic viewfinder operation).

  In step S1402, it is determined whether or not the moving image recording start is selected by the operation unit 70. If not selected, the process returns to step S1401 to repeat the above process. On the other hand, if moving image recording start is selected, the process advances to step S1403 to start moving image recording.

  When moving image recording starts, the audio signal processing circuit encodes audio data input from a microphone (not shown), and the encoded audio data is temporarily stored in an audio encoded data buffer in the memory 30. Is done. Since the temporarily stored moving image encoded data and audio encoded data increase with time, they are formed into a predetermined file format and written to the recording medium 200 via the interface 90 as needed.

  In step S1404, it is determined whether stop of moving image recording has been selected by the operation unit 70. If stop of moving image recording is not selected, the process returns to step S1403 to continue moving image recording. If the stop of moving image recording is selected, the process advances to step S1405 to determine whether stop of the live view operation is selected. If the stop of the live view operation is not selected, the process returns to step S1402 and waits for the next moving image recording. If the stop of the live view operation is selected, the moving image shooting routine is ended.

  Next, generation of a moving image file will be described.

  When moving image shooting is started by turning on the moving image recording button of the operation unit 70 in the moving image shooting mode, a new file is first generated, and the first fragment metadata BOX, moov, and media data BOX are used. Create a certain mdat.

  Next, dust position correction data is created. In the dust position correction data, an aperture value, which is lens information of a lens used for moving image shooting, lens pupil position information, and dust correction data shown in FIG. 5 are stored. The created dust position correction data is stored in the memory 52. The dust position correction data stored in the memory 52 is read and written in the metadata moov of the current fragment (foreign object information recording).

(Movie recording routine)
FIG. 14 is a flowchart showing a recording process performed for each frame at the time of moving image shooting in the present embodiment. This process is performed by the system control circuit 50 executing a moving image capturing process program stored in the memory 52.

  First, in step S1501, an image encoding process is performed, and the image data is compressed to a small data amount.

  In step S1502, as a result of the encoding process, compressed image data is recorded in a file (moving image file recording).

  In step S1503, the zoom magnification in the electronic zoom is acquired from the system control circuit 50 and recorded in a file (zoom magnification recording). When the electronic zoom is not operating, 1 is recorded as the zoom magnification. The record to the file is recorded in the metadata moof of the current fragment.

  In addition, since moving image data of a plurality of frames is recorded in one fragment, it is written in a format in which a plurality of pieces of cutout position information are added in the metadata moof of one fragment. Therefore, the information for associating the cut-out position information with the frame is simultaneously written.

  Since a series of processes in steps S1501 to S1503 is a process of recording an image for one frame, the series of processes is repeatedly performed during moving image recording.

(Dust removal processing routine)
Next, processing for removing the influence of dust from moving image data shot by electronic zoom will be described.

  This dust removal process is performed in the same manner as in the case of a still image except that it is performed in units of frames using dust correction data (dust information) recorded on the recording medium 200 in association with image data. The dust correction data is stored in the moov of the moving image file. In this embodiment, since dust correction data is acquired in a state where the electronic zoom is not operating, the position information and size of each dust region included in the dust correction data is based on the angle of view of the entire image sensor. For this reason, when the electronic zoom is operating, the angle of view is different for each zoom magnification, so it is necessary to convert the information into dust information that is adapted to each frame.

  FIG. 15 is a flowchart showing the flow of dust removal processing in the present embodiment.

  The dust removal process may be performed in the camera or may be performed using an image processing apparatus prepared separately.

  In this embodiment, first, in step S1601, the zoom magnification recorded for each frame is acquired from the moving image file. Next, in step S1602, it is determined whether or not the moving image file to be subjected to dust removal processing is an image shot in a state where the electronic zoom is operating. If the zoom magnification acquired in step S1601 is 1, the electronic zoom is regarded as not operating, and the process proceeds to step S1606. If the zoom magnification is greater than 1, the electronic zoom is in operation, and the process advances to step S1603 to convert dust position coordinates.

  A dust position coordinate conversion method will be described with reference to FIG.

  First, the image coordinate system will be described. As shown in FIG. 21, the image coordinate system in the present embodiment has the upper left of the photographing screen 211 as the origin, the positive direction of the x axis is the vertical downward direction (vertical direction), and the positive direction of the y axis is the horizontal right direction (horizontal direction). Direction). Similarly, it is assumed that the same image coordinate system is used for the position information of the dust region included in the dust correction data.

  Next, a method for converting dust position coordinates will be described. If the dust position coordinates before enlargement included in the zoom target area 212 of FIG. 21A are (x, y) and the upper left coordinates of the zoom target area 212 are (x0, y0), the enlargement of FIG. The subsequent coordinates (x ′, y ′) are obtained by the following equation using the zoom magnification M.

(X ′, y ′) = (M × (x−x0), M × (y−y0))
As is apparent from FIG. 21, the upper left coordinate (x0, y0) of the zoom target area 212 is converted to the origin after enlargement.

  Returning to the description of the flowchart of FIG. 15, in the next step S1604, the dust size is converted. From the dust size R before enlargement and the zoom magnification M, the dust size R ′ after enlargement is obtained by the following equation.

R ′ = M × R
Subsequently, in step S1605, it is determined whether or not the enlarged coordinates converted in step S1603 are present inside the shooting screen 211 in FIG. 21A. It is determined that there is no area, and this flowchart is ended. If it is determined in step S1605 that the image exists inside, dust correction processing is performed in step S1606 using the converted dust position coordinates and dust size. The dust correction processing performed here is a series of processing shown in FIG. If it is determined in step S1602 that the image is captured in a state where the electronic zoom is not operating, dust correction processing is performed in step S1606 using the dust position coordinates and dust size stored in the file.

  As described above, in this embodiment, the zoom magnification of the electronic zoom is recorded in a file for each frame during moving image shooting. Further, when performing the dust removal process, the dust removal process is performed after converting the dust position information and the dust size using the zoom magnification recorded for each frame.

  This makes it possible to perform dust removal processing on a moving image that has been operated with the electronic zoom during moving image recording.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. Since the main configuration of the imaging apparatus of the second embodiment is basically the same as that of the first embodiment, description of the same components is omitted, and only different components are scratched and described.

  FIG. 22 is a diagram illustrating an outline of the operation of electronic zoom (double zoom magnification) when the zoom target area is other than the center of the image in the imaging apparatus according to the second embodiment. In the present embodiment, it is assumed that the zoom target area is not limited to the center of the image, and an arbitrary area within the shooting screen 221 in FIG. 22A can be selected. Therefore, in order to determine the zoom target area, the zoom magnification and at least one coordinate (for example, the upper left) of the vertex of the zoom target area are required.

(Movie recording routine)
FIG. 23 is a flowchart showing a recording process performed for each frame at the time of moving image shooting in the present embodiment. This process is performed by the system control circuit 50 executing a moving image capturing process program stored in the memory 52.

  First, in step S2301, an image encoding process is performed, and the image data is compressed to a small data amount.

  In step S2302, the compressed image data is recorded in the file as a result of the encoding process.

  In step S2303, the zoom magnification in the electronic zoom is acquired from the system control circuit 50 and recorded in a file. When the electronic zoom is not operating, 1 is recorded as the zoom magnification. In step S2304, an image cutout position is acquired from the system control circuit 50 and recorded (cutout position recording). In the present embodiment, the upper left coordinates of the zoom target area 222 in FIG. 22A are recorded in a file.

(Dust removal processing routine)
Next, processing for removing the influence of dust from moving image data shot by electronic zoom will be described.

  FIG. 24 is a flowchart showing the flow of dust removal processing in the present embodiment.

  In this embodiment, first, in step S2401, the zoom magnification recorded for each frame is acquired from the moving image file. Next, in step S2402, it is determined whether or not the moving image file to be subjected to dust removal processing is an image shot in a state where the electronic zoom is operating. If the zoom magnification acquired in step S2401 is 1, the electronic zoom is regarded as not operating, and the process proceeds to step S2407. If the zoom magnification is larger than 1, it is determined that the electronic zoom is operating, and the process proceeds to step S2403. In step S2403, an image cutout position is acquired from the moving image file. In this embodiment, the upper left coordinate of the zoom target area is acquired. In step S2404, dust position coordinates are converted. The dust position coordinate conversion method is the same as that in the first embodiment except that the upper left coordinate of the zoom target area is (x0, y0) and the value acquired from the moving image file in step S2403 is used. Also, the processing after step S2405 is the same as the processing after step S1604 of the first embodiment, and the description thereof will be omitted.

  As described above, in this embodiment, the zoom magnification and image cutout position of the electronic zoom are recorded in a file for each frame during moving image shooting. Further, when performing the dust removal process, the dust removal process is performed after converting the dust position information and the dust size using the zoom magnification and the image cutout position recorded for each frame.

  This makes it possible to perform dust removal processing on a moving image that has been operated with an electronic zoom for an arbitrary area during moving image recording.

(Other embodiments)
The object of each embodiment is also achieved by the following method. That is, a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but the present invention includes the following cases. That is, an operating system (OS) running on a computer performs part or all of actual processing based on an instruction of a program code, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, the following cases are also included in the present invention. That is, the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion card or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the above storage medium, the storage medium stores program codes corresponding to the procedure described above.

1 is a block diagram illustrating a configuration of an imaging apparatus having an image processing function according to a first embodiment of the present invention. It is a flowchart which shows the process in the imaging device (this embodiment digital camera) at the time of acquiring the dust information in the 1st Embodiment of this invention. It is a figure which shows the setting parameter list at the time of the dust information acquisition in the 1st Embodiment of this invention. It is a figure which shows the outline | summary of dust area size calculation. It is a figure which shows the structure of a dust information profile. It is a figure for demonstrating the outline | summary of operation | movement of an electronic zoom. 6 is a flowchart illustrating still image shooting processing during normal shooting according to the first embodiment of the present invention. It is a flowchart which shows operation | movement of a dust removal process. It is a flowchart which shows the flow of an interpolation routine. It is a figure for demonstrating the data structure in MP4 file format. It is a figure for demonstrating the file structure of a fragment movie format. It is a figure for demonstrating the outline | summary of the operation | movement of the electronic zoom in 1st Embodiment. It is a flowchart for demonstrating the operation | movement at the time of video recording. It is a figure explaining the flow of the moving image recording routine in 1st Embodiment. It is a figure explaining the flow of the dust removal process routine in 1st Embodiment. H. 1 is a diagram illustrating a configuration of an image processing apparatus that compresses image data using an H.264 system. It is a figure which shows the example of the reference list at the time of encoding the picture P21. It is a figure which shows the example of the reference list at the time of encoding the picture P24. It is a figure which shows the mode of a change of a reference list for every picture. It is a figure which shows the mode of a change of the reference list in case B picture is added to a reference list. It is a figure explaining the outline | summary of the conversion method of a dust position coordinate. It is a figure for demonstrating the outline | summary of the operation | movement of the electronic zoom in 2nd Embodiment. It is a figure explaining the flow of the moving image recording routine in 2nd Embodiment. It is a figure explaining the flow of the dust removal process routine in 2nd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 12 Shutter 14 Image pick-up element 16 A / D converter 18 Timing generation circuit 20 Image processing circuit 22 Memory control circuit 24 Image display memory 26 D / A converter 28 Image display part 30 Memory 31 Dust removal circuit 32 Image compression / decompression circuit 33 Audio signal processing circuit 40 Shutter control unit 42 Focus adjustment unit 46 Photometry control unit 48 Flash 50 System control circuit 52 Memory 54 Notification unit 56 Non-volatile memory 58 Optical information storage memory 60 Mode dial switch 62 Shutter switch SW1
64 Shutter switch SW2
66 Playback Switch 68 Single / Continuous Shooting Switch 70 Operation Unit 72 Power Switch 80 Power Control Unit 82 Connector 84 Connector 86 Power Supply 90 Interface 92 Connector 94 Interface 96 Connector 98 Recording Medium Attachment / Disconnection Detection Circuit 100 Camera Body 104 Optical Finder 106 Lens Mount 110 Accessory shoe 112 External flash 120 Interface 122 Connector 130 Mirror 132 Mirror 200 Recording medium 202 Recording unit 204 Interface 206 Connector 210 PC
212 Recording Unit 214 Interface 216 Connector 300 Lens Unit 306 Lens Mount 310 Shooting Lens 312 Aperture 320 Interface 322 Connector 340 Aperture Control Unit 342 Focus Control Unit 344 Zoom Control Unit 350 Lens System Control Circuit

Claims (10)

An image pickup unit that includes an image pickup device that photoelectrically converts a subject image, and that generates an image signal based on an output signal from the image pickup device;
Electronic zoom processing means for performing electronic zoom processing for cutting out and enlarging a part of the image signal;
Foreign matter detection means for detecting foreign matter information, which is information containing at least information on the position and size of the foreign matter, from a foreign matter detection image containing a foreign matter image attached to the surface of the optical element disposed in front of the image sensor; ,
Foreign matter information recording means for recording the foreign matter information in an image file;
Zoom magnification recording means for recording the zoom magnification of the electronic zoom in the electronic zoom processing means in the image file;
Using the foreign matter information and the zoom magnification of the electronic zoom, the position and size of the foreign matter are converted into the position and size of the foreign matter in the image after the electronic zoom processing, and the image in the image after the electronic zoom processing is used. Foreign matter removing means for removing the image of the foreign matter by image processing;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein the image file is a moving image file.   The imaging apparatus according to claim 1, wherein a cutout position of a part of the image signal in the electronic zoom processing unit is a position corresponding to a central portion of the imaging element.   The imaging apparatus according to claim 1, further comprising a cut-out position recording unit that records a cut-out position of a part of the image signal in the electronic zoom processing unit in the image file.   The imaging apparatus according to claim 4, wherein the cutout position is a vertical coordinate and a horizontal coordinate of a part of the image signal with respect to the entire screen of the imaging device. A method for controlling an imaging device including an imaging device that photoelectrically converts a subject image and having an imaging means that generates an image signal based on an output signal from the imaging device,
An electronic zoom processing step of performing an electronic zoom process of cutting out and enlarging a part of the image signal;
A foreign matter detection step of detecting foreign matter information, which is information including at least the position and size information of the foreign matter, from a foreign matter detection image containing a foreign matter image attached to the surface of the optical element disposed in front of the imaging device; ,
A foreign matter information recording step for recording the foreign matter information in a moving image file;
A zoom magnification recording step of recording the zoom magnification of the electronic zoom in the electronic zoom processing step in the moving image file;
Using the foreign matter information and the zoom magnification of the electronic zoom, the position and size of the foreign matter are converted into the position and size of the foreign matter in the image after the electronic zoom processing, and the image in the image after the electronic zoom processing is used. A foreign matter removing step of removing an image of the foreign matter by image processing;
An image pickup apparatus control method comprising: A program for causing a computer to execute the control method according to claim 6 . Processes an image file in which foreign object information, which includes information on the position and size of a foreign object attached to the surface of an optical element arranged in front of the image sensor at the time of shooting, and the zoom magnification of the electronic zoom at the time of shooting is recorded An image processing apparatus,
Using the foreign object information and the zoom magnification of the electronic zoom, the position and size of the foreign object are converted into the position and size of the foreign object in the image after the electronic zoom process , and the converted position and size of the foreign object are converted. An image processing apparatus comprising: a foreign matter removing unit that removes a foreign matter image in the image after the electronic zoom processing based on the image processing. Processes an image file in which foreign object information, which includes information on the position and size of a foreign object attached to the surface of an optical element arranged in front of the image sensor at the time of shooting, and the zoom magnification of the electronic zoom at the time of shooting is recorded An image processing method comprising:
Using the foreign object information and the zoom magnification of the electronic zoom, the position and size of the foreign object are converted into the position and size of the foreign object in the image after the electronic zoom process , and the converted position and size of the foreign object are converted. And a foreign matter removing step of removing the foreign matter image in the image after the electronic zoom processing based on the image processing. A program for causing a computer to execute the image processing method according to claim 9.

Images