JP2018026743A - Image processor, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor storing a piece of RAW moving image data capable of storing a piece of information required for removing various kinds of noises while suppressing data capacity.SOLUTION: A digital still camera records a piece of RAW moving image data including plural RAW image frames in a memory 102. The digital still camera acquires information of photography conditions including, for example, sensitivity of an image sensor, an exposure time, and the temperature of the image sensor. The digital still camera determines whether to include an OB area or shading image in a predetermined RAW image frame depending on the photography condition.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, a control method, and a program.

  In recent years, various editing techniques for moving images have been proposed due to the popularization of digital video cameras (hereinafter referred to as cameras) and the improvement of the performance of personal computers. From this background, it is expected that RAW format data will be handled not only for still images but also for moving images, and cameras that can store files in RAW format will generally become popular.

  Patent Document 1 discloses an image processing apparatus that records development parameters for a specific RAW image frame in accordance with a user instruction when RAW moving image data is recorded. Further, Patent Document 2 captures a light-shielded image captured under the same photographing conditions as the processing target image at a desired time after the end of capturing a still image as a processing target image, and uses the light-shielded image to process the target image. An image processing system that performs noise cancellation processing of an image is disclosed.

JP 2011-244423 A JP 2007-88778 A

  RAW video data, unlike video data compressed by a predetermined method, has a very large amount of data, which may reduce the frame rate and increase the size of the device by squeezing the capacity and bandwidth of the internal memory and recording device. There is. Conventionally, in an image processing apparatus capable of recording such RAW moving image data, a video signal area is cut out and recorded without recording all areas read from the image sensor as RAW image data. In this method, the resolution and the frame rate of the RAW moving image data can be increased. On the other hand, the periphery (outside) of the cut-out area is cut, so that the extra pixels cannot be used as the light-blocking area.

  The light shielding area outside the video signal can be used for acquiring a noise signal such as a dark current component included in the video signal. This light shielding area is called an OB area (optical black area). By analyzing a random noise component included in the OB region, advanced noise removal (noise reduction, NR) can be performed. Since the OB area is cut, it becomes impossible to perform NR processing according to the noise component when developing RAW image data.

  In addition, when the exposure time is extended or in a high temperature environment, fixed pattern noise may occur in the video signal. The fixed pattern noise included in the video signal can detect a noise component by taking a light-shielded image. Noise is removed by subtracting the noise component of fixed pattern noise from the video signal. In still image shooting, the fixed pattern and noise can be removed by shooting the shading image after the imaging process is completed, but in moving image shooting, the image is lost by shooting the shading image. .

  As described above, when RAW moving image data is recorded, if an OB area or a light-shielded image is not recorded, NR processing according to the noise amount and the noise pattern cannot be applied when developing the RAW moving image data. End up.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus for recording RAW moving image data, which can record information necessary for removing various noises while suppressing the data capacity.

  An image processing apparatus according to an embodiment of the present invention includes a recording unit that records RAW moving image data including a plurality of RAW image frames, an acquisition unit that acquires shooting conditions, and information that is used for noise removal according to the shooting conditions. Determining means for determining whether or not to be included in a predetermined RAW image frame.

  According to the image processing apparatus of the present invention, in the image processing apparatus that records RAW moving image data, it is possible to record information necessary for removing various noises while suppressing the data capacity.

It is a figure which shows the structural example of the image processing apparatus which concerns on one Embodiment of this invention. It is a figure which shows the characteristic of the frame classification of RAW moving image data. It is a figure which shows the process which judges the presence or absence of recording of an OB area | region and a light-shielding image. It is a figure which shows the video area and OB area | region in a RAW image frame. It is a figure explaining the recording format of RAW moving image data.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram illustrating a configuration example of an image processing apparatus according to an embodiment of the present invention. In this embodiment, an example in which the image processing apparatus is applied to a digital still camera will be described, but the present invention is not limited to this. For example, the present invention can be applied to an imaging apparatus such as a digital movie camera, an information processing apparatus (for example, a personal computer or a portable terminal), and the like.

  In FIG. 1, an image sensor 101 is a solid-state image sensor such as a CCD image sensor or a CMOS sensor, and includes a primary color filter having a Bayer arrangement, for example, and can capture a color image. The strobe light 116 is a strobe light emitting device, and can be controlled to emit light in association with exposure by the image sensor 101 by adjusting the light emission amount by controlling it with an arbitrary light emission time.

  The CPU 114 calculates the shutter speed and the aperture value based on the brightness of the entire image, performs focus control using a predetermined focus area, and calculates the drive amount of the focus lens. The exposure value (shutter speed, aperture value) calculated by the CPU 114, the driving amount of the focus lens, and the flash emission amount are output to the control circuit 113. The control circuit 113, based on these values, extracts the aperture, shutter, and focus lens. Drive etc.

  The image processing circuit 115 can perform special image processing on the video signal (image data) stored in the memory 102. The image processing circuit 115 can be used to perform image processing such as face detection and subject area division based on color and luminance signals. The white balance (WB) control circuit calculates a WB correction value based on the video signal stored in the memory 102 and performs WB correction on the video signal stored in the memory 102 using the calculated WB correction value. Do.

  The color conversion matrix (MTX) circuit 104 converts the video signal corrected by the WB control circuit 103 into color difference signals RY and BY by applying a color gain so that the video signal is reproduced with an optimum color. A low-pass filter (LPF) circuit 105 limits the bands of the color difference signals RY and BY. A CSUP (Chroma Suppress) circuit 106 suppresses a false color signal in a saturated portion of the video signal band-limited by the LPF circuit 105. On the other hand, the video signal WB corrected by the WB control circuit 103 is also output to the luminance signal (Y) generation circuit 111 to generate the luminance signal Y.

  The edge enhancement circuit 112 performs edge enhancement processing on the generated luminance signal Y. The color difference signals RY and BY output from the CSUP circuit 106 and the luminance signal Y output from the edge enhancement circuit 112 are converted into RGB signals by the RGB conversion circuit 107, and a gamma (γ) correction circuit. At 108, gradation correction is performed. The RGB signal subjected to gradation correction is converted into a YUV signal by the color / luminance conversion circuit 109, further compressed by the JPEG compression circuit 110, and recorded as a video signal on an external recording medium or an internal recording medium.

  Next, the frame type of RAW moving image data will be described. In this specification, moving image data acquired or recorded in the RAW format is described as RAW moving image data. The frame image data constituting the RAW moving image data is described as RAW image data or RAW image frame.

  FIG. 2 is a diagram for explaining the characteristics of the frame type of RAW moving image data. Specifically, the RAW image frames are classified into four types depending on whether or not an OB area (optical black area) is recorded and whether or not dark photographing for acquiring a light-shielded image is performed.

  Type (Type) A indicates a normal RAW image frame in which only a video signal is recorded. Type B indicates a RAW image frame in which an OB area is recorded. The recording method of the OB area will be described later with reference to FIG. Since the OB area is recorded in addition to the video area, the recording area is expanded and the data capacity is increased as compared with the type A RAW image frame. Type C indicates a RAW image frame for recording a light-shielded image (dark photographed image). Type D indicates a RAW image frame in which an OB area and a light-shielded image are recorded.

  FIG. 3 is a diagram illustrating a flowchart for determining whether or not the OB area is recorded and whether or not the light-shielded image is recorded. In step S201, the CPU 114 acquires camera information. Specifically, information such as imaging conditions, such as sensitivity of the image sensor, exposure time, and temperature of the image sensor, is acquired as camera information. In this embodiment, information on the sensitivity, temperature, and exposure time of the image sensor is acquired. However, the present invention is not limited to this, and information on imaging conditions in which the noise amount or noise pattern changes is acquired.

  In step S202, the CPU 114 determines whether or not the change in exposure time is large in the shooting condition information acquired in step S201. If the change in exposure time is larger than the predetermined change amount, the process proceeds to step S204, and if the change in exposure time is equal to or less than the predetermined change amount, the process proceeds to step S203. In step S203, the CPU 114 determines whether or not the temperature change of the image sensor acquired in step S201 is large. When the temperature change of the image sensor is larger than the predetermined change amount, the process proceeds to step S204, and when the temperature change of the image sensor is equal to or less than the predetermined change amount, the process proceeds to step S205.

  In step S204, the CPU 114 determines that the type of the RAW image frame is type D, and records the OB area and the light-shielded image. That is, when the change in exposure time is large or the temperature change of the image sensor is large, the OB area and the light-shielded image are recorded in the RAW image frame. In step S205, the CPU 114 determines whether there is a change in the sensitivity (ISO sensitivity) of the image sensor acquired in step S201.

  If the sensitivity of the image sensor has not changed, the process proceeds to step S207. If the sensitivity of the image sensor has changed, the process proceeds to step S206. As the sensitivity of the image sensor, the maximum sensitivity (peak sensor sensitivity) and average sensitivity (average sensor sensitivity) from the start to the end of shooting of a moving image are stored. In step S206, the CPU 114 determines whether or not the change in the sensitivity of the image sensor is 1/3 or more. If the change in sensitivity of the image sensor is 1/3 step or more, the process proceeds to step S208. If the change in sensitivity of the image sensor is less than 1/3 step, the process proceeds to step S207.

  In step S208, the CPU 114 determines that the type of the RAW image frame is type B, and records the OB area. In step S207, the CPU 114 determines that the type of the RAW image frame is type A. That is, the OB area and the light-shielded image are recorded as information used for noise removal when there is a large change in shooting conditions such as a change in exposure time, a change in temperature of the image sensor, and a sensitivity of the image sensor. In this case, the OB area and the light-shielded image are not recorded.

  In step S209, the CPU 114 determines whether the moving image recording starts or ends. When moving picture recording starts or ends, the process proceeds to step S210. In step S210, the CPU 114 determines that the type of the RAW image frame is type D, and records the OB area and the light-shielded image. In the process of FIG. 2, the type C for recording only the light-shielded image is not determined. However, for example, the type C may be determined in step S204.

  FIG. 4 is a diagram showing a video area and an OB area in a RAW image frame. An area 301 represents a video area and is a video portion displayed as a moving image. A region 302, a region 304, and a region 306 indicate OB regions and are regions used for analyzing the dark current component of the image sensor. An area 303 shows an example in which fixed pattern noise has occurred, and shows a state in which horizontal stripes and unevenness occur in the video area. In the area 305, the fixed pattern noise in the area 303 is extracted. Fixed pattern noise can be extracted by performing dark photography with light shielding.

  FIG. 5 is a diagram for explaining a recording format of RAW moving image data. The RAW moving image data 501 shows an example of recording a RAW image frame with an OB area. The RAW moving image data includes a plurality of RAW image frames, and additional information of INVALID (D) is recorded in the first frame, indicating that the first frame is an invalid frame, and the type of the RAW image frame is a type. D. The type “D” indicates that the frame is a RAW image frame in which the OB area and the light-shielded image are recorded, and the hatched area is the OB area.

  Subsequently, image frames 001 (A), 003 (A), and 004 (A) are type A RAW image frames in which only the video area is recorded. The image frame 002 (B) is a type B RAW image frame that records an OB area in addition to a video area. In the final frame, incidental information of INVALID (D) is recorded, indicating that it is an invalid frame, as with the first frame, and that the type of the RAW image frame is type D.

  As described above, in the RAW moving image data 501, the OB area and the light-shielded image are embedded in a portion not used for moving image display as an invalid frame at the start and end of moving image recording. Thereby, it is possible to record the light-shielded image without impairing the continuity of the main image. The light-shielded image having INVALID (D) in the final frame is subjected to dark shooting after switching to either the highest sensitivity or the average sensitivity during moving image recording stored in step S205 in FIG. To do.

  Next, RAW moving image data 502 and 503 show an example in which an OB area and a light-shielded image are recorded in a sub-image container (sub-image area). Reference numeral 502 denotes a main image container (main image area) for recording a video area displayed as a moving image. Reference numeral 503 denotes a sub-image container for recording an image such as an OB area, a distance map, and a sub camera video. In the image frame 002 (B) of the RAW moving image data 502 and 503, the type of the RAW image frame is type B, the video area is recorded in the main image container, and the OB area is recorded in the sub image container.

  In the image frame 005 (C), the type of the RAW image frame is type C, and the light-shielded image is recorded in the sub-image container. At this time, the video of 004 (A) which is the previous frame is displayed twice in the main image container. Note that it is also possible to generate an intermediate image from the preceding and following frames using a frame interpolation technique, instead of displaying the 004 (A) video twice in the 005 (C) main image container. Like 501 to 503 described above, the RAW image frame type is recorded in association with the RAW image frame. That is, whether or not information used for noise removal such as an OB area and a light-shielded image is included in the RAW image frame and the type of the information are recorded in association with each other.

  As described above, according to the present embodiment, in an image processing apparatus that records RAW moving image data, an OB area, a light-shielded image, or the like is added to a predetermined RAW image frame in a shooting condition in which a noise amount or a noise pattern changes. Adds information used for noise removal. For this reason, it is possible to record information necessary for removing various noises while suppressing the data capacity, and at the time of developing RAW moving image data, processing necessary for removing various noises such as random noise and fixed pattern noise. It is possible to apply.

  In the present embodiment, a method using a light-shielded image as a fixed pattern / noise detection method has been described. However, the present invention is not limited to this. For example, in the case of fixed pattern noise such as horizontal stripes generated in the horizontal direction, it is possible to analyze using the OB region.

  In addition, when the type of the RAW image frame is determined as type B, type C, or type D in FIG. 3, an OB area or a light-shielded image is acquired immediately (immediately after the determination), or a predetermined number of RAW images are obtained. Whether to obtain the image frame after outputting it is arbitrary. Further, the OB area or the light-shielded image may be acquired after a predetermined time has elapsed. Temporary changes can be canceled by providing a delay without immediately responding at the timing of switching the exposure time. At this time, it is also possible to acquire the movement information of the subject or the camera body, and control so as not to acquire the OB area or the light-shielded image when the movement is determined.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

101 Image sensor 102 Memory 113 Control circuit 114 CPU
115 Image processing circuit

Claims (14)

Recording means for recording RAW moving image data including a plurality of RAW image frames;
Acquisition means for acquiring shooting conditions;
An image processing apparatus comprising: a determination unit configured to determine whether or not information used for noise removal is included in a predetermined RAW image frame according to the shooting condition.   The image processing apparatus according to claim 1, wherein the information used for noise removal is an optical black area.   When the determination means determines that the optical black area is to be included in the predetermined RAW image frame, the recording means expands the area for recording the video signal of the predetermined RAW image frame to increase the optical black area. 3. The image processing according to claim 2, wherein an area is recorded, or the video signal is recorded in a main image area of the predetermined RAW image frame, and the optical black area is recorded in a sub-image area. apparatus.   The image processing apparatus according to claim 1, wherein the information used for noise removal is a light-shielded image.   When the determination unit determines to include the light-shielded image in the predetermined RAW image frame, the recording unit records the light-shielded image in an area for recording a video signal of the predetermined RAW image frame, or The image processing apparatus according to claim 4, wherein the shaded image is recorded in a sub-image area of the predetermined RAW image frame.   The recording means records the video signal of the RAW image frame before the predetermined RAW image frame in the main image area when the light-shielded image is recorded in the sub-image area of the predetermined RAW image frame. The image processing apparatus according to claim 5.   The determination unit includes information used for noise removal in the predetermined RAW image frame in a shooting condition in which a noise amount or a noise pattern changes. An image processing apparatus according to 1.   The determination means includes an optical black region and a light-shielded image in the predetermined RAW image frame when the change in exposure time or sensor temperature is larger than a predetermined change amount, respectively, and changes in the exposure time and sensor temperature. The optical black region is included in the predetermined RAW image frame when the amount is less than or equal to the predetermined amount of change and the change in sensor sensitivity is greater than the predetermined amount of change. 8. The image processing apparatus according to 7.   The predetermined RAW image frame is a RAW image frame that is captured immediately after the determination unit determines that it is a shooting condition in which the noise amount or noise pattern changes, or after a predetermined number of RAW image frames are captured. The image processing apparatus according to claim 7, wherein the image processing apparatus is provided.   10. The image processing apparatus according to claim 1, wherein the recording unit includes information used for noise removal in a head or final RAW image frame of the RAW moving image data. 11.   The recording means includes a peak sensor sensitivity or an average sensor sensitivity between the start of shooting of the RAW moving image data and the end of shooting when the information used for noise removal is included in the final RAW image frame of the RAW moving image data. The image processing apparatus according to claim 10, wherein information used for noise removal is generated.   The recording unit records the RAW image frame in association with the RAW moving image data in association with whether or not the information used for noise removal is included in the RAW image frame and the type of the information. The image processing apparatus according to claim 1. A control method of an image processing apparatus for recording RAW moving image data including a plurality of RAW image frames,
An acquisition process for acquiring imaging conditions;
A determination step of determining whether or not information used for noise removal is included in a predetermined RAW image frame according to the shooting condition;
A recording step of recording the RAW image frame according to the determination in the determination step;
A control method characterized by comprising:   The program for functioning a computer as each means with which the image processing apparatus of any one of Claims 1 thru | or 12 is provided.