JP2012178713A - Image processing system, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system generating a still image of high image quality with less noise deterioration from a moving image encoded and recorded by image information compression using a correlation between frames and to provide an image processing method and a program.SOLUTION: An imaging apparatus 1 includes: an image processing section 126 functioning as a decoding section decoding compressed data which is compression-processed; a frame selecting section which selects two or more frame images as a selected frame image from a frame image string outputted from the image processing section 126, on the basis of an instruction from outside or automatically; and a filter processing section generating one frame image by filtering the selected frame image which the frame selecting section selects. The filter processing section performs a filter processing using a different weighting coefficient on the basis of a size of noise caused in a compression processing.

Description

  The present invention relates to an image processing device, an image processing method, and a program for acquiring a still image from a moving image, and in particular, an image processing device, an image processing method, and a program for acquiring a still image from a plurality of frame images. Regarding the program.

  In recent digital video cameras, a plurality of time-sequential frame image data (or field image data) input through an imaging system is converted into MPEG (Moving Picture Experts Group) -2 or H.264. H.264 / AVC or the like is used for encoding image information compression processing using correlation between frames, and recording is performed on a recording medium.

  MPEG-2 moving image data is compensated for motion between an I picture (Intra Picture) obtained by intra-frame compression using correlation only within a frame and a temporally previous picture. Perform motion-compensated inter-frame prediction between P pictures (Predictive coded: inter-frame forward-predictive coded images) that have been compressed using inter-frame correlation and inter-frame prediction, and multiple pictures. And a B picture (Bi-directionally predictive coded) that is compressed using correlation between frames. Further, in H.264 / AVC, the B picture is a bi-predictive picture, and in the case of MPEG-2, the previous and subsequent pictures are referred to, whereas H.264 / AVC. In the case of, the degree of freedom is increased, such as before and after, before and after, and after and after.

  FIG. 5 is a diagram showing a frame structure in MPEG-2. As shown in FIG. 5, each of a plurality of frames that are continuous in time series is treated as an I, P, or B picture. The arrows in the figure indicate the direction of motion compensation interframe prediction, and a P picture is predicted from a temporally preceding P picture or I picture. A B picture is predicted between motion compensated frames from a P picture or an I picture before and behind in time. In general, an I picture is quantized in the finest quantization step, a P picture is coarser, and a B picture is often quantized in a coarser quantization step.

  H. In H.264 / AVC, in addition to the structure of I picture, P picture, and B picture, one frame image is divided into a plurality of regions (called slices), and I slice, P slice, B slice can do. Also in this case, in general, the I slice is quantized by the finest quantization step, the P slice is coarser, and the B slice is often quantized by the coarser quantization step.

  Note that the quantization step has different values for each pixel block called a macroblock. Accordingly, macroblocks quantized in a relatively fine quantization step and macroblocks quantized in a relatively rough quantization step are mixed in one picture.

  By the way, video camera recorders that extract a part of signals and digital data recorded for moving images and generate a still image have been commercialized. In such an apparatus, it is conceivable to select one frame image selected by the user while reproducing the recorded moving image data and extract it as a still image.

On the other hand, when acquiring a still image from a moving image, a device for extracting a plurality of frame images and obtaining a still image from the plurality of frame images has been conventionally proposed. For example, in Patent Document 1, when a pause instruction operation input from a user is detected while a moving image is displayed, at least two or more still images of a plurality of still images are combined instead of the moving image. An imaging apparatus having a pause display control means for displaying a pause display composite image on a display monitor is disclosed.
This is because a clear image with no image flow is preferred for normal still image viewing, but when viewing a still image by pausing during playback of a moving image, an image with an image flow is more likely to move. Since it may be preferable because a flow occurs, it is intended to acquire such a still image.

  Further, in Patent Document 1, when the number of frames to be added at the time of generating a composite image used for moving image reproduction is set to “3”, the center frame is set as a reference image, and the previous image in the time axis direction (past ) And after (future) one frame at a time are set as interpolation images, and the addition ratio of the images is set to “0.25: 0.5: 0.25”. In the imaging apparatus described in Patent Document 1, the number of additions and the addition ratio when generating a composite image are controlled according to the magnitude of a motion vector (in other words, the amount of motion of a subject). As a result, when an image file is reproduced as a moving image, a composite image suitable for moving image reproduction can be easily generated by referring to the data of the number of added frames and the addition ratio.

JP 2010-045619 A JP 2008-003674 A

Supervised by Satoshi Okubo, “Revised H.264 / AVC Textbook (Impress Standard Textbook Series)”, Impress (2006)

  By the way, when one frame image is taken out from a moving image as a still image, there is a problem that a noise component due to the image sensor is more conspicuous than a moving image. This is because in the moving image, human vision in the time axis direction is insensitive, that is, since the moving image is interpolated between frames, the noise component in the moving image is less visible. Conceivable. In particular, when the shooting environment is dark, the noise component of each frame image increases. This is because the amount of light incident on each cell of the image sensor is small, so that the output of photoelectric conversion is small and buried in noise.

  Further, when the frame image selected by the user is a B picture, since the quantization is performed in a coarser quantization step than other pictures, deterioration due to a noise component due to the quantization is conspicuous.

  H. In H.264 / AVC, deblocking filter processing may be performed. In this method, a smoothing filter is applied to block boundaries in order to mitigate noise called block distortion. While this deblocking filter alleviates block distortion, it may cause a harmful effect of attenuating the high frequency components of the original image data. The strength of the deblocking filter is variable in units of slices, and whether or not the filtering process is performed on the pixels at the individual block boundaries is determined by the attributes of the blocks including the peripheral blocks.

  Although different from the purpose of Patent Document 1, it is possible to cancel out noise components included in the above-described image by combining a plurality of still images as in the imaging device described in Patent Document 1. However, noise components due to quantization and block distortion due to a deblocking filter cannot be removed by simply synthesizing a plurality of frame images.

  The present invention has been made to solve such problems, and from a moving image encoded and recorded by image information compression using correlation between frames, a high-quality still image with little noise deterioration is provided. An object of the present invention is to provide an image processing apparatus, an image processing method, and a program.

  An image processing apparatus according to the present invention includes a decoding unit that decodes compressed compressed data, and two or more frames from a frame image sequence output from the decoding unit based on an external instruction or automatically. A frame selecting unit that selects an image as a selected frame image; and a filter processing unit that generates one frame image by filtering the selected frame image selected by the frame selecting unit, the filter processing unit including: Filtering processing using different weighting factors is performed based on the magnitude of noise generated by the compression processing.

  In the present invention, when a single still image is generated by selecting a plurality of frames from a moving image, for example, based on the magnitude of noise generated by compression processing such as quantization noise and attenuation noise by a deblocking filter. By performing the filter processing using different weighting factors, noise at the time of compression can be reduced together with noise based on the image sensor.

  The decoding unit decodes compressed data that has been compressed using different quantization steps, and the filter processing unit is different based on a quantization step at the time of compression of the selected frame image. Filtering using a weighting factor can be performed. Thereby, quantization noise can be reduced.

  Further, the filter processing unit can use different weighting factors depending on the presence or absence of FIR (Finite Impulse Response Filter) filter processing used in the deblocking filter processing. Can be reduced.

  Another image processing apparatus according to the present invention includes a decoding unit that decodes compressed data that has undergone compression processing, and two or more frames from a frame image sequence that is output from the decoding unit based on an external instruction or automatically. A frame selecting unit that selects a frame image of the selected frame image as a selected frame image, and a filter processing unit that generates a single frame image by filtering the selected frame image selected by the frame selecting unit. The selected frame image is an I picture or slice (Intra Picture or slice), a P picture or slice (Predictive coded Picture or slice), or a B picture or slice. (Bi-predictive picture or slice or bi-predictive picture or slice ), A filtering process using different weighting factors is performed according to the picture type or the slice type indicating any of the above.

Further, when the weight coefficients of the I picture or slice, the P picture or slice, and the B picture or slice are Ki, Kp, and Kb, Ki, Kp, and Kb can satisfy the following. In other words, because the I picture or slice has the best image quality, the weighting coefficient is the largest, and the B picture or slice has the worst image quality. By making the weighting coefficient the smallest, the quantization noise can be reduced more efficiently. To do.
Ki ≧ Kp ≧ Kb
And Ki = Kp = Kb is not included

  The filter processing unit varies the weighting factors Ki, Kp, and Kb according to the compression rate at the time of image compression. When the compression rate is high, the ratio of each weighting factor is different from 1. When the compression ratio is low, the ratio of each weight coefficient can be set to a value close to 1. By varying the weighting factor for each picture and adopting different weighting factors within the same picture according to the compression rate, quantization noise can be reduced more efficiently.

  Specifically, the filter processing unit uses a different weighting factor predetermined for each selected frame based on the number of taps of the FIR filter used in the deblocking filter processing, or uses the deblocking filter Different weighting factors can be used based on the absolute value of the difference between the pixel values before and after the FIR filter processing used in the processing.

  In addition, the filter processing unit does not perform deblocking filter processing, and for pixels other than the block boundary, based on the quantization step of the macroblock, uses a weighting factor determined in advance for each selected frame, and For pixels that perform deblocking filter processing, different weighting factors may be used depending on the magnitude of the absolute value of the difference between pixel values before and after the quantization step and FIR filter processing used in the deblocking filter processing. In addition, various noises such as noise, attenuation noise, and quantization noise due to the image sensor can be reduced.

  An image processing method according to the present invention includes a decoding step for decoding compressed compressed data, and two or more frames from a frame image sequence output from the decoding unit based on an instruction from the outside or automatically. A frame selecting step for selecting an image as a selected frame image; and a filter processing step for generating a single frame image by filtering the selected frame image selected in the frame selecting step. Then, the selected frame image is an I picture or slice (Intra Picture or slice), a P picture or slice (Predictive coded Picture or slice), a B picture or slice (Bi-directional Predictive Picture or slice or Bi-predictive picture Bi-predic Filtering using different weighting factors is performed according to the picture type or slice type indicating which one is tive picture or slice).

  The program according to the present invention is configured to decode two or more frame images from a decoding process for decoding compressed compressed data and a frame image sequence output from the decoding unit based on an instruction from the outside or automatically. A frame selection process for selecting as a selected frame image, and a filter process for generating one frame image by filtering the selected frame image selected in the frame selection process. A frame picture is an I picture or slice (Intra Picture or slice), a P picture or slice (Predictive coded Picture or slice), or a B picture or slice (bidirectional prediction). Bi-directional Predictive Picture or slice or Bi-predictive Pict The filter processing using different weighting coefficients is executed in accordance with the picture type or slice type indicating which one is ure or slice).

  According to the present invention, an image processing device, an image processing method, and an image processing method capable of generating a high-quality still image with little noise degradation from a moving image encoded and recorded by image information compression using correlation between frames, And programs can be provided.

1 is an external view illustrating an example of an imaging apparatus according to a first embodiment of the present invention. 1 is a block diagram illustrating an imaging apparatus according to a first embodiment of the present invention. It is a flowchart which shows the still image data production | generation procedure in the imaging device concerning the 1st Embodiment of this invention. It is a figure which shows the frame structure in MPEG-2. (B) is a figure which shows the frame screen selected by the user, (b) is a figure which shows one of the frame screen before and behind that. It is a block diagram which shows the imaging device in the 3rd modification of the 1st Embodiment of this invention. It is a figure which shows the example in which one frame is divided | segmented into two slices. It is a figure which shows the imaging device in the 2nd Embodiment of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

  In this embodiment, the present invention uses different weighting factors based on the magnitude of noise generated in compression processing when selecting a plurality of frame images from a moving image and generating one still image. The present invention is applied to an image processing apparatus that performs the filtering process.

  As described above, the inventor of the present application has earnestly studied and can quantize noise and block even if a plurality of frame images can be simply averaged to remove random noise based on the image sensor. Knowing that distortion cannot be removed and performing filter processing using different weighting factors based on the magnitude of noise due to compression processing, removes noise due to the image sensor generated in the moving image and by compression processing I found a way to remove noise.

  Specifically, the noise generated by the compression processing is, for example, quantization noise or attenuation noise due to a deblocking filter. Here, in order to remove quantization noise, filter processing using different weighting factors is performed on a plurality of frame images based on a quantization step at the time of compression. In this case, for each selected frame image, it is also possible to perform a filtering process using different weighting factors for each frame, for each slice included in the frame, or for each macroblock included in the frame. The filtering process is, for example, based on a weighted average filter using a weighting factor corresponding to the quantization step. Furthermore, it is also possible to use different weighting factors depending on the picture type indicating whether the selected plurality of frame images are I pictures, P pictures, or B pictures.

  Also, in order to remove the attenuation noise of the deblocking filter, different weighting factors are used depending on the presence or absence of the FIR (Finite Impulse Response Filter) filter processing used in the deblocking filter processing when combining a plurality of frame images. Can be used. Also, a different weighting factor is used according to the number of taps of the FIR filter used in the deblocking filter process, or the absolute value of the difference between the pixel values before and after the FIR filter process used in the deblocking filter process is set. It is possible to use different weighting factors accordingly. Hereinafter, embodiments of the present invention will be described in detail.

<1> First Embodiment of the Present Invention In the present embodiment, two or more frame images are selected from a decoded frame image sequence based on an instruction from a user or automatically, and these frame images are selected. Depending on whether the picture is an I picture, P picture, or B picture, a weighting factor is changed to perform filter processing, thereby generating a high-quality still picture with little noise degradation.

<1-1> Imaging Device First, the imaging device according to the first embodiment will be described. FIG. 1 is an external view illustrating an example of an imaging apparatus 100 according to the first embodiment. The imaging apparatus 100 has portability, and includes a main body 102, an imaging lens 104, a plurality of operation units 106, and a liquid crystal monitor 108 as a monitor.

  The main body 102 records the image data captured through the imaging lens 104 so as to be re-viewable, and adjusts the recording timing and the angle of view according to the user input to the operation unit 106.

  In addition, in accordance with user input to the operation unit 106, playback of recorded moving image data, selection of a frame image, and generation of a still image are performed. The reproduced moving image is output to the outside and displayed on the liquid crystal monitor 108.

  FIG. 2 is a block diagram illustrating the imaging apparatus 100 according to the first embodiment. The imaging apparatus 100 includes an imaging unit 120, a signal processing unit 122, an image storage unit 124, an image processing unit 126, a recording I / F (interface) unit 128, an image output unit 130, and an imaging recording / playback control unit. 132 and a main storage unit 134. The operation unit 106, the signal processing unit 122, the image storage unit 124, the image processing unit 126, the recording I / F unit 128, the image output unit 130, the main storage unit 134, the drive control unit 154, and the imaging recording / playback control unit 132 They are connected via a system bus 136.

  The imaging unit 120 captures a subject through the imaging lens 104 and generates image data. Specifically, the imaging unit 120 is used for an IR (infrared) cut filter 140 that blocks near-infrared light, a zoom lens 142 for realizing a zoom function, a focus lens 144 used for focus adjustment, and exposure adjustment. An image sensor (imaging circuit) 148 composed of a CCD (Charge Coupled Devices) that photoelectrically converts light such as a subject image incident through the aperture 146 and the imaging lens 104 to generate an image signal, and image signals from the imaging element 148 An amplifier 150 that amplifies, an A / D (analog / digital) converter 152 that converts the amplified image signal into digital image data, a drive control unit 154 that controls driving of the zoom lens 142, the focus lens 144, and the aperture 146, and The driver circuit 156 is included.

  The signal processing unit 122 performs signal processing such as forming a luminance signal and a color signal on the input signal to form a color video signal, and transmits the color video signal to the image storage unit 124. Further, the average brightness of the screen is obtained, and the result is output to the imaging recording / playback control unit 132.

  The image storage unit 124 includes a buffer memory such as an SDRAM (Synchronous-DRAM), temporarily stores image data, and allows the image processing unit 126 and the like to refer to the image data.

  The image processing unit 126 converts the moving image data from the image storage unit 124 into MPEG-2, MPEG-4, H.264. Data for recording is generated by compression in a format such as H.264 / AVC. Further, the image processing unit 126 compresses the still image from the image storage unit 124 in a format such as JPEG (Joint Photographic Experts Group) and generates data for recording. Further, the image processing unit 126 generates a display image (view image) by reducing the image data in accordance with an instruction from the imaging recording / playback control unit 132. On the other hand, when reproducing the recorded data, the image processing unit 126 executes a process of decompressing and restoring the compressed storage data.

  The recording I / F unit 128 encodes image data through an encoding process to generate a recording signal (data stream), and records the storage signal on an arbitrary recording medium 160. As the arbitrary recording medium 160, a medium that does not require a power source such as a DVD or a BD, or a medium that requires a power source such as a RAM, an EEPROM, a nonvolatile RAM, a flash memory, or an HDD can be applied. Also, a separate recording medium that can be connected from the outside, such as an SD card, can be used.

  The drive control unit 154 controls the driving of the zoom lens 142, the focus lens 144, and the diaphragm 146 in response to an instruction from the imaging recording / playback control unit 132.

  The main storage unit 134 stores programs processed by the imaging recording / playback control unit 132.

  The imaging recording / playback control unit 132 manages and controls the entire imaging apparatus 100 using a semiconductor integrated circuit, and executes various calculations necessary for imaging and the like. The imaging recording / playback control unit 132 also functions as the still image generation unit 133.

  When a still image is generated from the recorded moving image data, the moving image data in the recording medium 160 is read out via the recording I / F unit 128, the compressed data is decompressed and restored by the image processing unit 126, and the result Are stored in the image storage unit 124. A frame image selected based on the user operation and a plurality of frame images before and after the frame image are stored in the image storage unit 124. In addition, information indicating whether each frame image has been encoded with an I, P, or B picture type is also stored.

  The still image generation unit 133 includes a frame selection unit 201 and a filter processing unit 202, and performs a motion compensation weighted average process, which is a type of filter processing, on a plurality of frame images stored in the image storage unit 124. Go to generate still image data.

<1-2> Configuration of Still Image Generating Unit Next, details of the still image generating unit 133 will be described. As illustrated in FIG. 2, the still image generation unit 133 includes a frame selection unit 201 and a filter processing unit 202. The frame selection unit 201 receives two or more frame images from the frame image sequence output from the image processing unit 126 functioning as a decoding unit that decodes the compressed data that has been compressed based on an instruction from the outside or automatically Is selected as the selected frame image. The selected frame image may be a continuous frame image or a discontinuous frame image with respect to the time axis. The filter processing unit 202 is a processing unit that generates one frame image by filtering the selected frame image selected by the frame selection unit 201. The filter processing unit 202 performs filter processing using different weighting factors based on the magnitude of noise generated by the compression processing. Specifically, depending on whether each of the plurality of frame images selected by the frame selection unit 201 is an I picture, a P picture, or a B picture, the still image data is generated by the weighted average filter by changing the weighting factor. To do.

<1-3> Still Image Data Generation Operation Next, a still image data generation procedure will be described. FIG. 3 is a flowchart showing a still image data generation procedure. By the user operation on the operation unit 106, the imaging apparatus 100 enters the still image data generation mode and starts image processing.

  As shown in FIG. 3, when image processing is started, a frame image is selected based on a user operation. The selection of the frame image is performed by the user operating the operation unit 106 while performing slow playback or frame advance playback of the moving image (step S200).

  When a frame image is selected, the frame image is stored in the image storage unit 124. Also, the immediately preceding two frame images and the immediately following two frame images are decompressed and restored in the time series of the frame images and stored in the image storage unit 124. Further, the picture type information of these five frame images is selected and stored in the image storage unit 124 (step S201).

  FIG. 4 is a diagram showing a frame structure in MPEG-2. Here, it is assumed that the frame image selected based on the user operation is the frame image indicated by the arrow in FIG. Here, it is represented by B (2). The last two pictures are an I picture and a B picture, which are represented by I (1) and B (1). The next two pictures are a P picture and a B picture, which are represented by P (1) and B (3).

  Assume that the weight coefficient for the I picture is Ki, the weight coefficient for the P picture is Kp, and the weight coefficient for the B picture is Kb. Here, Ki ≧ Kp ≧ Kb and Ki = Kp = Kb is not set.

The pixels of each frame are represented by pI (1), pB (1), pB (2), pP (1), and pB (3). When the subject is considered to be stationary through five pictures, each pixel of the frame image is subjected to a weighted averaging process based on the following equation (1) as follows. Pixel pS is calculated.
pS = (Ki × pI (1) + Kb × pB (1) + Kb × pB (2) + Kp × pP (1) + Kb × pB (3)) / (Ki + Kb + Kb + Kp + Kb) (1)

<1-4> Weighted Averaging Processing for Motion Compensation Next, processing when it is considered that a subject has motion will be described with reference to FIG. FIG. 5B shows a frame screen 303 selected by the user, and a frame image 301 shown in FIG. 5A is one of the frame screens before and after the frame image 303. The square gray object 302 at the upper left of the screen in FIG. 5A has moved slightly to the right in FIG. 5B (304). The background excluding the gray object 302 is assumed to be stationary. The motion vector V of the gray object 302 is calculated from the frame screens 301 and 303 in FIGS. 5 (a) and 5 (b). The motion vector V is calculated by a known method such as a gradient method or a matching method (for example, Patent Document 2).

  In performing the weighted averaging process of the above equation (1), the pixel belonging to the gray object 302 is moved by the motion vector V in FIG. 5A with respect to the pixel of the frame image 304 in FIG. An operation is performed using the pixel 302 that has been selected. That is, for the gray object 302, the calculation of the above equation (1) is performed on the gray object 302 in the frame image 301 and the object 304 in the frame image 303. On the other hand, for pixels belonging to a stationary background, calculation is performed using pixels at the same position in the screen. Also, in FIG. 5B, the region G newly appearing due to the movement of the gray object 302 does not have a pixel corresponding to FIG. 5A, so that the frame image 301 in FIG. The calculation of the above equation (1) is performed with the corresponding weighting factor being 0. The calculated still image is stored in the image storage unit 124 (step S220). The still image stored in the image storage unit 124 is read by the image processing unit 126, JPEG encoded, and recorded on the recording medium 160 (step S230). This completes the still image data generation procedure.

  In this embodiment, since one still image is generated from a plurality of frame images, noise based on the image sensor can be reduced, and different weights can be used depending on the picture types of I picture, P picture, and B picture. By performing the filter processing using the coefficients, it is possible to reduce quantization noise caused by quantization.

<1-5> First Modification Next, a second modification of the first embodiment will be described. In the first embodiment, the weighted averaging process for motion compensation has been described. However, the weighted averaging process for motion adaptation may be used. In the case of motion adaptation, a part of pixels considered not stationary is excluded from the calculation target. In the example of FIG. 5, for example, if the frame image 303 in FIG. 5B is a frame image selected by the user, the frame image 301 in FIG. 5A and the frame image 303 in FIG. For the pixels of the frame image 301 in FIG. 5A that belong to the region (G, 304), the weighting coefficient is set to 0 and the calculation of the above equation (1) is performed. For other pixels, calculation is performed using pixels at the same position in the screen. In terms of image quality, it is better to perform motion compensation, but motion adaptation has the advantage that the scale of the circuit that is the processing means can be reduced.

<1-6> Second Modification Next, a second modification of the first embodiment will be described. The frame selection unit 201 can exclude a frame image on which a zoom operation or a pan / tilt operation at a predetermined speed or more has been performed from the selected frame image. That is, when a zoom operation or a pan / tilt operation with relatively fast movement is performed during moving image shooting, those frame images are excluded. This is because when a zoom operation or a pan / tilt operation with relatively fast motion is performed, motion compensation is not performed properly.

  For example, in FIG. 4, for example, when five frame images are selected, a zoom operation or a pan / tilt operation with relatively fast movement is performed on a frame image behind the frame image selected based on the user operation. In this case, instead of the front 2 sheets + the back 2 sheets, the front 4 sheets are used. In determining whether or not a zoom operation or a pan / tilt operation with relatively fast movement has been performed, the frame selection unit 201 selects a selected frame image for a frame image in which the average value of motion vectors of the entire frame image is equal to or greater than a predetermined value. May be determined by calculating a motion vector based on a frame image, or operation information at the time of shooting may be stored in a moving image stream and selected at the time of reproduction.

<1-7> Third Modification Next, a third modification will be described. FIG. 6 is a block diagram illustrating an imaging apparatus 100a according to a third modification of the first embodiment. As shown in FIG. 6, the still image generation unit 133 includes an object detection unit 203 in addition to the configuration of FIG. 2. The target object detection unit 203 includes a specific target image in two or more frame images included in the frame image sequence output from the image processing unit 126 functioning as a decoding unit that decodes the compressed compressed data. Whether or not is detected. Then, based on the detection result of the object detection unit 203, the frame selection unit 201 selects two or more frames including the object as selected frame images.

  For example, the object detection unit 203 can automatically detect a user's smile in the video and select a frame image that includes the user's smile. In addition to a smile, a frame may be automatically selected when another object is included according to a user setting. Accordingly, the user can automatically acquire a desired still image from the moving image instead of the user's selection, and the operability is further improved.

<1-8> Other Modifications Next, a case where the selected frame is composed of five frame images will be described. First, the frame selection unit 201 can select so that one or more I pictures are included in the selected frame image. That is, frame image selection is limited so that the I picture is always included in the five frame images. For example, in FIG. 4, when the first P picture is selected by the user's instruction, the frame selection unit 201 sets the selected frame image by using the front three pictures and the rear one picture as the selected frame images. An I picture can be included in the image.

  By including the I picture in the selected frame image, since the I picture is the best in terms of image quality, the image quality of the still image can be improved. Similarly, since the picture quality of a B picture is inferior to that of an I picture or P picture, in the frame image selection in the frame selection unit 201, selection is limited to an I or P picture, and a weighted average is used for a B picture. By excluding from the conversion processing, the image quality of the still image to be generated can be improved. In this example, the still image generation unit 133 performs weighted averaging using five selected frames, but it goes without saying that the number of selected frames may be other than five.

  In the first embodiment and the modification, the frame image has been described as an example. However, in the case of interlaced scanning video data, a field image may be used. In that case, the same processing as in the case of the frame image may be performed. Or, the odd field is an odd field only and the motion compensation weighted averaging process is performed to create an image, and the even field is the even field only and the motion compensation weighted averaging process is performed to create an image. Alternatively, the still image may be generated by IP conversion.

  Furthermore, the filter processing unit 201 can vary the weighting factors Ki, Kp, and Kb according to the compression rate at the time of image compression, that is, the value ratio can be varied. In this case, when the compression rate is high, the ratio of each weighting factor can be set to a value away from 1, and when the compression rate is low, the ratio of each weighting factor can be set to a value close to 1.

  That is, when the compression rate is relatively small, Ki, Kp, and Kb are set to relatively close values. For example, {1, 0.9, 0.8}. On the other hand, when the compression ratio is relatively large, the ratios of the values of Ki, Kp, and Kb are made different. For example, {1, 0.7, 0.3}. When the compression ratio is relatively small, the amount of quantization noise of the P picture and B picture is relatively small. Therefore, even if the mixing ratio of the pixel values of the P picture and B picture is increased, the quantization noise is reduced. This is because it is not easily affected.

<2> Second Embodiment of the Present Invention Next, a second embodiment of the present invention will be described. This embodiment is an embodiment of the present invention. The present invention is applied to an imaging apparatus that generates a still image from a moving image composed of an I slice, a P slice, and a B slice standardized by the H.264 / AVC standard.

  FIG. 7 is a diagram illustrating an example in which one frame is divided into two slices. In this case, a picture in the first embodiment is a slice in the present embodiment.

  In the present embodiment, the configuration of the imaging device is the same as that of the imaging device according to the first embodiment shown in FIG. 2, and the frame selection unit is slice-selected in the still image generation unit shown in FIG. Part. FIG. 8 is a diagram illustrating the imaging device 100b according to the second embodiment. As shown in FIG. 8, a slice selection unit 204 is provided instead of the frame selection unit 201 shown in FIG. The slice selection unit 204, based on an instruction from the outside, or automatically, two or more slice images from the frame image sequence output from the image processing unit 126 that functions as a decoding unit that decodes the compressed compressed data. Preferably, four or more slice images are selected as selected slice images. The filter processing unit 202 is a processing unit that generates one frame image by filtering the selected slice image selected by the slice selection unit 204. The filter processing unit 202 performs filter processing using different weighting factors based on the magnitude of noise generated by the compression processing. Specifically, depending on whether each of the plurality of slice images selected by the slice selection unit 204 is an I slice, a P slice, or a B slice, the still image data is generated by the weighted average filter by changing the weighting factor. To do.

  In the second embodiment, the same effects as in the first embodiment are obtained. Furthermore, the first to third modifications described above and other modifications can also be applied in the present embodiment.

<3> Third Embodiment of the Present Invention Next, a third embodiment of the present invention will be described. In the first embodiment, the image quality depends on the I, P, and B picture types, but more specifically, the image quality depends on the size of the quantization step of each macroblock.

  Therefore, when the recorded moving image data is reproduced and decompressed and restored, the quantization steps of all macroblocks of each frame image are extracted and stored in the image storage unit 124 shown in FIG. A weighting factor K is determined corresponding to the value of the quantization step. The weighting factor K is determined to be smaller as the quantization step is larger. In the first embodiment, the weighting factor corresponding to the picture type is used, whereas in this embodiment, the weighting factor corresponding to the macroblock quantization step is used.

  In the present embodiment, the configuration of the imaging device is the same as the configuration of the imaging device according to the first embodiment shown in FIG. 2, and the configuration of the still image generation unit is the same as that of the still image generation unit shown in FIG. It is the same. Here, the filter processing unit 202 is a processing unit that generates one frame image by filtering the selected frame image selected by the frame selection unit 201, but the filter processing unit 202 in the present embodiment is a frame selection unit. In the selected frame image selected by the unit 201, still image data is generated by a weighted average filter using a weighting factor corresponding to the quantization step of the macroblock.

  Further, instead of the quantization step of each macroblock, an average value of quantization steps for each picture may be calculated, and a weighting factor K for each picture may be determined corresponding to the average value. In that case, the filter processing unit 202 may perform a filter operation using a weighting factor determined for each picture, not for each picture type.

  The third embodiment also has the same effect as the first embodiment. Furthermore, the first to third modifications described above and other modifications can also be applied in the present embodiment.

<4> Fourth Embodiment of the Present Invention Next, a fourth embodiment of the present invention will be described. The configurations of the imaging device and the still image generation unit according to the present embodiment are the same as the configurations of the imaging device and the still image generation unit according to the first embodiment shown in FIG. As mentioned above, H.M. In the H.264 / AVC deblocking filter processing, a 3 to 5 tap FIR filter is applied to pixels at a specific block boundary in accordance with a setting parameter for each slice. Block distortion due to quantization is reduced by the FIR filter processing. On the other hand, the spatial frequency high frequency component of the original signal may be lost by the FIR filter processing.

  When the recorded moving image data is reproduced and decompressed and restored, information on what FIR filter is applied to the pixels at the boundaries of all blocks of each frame image is extracted and stored in the image storage unit 124. . And the filter process part 201 shall use a different weighting coefficient according to the presence or absence of the FIR filter process used by a deblocking filter process. That is, the weighting coefficient K is determined in accordance with what kind of FIR filter is applied. In this case, for example, the weight coefficient K is determined to be smaller as the number of filter taps is larger. The weight coefficient of the unfiltered pixel is maximized.

  Alternatively, instead of information on what kind of FIR filter is applied, absolute information of a difference between a pixel value before being applied and a pixel value after being applied may be extracted. The larger the absolute value of the difference, the smaller the weighting factor K can be determined.

  It should be noted that the weighting factor K is determined in accordance with what kind of FIR filter is applied, or the combination of the absolute value of the difference before and after applying the FIR filter and the macroblock quantization step in the third embodiment. It may be. That is, for pixels other than the block boundary, the weighting factor K is determined according to the quantization step of the macroblock. For the block boundary, in addition to the quantization step, the weight coefficient K is determined by taking into account the absolute value of the difference before and after applying the FIR filter. A pixel other than the block boundary or a pixel with a difference of 0 is set to a weighting coefficient K corresponding to the quantization step, and a pixel with a difference of not 0 is further reduced according to the absolute value of the difference.

  In the present embodiment, since one still image is generated from a plurality of frame images, noise based on the image sensor can be reduced, and different weighting factors can be used depending on the presence or absence of FIR filter processing. H. It is possible to reduce noise during compression processing (attenuation noise caused by the deblocking filter) caused by the H.264 / AVC deblocking filter processing. Furthermore, if a weighting factor corresponding to the quantization step is used, it is possible to reduce quantization noise.

  It should be noted that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  For example, in the above-described embodiment, the hardware configuration has been described. However, the present invention is not limited to this, and arbitrary processing may be realized by causing a CPU (Central Processing Unit) to execute a computer program. Is possible. In this case, the computer program can be stored using various types of non-transitory computer readable media and supplied to the computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) are included. The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

DESCRIPTION OF SYMBOLS 100 Imaging device 102 Main body 104 Imaging lens 106 Operation part 108 Liquid crystal monitor 120 Imaging part 122 Signal processing part 124 Image storage part 126 Image processing part 128 part 130 Image output part 132 Imaging recording / reproduction control part 133 Still image generation part 134 Main storage part 136 System Bus 140 Cut Filter 142 Zoom Lens 144 Focus Lens 148 Image Sensor 150 Amplifier 152 Converter 154 Drive Control Unit 156 Drive Circuit I / F Recording 201 Frame Selection Unit 202 Filter Processing Unit 203 Object Detection Unit 204 Slice Selection Unit

Claims (12)

A decoding unit that decodes the compressed data that has been compressed;
A frame selection unit that selects two or more frame images as selection frame images from a frame image sequence output from the decoding unit based on an instruction from the outside or automatically;
A filter processing unit that generates one frame image by filtering the selected frame image selected by the frame selection unit;
The image processing apparatus, wherein the filter processing unit performs filter processing using different weighting factors based on the magnitude of noise generated in the compression processing. The decoding unit decodes compressed data compressed using different quantization steps,
The image processing apparatus according to claim 1, wherein the filter processing unit performs filter processing using different weighting factors based on a quantization step when the selected frame image is compressed.   The image processing apparatus according to claim 1, wherein the filter processing unit uses different weighting factors according to FIR filter processing used in deblocking filter processing. A decoding unit that decodes the compressed data that has been compressed;
A frame selection unit that selects two or more frame images as selection frame images from a frame image sequence output from the decoding unit based on an instruction from the outside or automatically;
A filter processing unit that generates one frame image by filtering the selected frame image selected by the frame selection unit;
The filter processing unit performs a filter process using different weighting factors depending on a picture type or a slice type indicating whether the selected frame image is an I picture or a slice, a P picture or a slice, or a B picture or a slice. An image processing apparatus to be implemented. The Ki, Kp, and Kb satisfy the following conditions when the weight coefficients of the I picture or slice, the P picture or slice, and the B picture or slice are Ki, Kp, and Kb, respectively. Image processing device.
Ki ≧ Kp ≧ Kb
And Ki = Kp = Kb is not included   The filter processing unit varies the weighting factors Ki, Kp, and Kb according to the compression rate at the time of image compression, and when the compression rate is high, the ratio of each weighting factor is a value that deviates from 1. 6. The image processing apparatus according to claim 4, wherein when the compression rate is low, the ratio of each weight coefficient is set to a value close to 1.   The said filter process part produces | generates said one frame image by carrying out the weighting averaging process of the said selection frame image for motion adaptation or motion compensation, The any one of Claims 1-6 characterized by the above-mentioned. The image processing apparatus described.   4. The image processing according to claim 3, wherein the filter processing unit uses a different weighting factor predetermined for each of the selected frames based on the number of taps of the FIR filter used in the deblocking filter processing. apparatus.   The filter processing unit uses a weighting factor determined in advance for each selected frame based on a magnitude of an absolute value of a difference between pixel values before and after the FIR filter process used in the deblocking filter process. The image processing apparatus according to claim 3.   The filter processing unit does not perform deblocking filter processing, and for pixels other than the block boundary, based on a macroblock quantization step, uses a weighting factor predetermined for each selected frame, and performs the deblocking -For a pixel to be subjected to filter processing, a weighting factor that differs depending on one or more selected from the magnitude of the absolute value of the difference between the pixel values before and after the quantization step and the FIR filter processing used in the deblocking filter processing The image processing apparatus according to claim 3, wherein the image processing apparatus is used. A decoding step of decoding the compressed data that has been compressed;
A frame selection step of selecting two or more frame images as selection frame images from a frame image sequence output from the decoding unit based on an instruction from the outside or automatically;
A filter processing step of generating one frame image by filtering the selected frame image selected in the frame selection step,
In the filtering process step, a filtering process using different weighting factors depending on a picture type or a slice type indicating whether the selected frame image is an I picture or a slice, a P picture or a slice, or a B picture or a slice. An image processing method characterized by being implemented. A decoding process for decoding the compressed compressed data;
A frame selection process for selecting two or more frame images as selection frame images from a frame image sequence output from the decoding unit based on an instruction from the outside or automatically;
Filtering the selected frame image selected in the frame selection process to generate a single frame image, and
In the filtering process, a filtering process using a weighting factor that differs depending on a picture type or a slice type indicating whether the selected frame image is an I picture or a slice, a P picture or a slice, or a B picture or a slice is executed. A program characterized by letting

Images