JP6378903B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly, to a noise removal apparatus and a noise removal method for generating an output image by performing noise removal processing on an input image.

  A general noise removal device applies a spatial filter such as a low-pass filter that removes high-frequency components uniformly to the image data of an input image in the spatial domain, so that the noise contained in the image Remove.

  Patent Document 1 listed below discloses a noise removal device that applies a median filter to a contour portion of an input image and applies an averaging filter to a portion other than the contour.

JP 9-233369 A

  According to the general noise removal apparatus described above, the noise removal effect is low only by applying the spatial filter once to the input image. Although the noise removal effect can be improved by applying the spatial filter multiple times, in this case, the resolution is lowered and the contour portion becomes unclear, and the image quality seems to be deteriorated on the whole image. is there. Further, since the spatial filter needs to be applied a plurality of times, the processing time increases, and a temporal memory for storing intermediate values is required, which increases the circuit scale.

  The present invention has been made in view of such circumstances, and can achieve a high noise removal effect while avoiding deterioration of image quality, and can reduce processing time and circuit scale. An object is to obtain a possible image processing apparatus and image processing method.

An image processing apparatus according to a first aspect of the present invention is an image processing apparatus that generates an output image by performing noise removal processing on an input image, and is extracted from the input image in predetermined block units. By converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A quantization unit to be generated, a calculation unit that calculates an activity evaluation value for each block based on the first image data, and a quantization for each block used for quantization processing based on the activity evaluation value for each block A quantization value setting unit that sets a value, a smoothing processing unit that calculates a smoothing value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel, Provided, the arithmetic unit A smoothing value of the unit, calculates a difference absolute value sum of the block mean value of the smoothed value, the block average value of the sum of absolute differences, and calculates the active evaluation value for each block Is.

According to the image processing apparatus according to the first aspect, the quantization value setting unit sets the quantization value for each block used for the quantization processing based on the activity evaluation value for each block. As described above, by setting the quantization value for each block based on the activity evaluation value for each block, it is possible to execute an appropriate noise removal process for each block according to the attribute of the block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
Further, according to the image processing apparatus according to the first aspect, the calculation unit calculates a sum of absolute differences between the smoothed value of each pixel and the average value of the smoothed values in the block, and the block of the sum of the absolute differences The activity evaluation value for each block is calculated as the average value. Thereby, it becomes possible to obtain | require appropriately the activeness evaluation value for every block by a calculating part.

An image processing apparatus according to a second aspect of the present invention is an image processing apparatus that generates an output image by performing noise removal processing on an input image, and is extracted from the input image in predetermined block units. By converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A quantization unit to be generated, a calculation unit that calculates an activity evaluation value for each block based on the first image data, and a quantization for each block used for quantization processing based on the activity evaluation value for each block A quantization value setting unit for setting a value, and the conversion unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap the input image, and home The image portion where the number of blocks overlap, the optimum value obtained from a plurality of blocks that overlap is characterized in that the output as image data of the image portion.

According to the image processing device according to the second aspect, the quantization value setting unit sets the quantization value for each block used for the quantization processing based on the activity evaluation value for each block. As described above, by setting the quantization value for each block based on the activity evaluation value for each block, it is possible to execute an appropriate noise removal process for each block according to the attribute of the block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
Further, according to the image processing device according to the second aspect, the transform unit and the quantization unit sequentially perform processing on a plurality of blocks extracted so as to partially overlap from the input image. In this way, it is possible to suppress the occurrence of block noise by overlapping parts of the blocks with each other. For an image portion where a plurality of blocks overlap in the output image, the optimum value obtained from the plurality of overlapping blocks is output as image data of the image portion. As described above, by outputting an optimum value (for example, an average value) obtained from a plurality of blocks for overlapping portions, it is possible to achieve a high noise removal effect while avoiding deterioration in image quality.

An image processing apparatus according to a third aspect of the present invention is an image processing apparatus that generates an output image by performing noise removal processing on an input image, and is extracted from the input image in predetermined block units. By converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A quantization unit to be generated, a calculation unit that calculates an activity evaluation value for each block based on the first image data, and a quantization for each block used for quantization processing based on the activity evaluation value for each block A quantization value setting unit for setting a value, and the conversion unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap the input image, and home For an image portion where a number of blocks overlap, the output data of the image portion in the processing related to the current block is used as input data of the image portion in the processing related to the next macro block. .

According to the image processing apparatus according to the third aspect, the quantization value setting unit sets the quantization value for each block used for the quantization processing based on the activity evaluation value for each block. As described above, by setting the quantization value for each block based on the activity evaluation value for each block, it is possible to execute an appropriate noise removal process for each block according to the attribute of the block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
Further, according to the image processing device according to the third aspect, the conversion unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap from the input image. In this way, it is possible to suppress the occurrence of block noise by overlapping parts of the blocks with each other. For an image portion where a plurality of blocks overlap in the output image, the output data of the image portion in the process related to the current block is used as input data of the image portion in the process related to the next macroblock. Therefore, since the noise removal process is performed substantially multiple times for the same image portion in a series of processes in which a plurality of blocks are sequentially processed, it is compared with the case where the optimum value is obtained from the output values of a plurality of overlapping blocks. Thus, the processing time can be shortened.

The image processing apparatus according to the fourth aspect of the present invention performs correction processing for each frequency component on the third image data , particularly in the image processing apparatus according to any one of the first to third aspects. The correction unit for generating the fourth image data and the correction value setting unit for setting the correction value for each block used for the correction processing based on the activity evaluation value for each block are further provided. Is.

According to the image processing apparatus according to the fourth aspect, the correction value setting unit sets the correction value for each block used for the correction processing based on the activity evaluation value for each block. Thus, by setting the correction value for each block based on the activity evaluation value for each block, it is possible to execute an appropriate correction process for each block according to the attribute of the block. As a result, the noise removal effect for each block can be improved, so that the image quality of the entire image can be improved.

The image processing device according to a fifth aspect of the present invention is the image processing device according to any one of the first to fourth aspects, in particular, the quantized value setting unit is a block having a low activity evaluation value. For a block with a high activity evaluation value, a medium quantization value is set, and for a block with a high activity evaluation value, a low quantization value is set. It is characterized by setting.

According to the image processing device according to the fifth aspect, the quantization value setting unit sets a high quantization value for a block having a low activity evaluation value. For a flat portion with a low activity evaluation value and noise that is conspicuous, it is possible to obtain a high noise removal effect by quantization processing by setting the quantization value high. The quantized value setting unit sets a low quantized value for a block having a high activity evaluation value. For edges and textures with high activity evaluation values, by setting the quantization value low, it is possible to avoid blurring of edges and textures due to quantization and improve image quality. It becomes. The quantized value setting unit sets an intermediate quantized value for a block having a medium activity evaluation value. For image portions other than the flat portion, the edge portion, and the texture portion, by setting the quantization value to a medium level, it is possible to avoid that the noise removal effect on the image portion is excessive or small. .

In the image processing device according to the sixth aspect of the present invention, in particular, in the image processing device according to the fourth aspect, the correction value setting unit uses a mask in which an arbitrary multiplier is set for each frequency component as an activity evaluation value. A multiplier set in a mask corresponding to the activity evaluation value calculated by the calculation unit is set as a correction value.

According to the image processing device of the sixth aspect, the correction value setting unit holds a plurality of types of masks each having an arbitrary multiplier set for each frequency component according to the activity evaluation value, and is calculated by the calculation unit. The multiplier set in the mask corresponding to the activated activity evaluation value is set as the correction value. As a result, the correction value corresponding to the activity evaluation value for each block can be set easily and appropriately.

An image processing apparatus according to a seventh aspect of the present invention is an image processing apparatus that generates an output image by performing noise removal processing on an input image, and is extracted from the input image in predetermined block units. By converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A quantization unit to be generated, a first calculation unit that calculates a noise value for each block based on the first image data, and a quantization value for each block used for quantization processing based on the noise value for each block And a smoothing processing unit that calculates a smoothing value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel. The first arithmetic unit is The difference absolute value sum between the pixel value of each pixel and the smooth value of each pixel is calculated, and the noise value for each block is calculated as an average value in the block of the difference absolute value sum. .

According to the image processing device according to the seventh aspect, the quantization value setting unit sets the quantization value for each block used for the quantization processing based on the noise value for each block. In this way, by setting the quantization value for each block based on the noise value for each block, it is possible to execute an appropriate noise removal process for each block according to the noise level for each block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
Moreover, according to the image processing apparatus which concerns on a 7th aspect, the 1st calculating part calculates the difference absolute value sum of the pixel value of each pixel, and the smooth value of each pixel, and the said difference absolute value sum The noise value for each block is calculated as the average value within the block. Thereby, the noise value for each block can be appropriately obtained by the first calculation unit.

An image processing apparatus according to an eighth aspect of the present invention is an image processing apparatus that generates an output image by performing noise removal processing on an input image, and is extracted from the input image in predetermined block units. By converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A quantization unit to be generated, a first calculation unit that calculates a noise value for each block based on the first image data, and a quantization value for each block used for quantization processing based on the noise value for each block A quantizing value setting unit for setting a plurality of blocks extracted so as to partially overlap from the input image, and the conversion unit and the quantization unit sequentially perform processing on the output image. Duplicate The image portion of the block overlap, the optimum value obtained from a plurality of blocks that overlap is characterized in that the output as image data of the image portion.

According to the image processing device according to the eighth aspect, the quantization value setting unit sets the quantization value for each block used for the quantization processing based on the noise value for each block. In this way, by setting the quantization value for each block based on the noise value for each block, it is possible to execute an appropriate noise removal process for each block according to the noise level for each block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
Further, according to the image processing device according to the eighth aspect, the conversion unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap from the input image. In this way, it is possible to suppress the occurrence of block noise by overlapping parts of the blocks with each other. For an image portion where a plurality of blocks overlap in the output image, the optimum value obtained from the plurality of overlapping blocks is output as image data of the image portion. As described above, by outputting an optimum value (for example, an average value) obtained from a plurality of blocks for overlapping portions, it is possible to achieve a high noise removal effect while avoiding deterioration in image quality.

An image processing apparatus according to a ninth aspect of the present invention is an image processing apparatus that generates an output image by performing noise removal processing on an input image, and is extracted from the input image in predetermined block units. By converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A quantization unit to be generated, a first calculation unit that calculates a noise value for each block based on the first image data, and a quantization value for each block used for quantization processing based on the noise value for each block A quantizing value setting unit for setting a plurality of blocks extracted so as to partially overlap from the input image, and the conversion unit and the quantization unit sequentially perform processing on the output image. Duplicate The image portion of the block overlap, the output data of the image portion in the processing relating to the current block, is characterized in that is used as input data of the image portion in the processing relating to the next macroblock.

According to the image processing device according to the ninth aspect, the quantization value setting unit sets the quantization value for each block used for the quantization processing based on the noise value for each block. In this way, by setting the quantization value for each block based on the noise value for each block, it is possible to execute an appropriate noise removal process for each block according to the noise level for each block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
Further, according to the image processing device according to the ninth aspect, the conversion unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap from the input image. In this way, it is possible to suppress the occurrence of block noise by overlapping parts of the blocks with each other. For an image portion where a plurality of blocks overlap in the output image, the output data of the image portion in the process related to the current block is used as input data of the image portion in the process related to the next macroblock. Therefore, since the noise removal process is performed substantially multiple times for the same image portion in a series of processes in which a plurality of blocks are sequentially processed, it is compared with the case where the optimum value is obtained from the output values of a plurality of overlapping blocks. Thus, the processing time can be shortened.

The image processing apparatus according to the tenth aspect of the present invention is the image processing apparatus according to any one of the seventh to ninth aspects, and in particular, executes correction processing for each frequency component on the third image data. The correction unit that generates the fourth image data, the second arithmetic unit that calculates the activity evaluation value for each block based on the first image data, and the correction based on the activity evaluation value for each block And a correction value setting unit that sets a correction value for each block used for processing.

According to the image processing device according to the tenth aspect, the correction value setting unit sets the correction value for each block used for the correction processing based on the activity evaluation value for each block. Thus, by setting the correction value for each block based on the activity evaluation value for each block, it is possible to execute an appropriate correction process for each block according to the attribute of the block. As a result, the noise removal effect for each block can be improved, so that the image quality of the entire image can be improved.

The image processing device according to an eleventh aspect of the present invention is the image processing device according to any one of the seventh to tenth aspects, in particular, the quantization value setting unit has a higher quantization value as the noise value is higher. A value is set.

According to the image processing device of the eleventh aspect, the quantization value setting unit sets a higher quantization value as the noise value is higher. As described above, for a block having a high noise value, a high noise removal effect by the quantization process can be obtained by setting the quantization value high. In addition, by setting a low quantization value for a block having a low noise value, it is possible to avoid deterioration in image quality due to quantization processing.

In the image processing device according to the twelfth aspect of the present invention, in the image processing device according to the tenth aspect, in particular, based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel, The image processing apparatus further includes a smoothing processing unit that calculates a smooth value of each pixel, and the second calculation unit calculates a sum of absolute differences between the smooth value of each pixel and the average value of the smooth value in the block, and the difference The activity evaluation value for each block is calculated as the average value within the block of the sum of absolute values.

According to the image processing device of the twelfth aspect, the second calculation unit calculates a sum of absolute differences between the smooth value of each pixel and the average value of the smooth values in the block, The activity evaluation value for each block is calculated as the average value within the block. Thereby, the activity evaluation value for each block can be appropriately obtained by the second calculation unit.

An image processing apparatus according to a thirteenth aspect of the present invention is the image processing apparatus according to the tenth aspect, in which the correction value setting unit uses a mask in which an arbitrary multiplier is set for each frequency component as an activity evaluation value. A multiplier set in the mask corresponding to the activity evaluation value calculated by the second calculation unit is set as a correction value.

According to the image processing apparatus of the thirteenth aspect, the correction value setting unit holds a plurality of types of masks each having an arbitrary multiplier set for each frequency component according to the activity evaluation value, and the second calculation The multiplier set in the mask corresponding to the activity evaluation value calculated by the unit is set as the correction value. As a result, the correction value corresponding to the activity evaluation value for each block can be set easily and appropriately.

In the image processing device according to the fourteenth aspect of the present invention, in particular, in the image processing device according to the sixth or thirteenth aspect, the correction value setting unit is a low frequency for a block having a low activity evaluation value. In the component, 1 times is set as the correction value, and in the medium frequency component and the high frequency component, approximately zero times is set as the correction value. For the block whose activity evaluation value is the medium level, the low frequency component and the medium frequency component are set. 1 is set as the correction value, and the high frequency component is set to approximately zero times as the correction value. For blocks with a high activity evaluation value, the low frequency component, the medium frequency component, and the high frequency component are set to 1 time. Is set as a correction value.

According to the image processing apparatus of the fourteenth aspect, the correction value setting unit sets , as a correction value, a low-frequency component as a correction value for a low-frequency component for a block whose activity evaluation value is at a low level. In the component, approximately zero times is set as the correction value. With respect to a flat portion having a low activity evaluation value and noise that is conspicuous, it is possible to obtain a high noise removal effect by correction processing by performing masking that cuts off the medium frequency component and the high frequency component. In addition, the correction value setting unit sets 1 time as a correction value for the low-frequency component, the medium-frequency component, and the high-frequency component for a block having a high activity evaluation value. For edges and textures with high activity evaluation values, by performing masking that does not cut all frequency components, correction processing that sharpens edges and textures can be realized, and image quality can be improved. . In addition, the correction value setting unit sets 1 times as a correction value for a low frequency component and a medium frequency component as a correction value for a block whose activity evaluation value is a medium level, and substantially zero times as a correction value for a high frequency component. To do. For image portions other than the flat portion, the edge portion, and the texture portion, it is possible to obtain a moderate noise removal effect by the correction processing by performing masking that cuts only high-frequency components.

In the image processing device according to the fifteenth aspect of the present invention, in the image processing device according to any one of the first to fourteenth aspects, the noise value of the entire input image, the activity evaluation value of the entire input image, And a determination unit that determines whether or not the noise removal process needs to be executed.

According to the image processing device of the fifteenth aspect, the determination unit determines whether or not the noise removal process is necessary based on the noise value of the entire input image and the activity evaluation value of the entire input image. As a result, noise removal processing can be executed for an input image that truly requires noise removal processing, and execution of noise removal processing on an unnecessary input image can be avoided, so that power consumption can be reduced.

In the image processing device according to the sixteenth aspect of the present invention, in particular, in the image processing device according to the fifteenth aspect, the determination unit divides the noise value of the entire input image by the activity evaluation value of the entire input image. The noise intensity is calculated, and if the noise intensity is less than the predetermined threshold, it is determined that the noise removal process is not necessary. If the noise intensity is greater than or equal to the threshold, noise removal is performed. It is characterized by determining that the execution of the process is necessary.

According to the image processing device of the sixteenth aspect, the determination unit calculates the noise intensity by dividing the noise value of the entire input image by the activity evaluation value of the entire input image. If the noise intensity is less than a predetermined threshold value, it is determined that the noise removal process is not necessary. If the noise intensity is equal to or greater than the threshold value, the noise removal process needs to be performed. judge. When the determination unit performs such determination processing, it is possible to specify with high accuracy an input image that truly requires noise removal processing.

An image processing apparatus according to a seventeenth aspect of the present invention is the image processing apparatus according to any one of the first to sixteenth aspects, in particular, the input image has a plurality of color components, and the image processing apparatus includes: The present invention is characterized in that noise removal processing is executed independently for each color component.

According to the image processing apparatus of the seventeenth aspect, the noise removal process is executed independently for each color component. As described above, by performing the noise removal process independently on each color component of the input image, an optimum noise removal effect can be obtained for each color component.

An image processing device according to an eighteenth aspect of the present invention is an image processing device that generates an output image by performing noise removal processing on an input image, and is extracted from the input image in predetermined block units. By converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A quantization unit to be generated, a calculation unit that calculates an activity evaluation value for each block based on the first image data, and a quantization for each block used for quantization processing based on the activity evaluation value for each block A quantized value setting unit for setting a value, wherein the input image has a plurality of color components, and the image processing device performs noise removal processing on the plurality of color components in an integrated manner. Is what .

According to the image processing device according to the eighteenth aspect, the quantization value setting unit sets the quantization value for each block used for the quantization processing based on the activity evaluation value for each block. As described above, by setting the quantization value for each block based on the activity evaluation value for each block, it is possible to execute an appropriate noise removal process for each block according to the attribute of the block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
Further, according to the image processing device according to the eighteenth aspect, the noise removal processing is executed on the plurality of color components in an integrated manner. Thus, noise detection accuracy can be improved by performing integrated noise removal processing on a plurality of color components. In addition, since the variation in the noise removal effect between the color components is suppressed, the image quality can be improved.

An image processing apparatus according to a nineteenth aspect of the present invention is an image processing apparatus that generates an output image by performing noise removal processing on an input image, and is extracted from the input image in predetermined block units. By converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A quantization unit to be generated, a first calculation unit that calculates a noise value for each block based on the first image data, and a quantization value for each block used for quantization processing based on the noise value for each block The input image has a plurality of color components, and the image processing apparatus performs noise removal processing on the plurality of color components in an integrated manner. To do

According to the image processing device according to the nineteenth aspect, the quantization value setting unit sets the quantization value for each block used for the quantization processing based on the noise value for each block. In this way, by setting the quantization value for each block based on the noise value for each block, it is possible to execute an appropriate noise removal process for each block according to the noise level for each block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
Moreover, according to the image processing device which concerns on a 19th aspect, a noise removal process is integrally performed with respect to a several color component. Thus, noise detection accuracy can be improved by performing integrated noise removal processing on a plurality of color components. In addition, since the variation in the noise removal effect between the color components is suppressed, the image quality can be improved.
An image processing method according to a twentieth aspect of the present invention is an image processing method for generating an output image by executing a noise removal process on an input image, wherein the image is extracted in units of predetermined blocks from the input image. The step (A) of converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A step (B) for generating data, a step (C) for calculating an activity evaluation value for each block based on the first image data, and a quantization process based on the activity evaluation value for each block A step (D) of setting a quantization value for each block, and a step of calculating a smoothing value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel (E )When The step (C) calculates the sum of absolute differences between the smoothed value of each pixel and the average value of the smoothed values in the block, and uses the difference absolute value sum as the average value of the difference in the block. The method includes a step of calculating a sex evaluation value.
According to the image processing method of the twentieth aspect, the quantization value for each block used for the quantization processing is set based on the activity evaluation value for each block. As described above, by setting the quantization value for each block based on the activity evaluation value for each block, it is possible to execute an appropriate noise removal process for each block according to the attribute of the block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
According to the image processing method of the twentieth aspect, step (C) calculates the sum of absolute differences between the smoothed value of each pixel and the average value of the smoothed values in the block, and calculates the sum of the absolute differences. A step of calculating an activity evaluation value for each block as the average value within the block. Thereby, it becomes possible to obtain | require appropriately the activeness evaluation value for every block by step (C).
An image processing method according to a twenty-first aspect of the present invention is an image processing method for generating an output image by performing a noise removal process on an input image, which is extracted from the input image in predetermined block units. The step (A) of converting the first image data in the spatial domain to the second image data in the frequency domain including a plurality of frequency components, and performing the quantization process on the second image data, A step (B) for generating data, a step (C) for calculating a noise value for each block based on the first image data, and a quantum for each block used for the quantization process based on the noise value for each block. A step (D) for setting a conversion value, and a step (E) for calculating a smoothing value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel. Preparation The step (C) calculates a sum of absolute differences between the pixel value of each pixel and a smooth value of each pixel, and calculates a noise value for each block as an average value in the block of the difference absolute value sum. It is characterized by including.
According to the image processing method according to the twenty-first aspect, the quantization value for each block used for the quantization processing is set based on the noise value for each block. In this way, by setting the quantization value for each block based on the noise value for each block, it is possible to execute an appropriate noise removal process for each block according to the noise level for each block. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.
According to the image processing method of the twenty-first aspect, step (C) calculates a sum of absolute differences between the pixel value of each pixel and the smooth value of each pixel, and blocks the sum of the absolute differences A step of calculating a noise value for each block as the inner average value is included. As a result, the noise value for each block can be appropriately obtained in step (C).

  According to the present invention, an image processing apparatus and an image processing method capable of realizing a high noise removal effect while avoiding deterioration in image quality, and capable of reducing processing time and circuit scale. Can be obtained.

It is a figure which shows the structure of the image processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the flow of the noise removal process which an image processing apparatus performs. It is a figure for demonstrating the setting process of the quantization value by a setting part. It is a figure for demonstrating the setting process of the scaling matrix by a setting part. It is a figure for demonstrating the setting process of the scaling matrix by a setting part. It is a figure for demonstrating the setting process of the scaling matrix by a setting part. It is a figure for demonstrating the setting process of the scaling matrix by a setting part. It is a figure for demonstrating the setting process of the scaling matrix by a setting part. It is a figure which shows an overlap process. It is a figure for demonstrating the 1st setting process of the quantization value by a setting part. It is a figure for demonstrating the 2nd setting process of the quantization value by a setting part. It is a figure which shows the structure of the image processing apparatus for implement | achieving the 2nd setting process of a quantization value. It is a figure for demonstrating the setting process of the scaling matrix by a setting part. It is a figure for demonstrating the setting process of the scaling matrix by a setting part. It is a figure for demonstrating the setting process of the scaling matrix by a setting part. It is a figure which shows the structure of the image processing apparatus which concerns on a 1st modification. It is a flowchart which shows the flow of the noise removal process which the image processing apparatus which concerns on a 1st modification performs. It is a figure which shows the structure of the image processing apparatus which concerns on a 2nd modification. It is a flowchart which shows the flow of the noise removal process which the image processing apparatus which concerns on a 2nd modification performs.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, the element which attached | subjected the same code | symbol in different drawing shall show the same or corresponding element.

  FIG. 1 is a diagram showing a configuration of an image processing apparatus 1 according to an embodiment of the present invention. Bayer array input images having R, Gr, Gb, and B color components are classified and input to the image processing apparatus 1 for each color component. The image processing apparatus 1 generates and outputs an output image by performing noise removal processing independently on the input image of each color component. The color components of the input image are not limited to R, Gr, Gb, and B, but may be R, G, B, Y, U, V, or the like.

  As shown in the connection relationship of FIG. 1, the image processing apparatus 1 includes a transform unit 11, a quantizer 12, a corrector 13, an inverse quantizer 14, an inverse transform unit 15, a selector 16, a setting unit 17, a filter 18, Arithmetic units 19 and 20 and setting units 21 to 24 are provided.

  The conversion unit 11 performs a plurality of frequency components on the image data D1 of the spatial region extracted from the input image in a predetermined block unit (in the example of the present embodiment, a macroblock unit of 8 rows × 8 columns) by DCT conversion. Is converted into image data D2 in the frequency domain including The block size is not limited to 8 rows × 8 columns, but may be 16 rows × 16 columns. As the block size increases, the circuit scale increases, but the image quality improves because the frequency component can be finely controlled.

  The quantization unit 12 generates image data D3 by performing quantization processing on the image data D2.

  The correction unit 13 generates image data D4 by executing correction processing for each frequency component on the image data D3.

  The inverse quantization unit 14 generates image data D5 by performing inverse quantization processing on the image data D4.

  The inverse transform unit 15 transforms the frequency domain image data D5 into the spatial domain image data D6 by inverse DCT transform.

  The selector 16 selects and outputs the image data D1 or the image data D6.

  The CPU 2 determines whether or not the noise removal process is necessary based on the noise value NOISE (PIC) of the entire input image and the activity evaluation value ACT (PIC) of the entire input image. Then, the determination result is set in the setting unit 17. The setting unit 17 causes the selector 16 to select the image data D6 when execution is necessary, and causes the selector 16 to select image data D1 when execution is not necessary.

  The filter 18 is an arbitrary smoothing filter (a Gaussian filter of 3 rows × 3 columns in the example of the present embodiment), and a pixel value of each pixel and a plurality (3 rows × 3 columns of pixels) adjacent to the pixel. In this case, the weighted average value GAUS (i, j) of each pixel is calculated based on the pixel values of the surrounding pixels.

  The computing unit 19 calculates a noise value NOISE (MB) for each macroblock based on the image data D1 and the image data D1 after weighted averaging by the filter 18. In addition, the calculation unit 19 calculates a noise value NOISE (PIC) as an average value of the noise values NOISE (MB) regarding all the macroblocks in the input image. Note that the total value of noise values NOISE (MB) for all macroblocks in the input image is input from the arithmetic unit 19 to the CPU 2, and the CPU 2 divides the total value by the number of macroblocks, thereby the noise value NOISE (PIC). May be calculated.

  The computing unit 20 calculates the activity evaluation value ACT (MB) for each macroblock based on the image data D1 after weighted averaging by the filter 18. In addition, the calculation unit 20 calculates the activity evaluation value ACT (PIC) as the average value of the activity evaluation values ACT (MB) for all the macroblocks in the input image. The sum of the activity evaluation values ACT (MB) for all the macroblocks in the input image is input from the arithmetic unit 20 to the CPU 2, and the CPU 2 divides the sum by the number of macroblocks to determine the activity evaluation value. ACT (PIC) may be calculated.

  The setting unit 21 sets a quantization value QP (PIC) for each input image used for the quantization process by the quantization unit 12 based on the noise value NOISE (PIC).

  The setting unit 22 uses the quantization unit 12 based on internal parameters of the image processing apparatus 1 (for example, the quantization value QP (PIC), the activity evaluation value ACT (MB), the noise value NOISE (MB), etc.). A quantization value QP (MB) for each macro block used for quantization processing is set.

  The setting unit 23 sets a scaling matrix SM (PIC) describing a correction value for each input image used for correction processing by the correction unit 13 based on the image data D2 of the entire input image.

  The setting unit 24 sets internal parameters of the image processing apparatus 1 (for example, scaling matrix SM (PIC), activity evaluation value ACT (MB), noise value NOISE (PIC), activity evaluation value ACT (PIC), etc.)). Based on this, the scaling matrix SM (MB) describing the correction value for each macroblock used for the correction processing by the correction unit 13 is set.

  FIG. 2 is a flowchart showing the flow of noise removal processing executed by the image processing apparatus 1. First, in step P101, the setting unit 17 sets the selector 16 based on the determination result of whether or not the noise removal processing by the CPU 2 is necessary. If execution is not necessary, the process ends.

  If execution is necessary, in step P102, the setting unit 21 sets a quantized value QP (PIC).

  Next, in step P103, the setting unit 23 sets a scaling matrix SM (PIC).

  Next, in steps P104 to P110, the noise removal processing for each macroblock is repeatedly executed for all the macroblocks included in the input image.

  In step P104, the arithmetic unit 20 calculates the activity evaluation value ACT (MB).

  Next, in step P105, the setting unit 22 sets the quantization value QP (MB). The setting unit 24 sets a scaling matrix SM (MB).

  Next, in step P106, the conversion unit 11 converts the image data D1 into image data D2.

  Next, in step P107, the quantization unit 12 generates image data D3 from the image data D2 by quantization processing using the quantization value QP (MB).

  Next, in step P108, the correction unit 13 generates image data D4 from the image data D3 by correction processing using the scaling matrix SM (MB).

  Next, in step P109, the inverse quantization unit 14 generates image data D5 from the image data D4 by inverse quantization processing.

  Next, in step P110, the inverse conversion unit 15 converts the image data D5 into image data D6 by inverse DCT conversion. When the noise removal processing for each macro block is completed for all the macro blocks included in the input image, the processing ends.

  Hereinafter, details of the processing contents executed by each processing unit shown in FIG. 1 will be described in order.

<Calculation processing of GAUS (i, j)>
The filter 18 sets the pixel value of each pixel in the macroblock as CURR (i, j), and performs the calculation represented by the following formula (1), thereby obtaining the weighted average value GAUS (i, j) of each pixel. calculate.

<NOISE (MB) calculation process>
The calculation unit 19 performs the calculation represented by the following expression (2), thereby calculating the absolute difference between the pixel value CURR (i, j) of each pixel and the weighted average value GAUS (i, j) of each pixel. Calculate the sum. Then, a noise value NOISE (MB) for each macroblock is calculated as an average value in the macroblock of the sum of absolute differences by performing the calculation represented by the following formula (3).

<Setting process of QP (PIC)>
FIG. 3 is a diagram for explaining the setting process of the quantization value QP (PIC) by the setting unit 21. The calculation unit 19 calculates a noise value NOISE (PIC) as an average value of the noise values NOISE (MB) regarding all macroblocks in the input image. The setting unit 21 holds a look-up table that describes an optimum quantized value QP (PIC) for each noise value NOISE (PIC). The setting unit 21 normalizes the lookup table in the distribution range of the weighted average value GAUS (i, j) in the histogram of the weighted average value GAUS (i, j) and the number of pixels. Then, by using the normalized lookup table, the quantization value corresponding to the noise value NOISE (PIC) is set as the quantization value QP (PIC) related to the input image. As illustrated in FIG. 3, the setting unit 21 sets a higher quantized value QP (PIC) as the noise value NOISE (PIC) is higher.

<SM (PIC) setting process>
4 to 8 are diagrams for explaining the setting process of the scaling matrix SM (PIC) by the setting unit 23. FIG. First, the setting unit 23 obtains a quantization value QP (PIC) by the same method as the setting unit 21.

  Next, the setting unit 23 acquires the image data D3 after the DCT conversion process and the quantization process using the quantization value QP (PIC) are completed for all the macroblocks in the input image. Then, for each frequency component in the image data D3, the absolute value sum K1 of each frequency component value related to the input image is obtained by accumulating the absolute value of the frequency component value for all macroblocks. An example of the absolute value sum K1 is shown in FIG. The upper left corner is a frequency component value corresponding to the DC component, and the lower right corner is a frequency component value corresponding to the highest frequency component.

  The setting unit 23 acquires the image data D2 after the DCT conversion process is completed for all macroblocks in the ideal image prepared in advance. An ideal image is an image that does not include noise and is obtained by photographing the same subject with the same composition as the input image. For example, when the image processing apparatus 1 is used for a surveillance camera, an image including a lot of noise taken in a low illuminance environment such as night becomes an input image and is photographed in a high illuminance environment such as daytime. An image containing almost no noise is an ideal image. For each frequency component in the image data D2, the setting unit 23 obtains an absolute value sum K2 of each frequency component value related to the ideal image by accumulating the absolute value of the frequency component value for all the macroblocks. An example of the absolute value sum K2 is shown in FIG.

  Next, the setting unit 23 normalizes each frequency component value in the absolute value sum K2 so that the frequency component value of the DC component in the absolute value sum K2 is equal to the frequency component value of the DC component in the absolute value sum K1. Thus, the normalized absolute value sum K3 is created. An example of the absolute value sum K3 is shown in FIG.

  Next, the setting unit 23 converts each frequency component value of the absolute value sum K1 so that each frequency component value in the absolute value sum K1 is equal to (including approaching) each frequency component value in the absolute value sum K3. A conversion value (multiplier) is calculated. That is, the setting unit 23 uses, as a correction value, a conversion value for converting each frequency component value in the input image so that the ratio of each frequency component value to the DC component in the input image is equal to the ratio in the ideal image. Set.

  Next, the setting unit 23 creates a scaling matrix SM (PIC) by describing the multiplier of each frequency component using a predetermined coefficient that substitutes each multiplier. An example of the scaling matrix SM (PIC) is shown in FIG. An example of the correspondence between the coefficient and the multiplier is shown in FIG. In the scaling matrix SM (PIC), the coefficient “16” corresponds to the multiplier “1” (that is, through), and the coefficient “255” corresponds to the minimum multiplier (that is, cut) having a substantially zero value.

  If an ideal image cannot be prepared, a scaling matrix SM (PIC) that causes the correction unit 13 to substantially function as a through filter is created by describing the coefficient “16” for all frequency components. Alternatively, a universally usable scaling matrix SM (PIC) may be prepared in advance, and when an ideal image cannot be prepared, the scaling matrix SM (PIC) may be set.

<Overlap processing>
In the image processing apparatus 1, the macroblock overlap processing is performed by extracting the next macroblock from the position shifted by 4 pixels from the 8 × 8 macroblock to obtain the image data D <b> 1. . The pixel shift width is not limited to four pixels, and an arbitrary shift width can be set. As the shift width becomes smaller, the processing time becomes longer, but the image quality improves.

  FIG. 9 is a diagram illustrating the overlap processing. In FIG. 9, an input image of 16 rows × 16 columns is assumed to simplify the description. First, the macro block MB1 is extracted starting from the upper left corner of the input image. Next, the macro block MB2 is extracted from a position shifted by 4 pixels in the row direction with respect to the macro block MB1. Next, the macro block MB3 is extracted from a position shifted by 4 pixels in the row direction with respect to the macro block MB2. Next, the macro block MB4 is extracted from a position shifted by 4 pixels in the column direction with respect to the macro block MB1. Thereafter, similarly, macroblocks MB5 to MB9 are sequentially extracted. Thereby, the transformation unit 11, the quantization unit 12, the correction unit 13, the inverse quantization unit 14, and the inverse transformation unit 15 relate to the plurality of macro blocks MB1 to MB9 extracted so as to partially overlap from the input image. The process is performed sequentially.

  For an image portion where a plurality of macroblocks overlap in the output image, an optimum value (in this example, an average value) obtained from the plurality of overlapping macroblocks is output as image data of the image portion. . For example, in the image portion of 4 rows × 4 columns with sandy hatching in FIG. 9, four macroblocks MB1, MB2, MB4, and MB5 are overlapped, and an image of each of these four macroblocks is displayed. Data D6 is obtained. Therefore, by adding the image data D6 of the corresponding part in these four macroblocks and dividing the added value by a plurality of “4” s, the average value of the image data D6 relating to a plurality of overlapping macroblocks is obtained. Calculated. Then, the average value is output from the image processing apparatus 1 as the image data D6 of the image portion.

  As a first modified example of the overlap processing, instead of the above average value, an output value (image data D6) related to a macroblock having a minimum noise value NOISE (MB) among a plurality of overlapping macroblocks is used. It may be used as the optimum value of.

  As a second modification of the overlap processing, for an image portion in which a plurality of macroblocks overlap, the output value of the image portion in the processing related to the current macroblock is used as the image portion in the processing related to the next macroblock. May be used as an input value. For example, the output value (image data D6) in the process related to the macroblock MB1 is changed to the input value (image data D1 in the process related to the macroblock MB2) for the image portion of 4 rows × 4 columns with sandy hatching in FIG. ), The output value in the process related to the macroblock MB2 is used as the input value in the process related to the macroblock MB4, and the output value in the process related to the macroblock MB4 is used as the input value in the process related to the macroblock MB5. use. According to this method, noise removal processing is performed substantially multiple times for the same image portion in a series of processes that sequentially process a plurality of macroblocks. Compared with the case where an average value of a plurality of output values is obtained. Thus, the processing time can be shortened.

<ACT (MB) calculation process>
The calculation unit 20 calculates the average value AVE_BLK in the macroblock of the weighted average value by performing the calculation represented by the following formula (4). Next, by calculating the following equation (5), the sum of absolute differences between the weighted average value GAUS (i, j) of each pixel and the in-block average value AVE_BLK is calculated. Then, an activity evaluation value ACT (MB) for each macroblock is calculated as an average value within the macroblock of the sum of absolute differences by performing the calculation represented by the following formula (6).

<First setting process of QP (MB)>
FIG. 10 is a diagram for explaining a first setting process of the quantization value QP (MB) by the setting unit 22. The setting unit 22 sets the quantized value QP (MB) based on the quantized value QP (PIC) and the activity evaluation value ACT (MB). The setting unit 22 holds a lookup table that describes the optimum quantized value QP (MB) for each activity evaluation value ACT (MB). As shown in FIGS. 10A and 10B, the setting unit 22 has a distribution range of the activity evaluation value ACT (MB) in the histogram of the activity evaluation value ACT (MB) and the number of macroblocks. , Normalize the lookup table. As shown in FIG. 10B, according to the lookup table, the higher the activity evaluation value ACT (MB), the lower the quantization value QP (MB) is set. Further, as illustrated in FIG. 10C, the setting unit 22 has an upper limit quantized value QPH that is larger than the quantized value QP (PIC) by a value ΔH around the quantized value QP (PIC), A lower limit quantized value QPL smaller than the quantized value QP (PIC) by a value ΔL is set. Then, the setting unit 22 is a macroblock in which the quantization value QP (MB) in FIG. 10B exceeds the upper limit quantization value QPH, that is, the activity evaluation value ACT (MB) is at a low level (less than the value A1). For the macroblock, the upper limit quantized value QPH is set as the quantized value QP (MB). 10B is a macroblock in which the quantized value QP (MB) is equal to or lower than the upper limit quantized value QPH and equal to or higher than the lower limit quantized value QPL, that is, the activity evaluation value ACT (MB) is at a medium level (value A1 or higher For the macroblock having the value A2 or less), the quantized value QP (MB) in FIG. 10B is set as the quantized value QP (MB) as it is. In addition, regarding a macroblock whose quantized value QP (MB) in FIG. 10B is lower than the lower limit quantized value QPL, that is, a macroblock whose activity evaluation value ACT (MB) is at a high level (value A2 exceeds), The lower limit quantized value QPL is set as the quantized value QP (MB).

<Second setting process of QP (MB)>
FIG. 11 is a diagram for explaining the second setting process of the quantized value QP (MB) by the setting unit 22. The setting unit 22 sets the quantized value QP (MB) based on the quantized value QP (PIC) and the noise value NOISE (MB). The setting unit 22 calculates the quantized value QP (MB) based on the noise value NOISE (MB) by performing the calculation represented by the following formula (7). As illustrated in FIG. 11, the setting unit 22 sets a higher quantized value QP (MB) as the noise value NOISE (MB) is higher.

  Further, the setting unit 22 uses the quantized value QP (PIC) as the upper limit value of the quantized value QP (MB), so that the quantized value QP (MB) calculated by the calculation of Expression (7) is the quantized value. When exceeding QP (PIC), the quantized value QP (PIC) is set as the quantized value QP (MB).

  FIG. 12 is a diagram illustrating a configuration of the image processing apparatus 1 for realizing the second setting process of the quantization value QP (MB). The noise value NOISE (MB) is input from the calculation unit 19 to the setting unit 22.

  In Expression (7), the quantized value QP (PIC) can be calculated by substituting the noise value NOISE (PIC) for the noise value NOISE (MB). Therefore, the setting unit 21 may set the quantized value QP (PIC) by calculation using Expression (7) instead of the lookup table shown in FIG.

<SM (MB) setting process>
13 to 15 are diagrams for explaining the setting process of the scaling matrix SM (MB) by the setting unit 24. The setting unit 24 sets the scaling matrix SM (MB) based on the scaling matrix SM (PIC) and the activity evaluation value ACT (MB).

  Referring to FIG. 13, setting unit 24 uses a low-level mask ML to set scaling matrix SM (PIC) for a macroblock whose activity evaluation value ACT (MB) is low (less than value A1). Mask. In the mask ML, a mask value that passes through the low-frequency component by a multiplier of “1” and cuts the medium-frequency component and high-frequency component by a multiplier of substantially zero value is set. As a result, the correction value of the scaling matrix SM (PIC) is set as the correction value of the scaling matrix SM (MB) for the low frequency component, and the substantially zero value is set for the scaling matrix SM (MB) for the medium frequency component and the high frequency component. Set as a correction value.

  Referring to FIG. 14, setting unit 24 uses scaling mask SM for medium level mask macro MM with respect to a macroblock whose activity evaluation value ACT (MB) is medium level (value A1 or more and value A2 or less). Mask (PIC). In the mask MM, a mask value that passes through the low-frequency component and the medium-frequency component by a multiplier of “1” and cuts the high-frequency component by a multiplier of substantially zero value is set. As a result, the correction value of the scaling matrix SM (PIC) is set as the correction value of the scaling matrix SM (MB) for the low frequency component and the medium frequency component, and the substantially zero value is set for the scaling matrix SM (MB) for the high frequency component. Set as a correction value.

  Referring to FIG. 15, setting unit 24 uses a high-level mask MH to set scaling matrix SM (PIC) for a macroblock whose activity evaluation value ACT (MB) is at a high level (value A2 exceeds). Mask. In the mask MH, a mask value that passes through a low frequency component, a medium frequency component, and a high frequency component is set by a multiplier of “1”. Thereby, the correction value of the scaling matrix SM (PIC) is set as the correction value of the scaling matrix SM (MB) in all of the low frequency component, the medium frequency component, and the high frequency component.

  In addition, although the example which performs the masking process of 3 steps | paragraphs using three types of masks ML, MM, and MH was demonstrated above, four or more types of masks are prepared according to variables, such as activity evaluation value ACT (MB). By doing so, you may perform the masking process of four steps or more (for example, eight steps). Further, the multipliers of the masks ML, MM, and MH shown in FIGS. 13 to 15 are examples, and are not limited to these examples. The multiplier set for each mask ML, MM, MH can be arbitrarily determined according to the amount of noise or the like.

<Necessity determination processing for noise removal processing>
The calculation unit 19 calculates a noise value NOISE (PIC) as an average value of the noise values NOISE (MB) regarding all macroblocks in the input image, and inputs the noise value NOISE (PIC) to the CPU 2. In addition, the arithmetic unit 20 calculates the activity evaluation value ACT (PIC) as an average value of the activity evaluation values ACT (MB) for all the macroblocks in the input image, and the activity evaluation value ACT (PIC). Is input to the CPU 2.

  Based on the noise value NOISE (PIC) of the entire input image and the activity evaluation value ACT (PIC) of the entire input image, the CPU 2 determines whether it is necessary to execute the noise removal process for each input image.

  The CPU 2 calculates the noise intensity by performing an operation (NOISE (PIC) / ACT (PIC)) for dividing the noise value NOISE (PIC) by the activity evaluation value ACT (PIC). Then, when the noise intensity is less than a predetermined threshold value (for example, “1”), it is determined that the noise removal process is unnecessary. When the noise intensity is equal to or higher than the threshold value, noise removal is performed. It is determined that execution of processing is necessary. As described above, the power consumption can be reduced by controlling ON / OFF of the execution of the noise removal processing in units of pictures.

  When the processing delay time is not a problem, for example, when the input image is a still image, the CPU 2 calculates the noise value NOISE (PIC) and the activity evaluation value ACT (PIC) of the input image itself, What is necessary is just to determine the necessity of execution of a noise removal process based on them.

  On the other hand, when the processing delay time becomes a problem, for example, when the input image is a moving image, the CPU 2 determines the noise value NOISE (PIC) and the activity evaluation value ACT (PIC) of the input image immediately before (one frame before). ) To determine whether or not it is necessary to perform noise removal processing on the current input image.

<First Modification>
In the above embodiment, the quantized value is set based on both the quantized value QP (PIC) and the quantized value QP (MB), and based on both the scaling matrix SM (PIC) and the scaling matrix SM (MB). However, it is also possible to set the quantization value based only on the quantization value QP (PIC) and set the correction value based only on the scaling matrix SM (PIC).

  FIG. 16 is a diagram illustrating a configuration of the image processing apparatus 1 according to the present modification. The setting units 22 and 24 are omitted from the configuration shown in FIG.

  FIG. 17 is a flowchart showing the flow of noise removal processing executed by the image processing apparatus 1 according to this modification. First, in step P201, the setting unit 17 sets the selector 16 based on the determination result of whether or not the noise removal processing by the CPU 2 is necessary. If execution is not necessary, the process ends.

  If execution is necessary, in step P202, the setting unit 21 sets a quantized value QP (PIC).

  Next, in step P203, the setting unit 23 sets a scaling matrix SM (PIC).

  Next, in steps P204 to P208, the noise removal processing for each macroblock is repeatedly executed for all the macroblocks included in the input image.

  In step P204, the conversion unit 11 converts the image data D1 into image data D2.

  Next, in step P205, the quantization unit 12 generates image data D3 from the image data D2 by quantization processing using the quantization value QP (PIC).

  Next, in Step P206, the correction unit 13 generates image data D4 from the image data D3 by correction processing using the scaling matrix SM (PIC).

  Next, in step P207, the inverse quantization unit 14 generates image data D5 from the image data D4 by inverse quantization processing.

  In step P208, the inverse conversion unit 15 converts the image data D5 into image data D6 by inverse DCT conversion. When the noise removal processing for each macro block is completed for all the macro blocks included in the input image, the processing ends.

<Second Modification>
In the above embodiment, the quantized value is set based on both the quantized value QP (PIC) and the quantized value QP (MB), and based on both the scaling matrix SM (PIC) and the scaling matrix SM (MB). However, it is also possible to set the quantization value based only on the quantization value QP (MB) and set the correction value based only on the scaling matrix SM (MB).

  FIG. 18 is a diagram illustrating a configuration of the image processing apparatus 1 according to the present modification. The setting units 21 and 23 are omitted from the configuration shown in FIG.

  FIG. 19 is a flowchart showing the flow of noise removal processing executed by the image processing apparatus 1 according to the present modification. First, in step P301, the setting unit 17 sets the selector 16 based on the determination result of whether or not the noise removal processing by the CPU 2 is necessary. If execution is not necessary, the process ends.

  Next, in steps P302 to P308, noise removal processing for each macroblock is repeatedly executed for all macroblocks included in the input image.

  In step P302, the computing unit 20 calculates the activity evaluation value ACT (MB).

  Next, in step P303, the setting unit 22 sets the quantization value QP (MB). The setting unit 24 sets a scaling matrix SM (MB). In the present modification, the setting unit 24 uses the low-level mask ML (FIG. 13) as the scaling matrix SM (MB) for the macroblock whose activity evaluation value ACT (MB) is low (less than the value A1). Use. Further, the setting unit 24 sets the mask MM (FIG. 14) for the medium level to the scaling matrix SM (MB) for the macro block whose activity evaluation value ACT (MB) is the medium level (value A1 or more and value A2 or less). Used as Further, the setting unit 24 uses the high-level mask MH (FIG. 15) as the scaling matrix SM (MB) for the macroblock whose activity evaluation value ACT (MB) is at a high level (value A2 exceeds).

  Next, in step P304, the conversion unit 11 converts the image data D1 into image data D2.

  Next, in step P305, the quantization unit 12 generates image data D3 from the image data D2 by quantization processing using the quantization value QP (MB).

  Next, in Step P306, the correction unit 13 generates image data D4 from the image data D3 by correction processing using the scaling matrix SM (MB).

  Next, in step P307, the inverse quantization unit 14 generates image data D5 from the image data D4 by inverse quantization processing.

  Next, in step P308, the inverse conversion unit 15 converts the image data D5 into image data D6 by inverse DCT conversion. When the noise removal processing for each macro block is completed for all the macro blocks included in the input image, the processing ends.

<Third Modification>
In the above embodiment, noise removal processing is performed independently for each color component of the input image, but integrated noise removal processing can also be performed for all (or some of a plurality of) color components.

  The arithmetic unit 19 calculates the average noise value NOISE (PIC) for all the color components by averaging the noise values NOISE (PIC) of the respective color components. The computing unit 20 calculates the average activity evaluation value ACT (PIC) for all color components by averaging the activity evaluation value ACT (PIC) of each color component.

  The arithmetic unit 19 calculates the average noise value NOISE (MB) for all the color components by averaging the noise values NOISE (MB) of the respective color components. The computing unit 20 calculates the average activity evaluation value ACT (MB) for all color components by averaging the activity evaluation value ACT (MB) of each color component.

  The common quantization value QP common to all color components set by using these average noise values NOISE (PIC), NOISE (MB) and average activity evaluation values ACT (PIC), ACT (MB). An integrated noise removal process is executed using (PIC), QP (MB) and common scaling matrices SM (PIC), SM (MB).

  Thus, noise detection accuracy can be improved by performing an integrated noise removal process on all (or some of a plurality of) color components. In addition, since the variation in the noise removal effect between the color components is suppressed, the image quality can be improved.

  On the other hand, by performing noise removal processing independently on each color component of the input image as in the above embodiment, an optimum noise removal effect can be obtained for each color component. The method of the above embodiment is effective when the amount of noise differs greatly between color components, such as when the amount of noise of only some color components is prominent.

<Fourth Modification>
In the above embodiment, an example in which the image processing apparatus 1 is configured by hardware such as a dedicated LSI has been described. However, the CPU 2 reads and executes a program recorded on a recording medium such as a ROM, whereby the image processing apparatus 1 is read. The CPU 2 may be configured to realize the same function by software processing.

<Summary>
According to the image processing device 1 according to the above embodiment and the image processing device 1 according to the second modification, the setting unit 22 (quantization value setting unit) has the activity evaluation value ACT (MB) for each macroblock. ), A quantization value QP (MB) for each macroblock used for the quantization process is set. Thus, by setting the quantization value QP (MB) for each macroblock based on the activity evaluation value ACT (MB) for each macroblock, an appropriate value for each macroblock is set according to the attribute of the macroblock. Noise removal processing can be executed. As a result, it is possible to effectively avoid a decrease in resolution, and it is possible to improve the image quality of the entire image. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.

  In addition, according to the image processing device 1 according to the above-described embodiment and the image processing device 1 according to the above-described second modification, the setting unit 24 (correction value setting unit) includes the activity evaluation value ACT ( Based on (MB), a scaling matrix SM (MB) (correction value) for each macroblock used for correction processing is set. In this way, by setting a correction value for each macroblock based on the activity evaluation value ACT (MB) for each macroblock, an appropriate correction process is executed for each macroblock according to the attribute of the macroblock. be able to. As a result, the noise removal effect for each block can be improved, so that the image quality of the entire image can be improved.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the calculation unit 20 calculates the absolute difference between the smooth value of each pixel (the weighted average value in the above example) and the average value of the smooth value in the macroblock. A value sum is calculated, and an activity evaluation value ACT (MB) for each macroblock is calculated as an average value in the macroblock of the sum of absolute differences. As a result, the activity evaluation value ACT (MB) for each macroblock can be appropriately obtained by the calculation unit 20.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the setting unit 22 sets a high quantized value QP (MB) for a macroblock with a low activity evaluation value ACT (MB). For flat portions where the activity evaluation value ACT (MB) is low and noise is conspicuous, by setting the quantization value QP (MB) high, it is possible to obtain a high noise removal effect by the quantization processing. The setting unit 22 sets a low quantization value QP (MB) for a macroblock with a high activity evaluation value ACT (MB). For edges and textures with high activity evaluation values ACT (MB), by setting the quantization value QP (MB) low, it is possible to avoid blurring of edges and textures due to quantization. The image quality can be improved. In addition, the setting unit 22 sets an intermediate quantized value QP (MB) for a macroblock having a medium activity evaluation value ACT (MB). For image portions other than the flat portion, the edge portion, and the texture portion, by setting the quantized value QP (MB) to a medium level, avoiding the noise removal effect on the image portion being excessive or small. Is possible.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the setting unit 22 uses the quantization value QP (MB) for each macroblock used for the quantization process based on the noise value NOISE (MB) for each macroblock. ) Is set. Thus, by setting the quantization value QP (MB) for each macroblock based on the noise value NOISE (MB) for each macroblock, an appropriate value for each macroblock is set according to the noise level for each macroblock. Noise removal processing can be executed. As a result, it is possible to effectively avoid a reduction in resolution, so that the image quality of the entire image can be improved. In addition, the processing time can be shortened and the circuit scale can be reduced because a temporal memory for storing intermediate values is not required, compared to the case where the spatial filter is applied multiple times to improve the noise removal effect. It becomes.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the setting unit 24 sets the correction value for each macroblock used for the correction processing based on the activity evaluation value ACT (MB) for each macroblock. . In this way, by setting a correction value for each macroblock based on the activity evaluation value ACT (MB) for each macroblock, an appropriate correction process is executed for each macroblock according to the attribute of the macroblock. be able to. As a result, since the noise removal effect for each macroblock can be improved, the image quality of the entire image can be improved.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the calculation unit 19 (first calculation unit) calculates the sum of absolute differences between the pixel value of each pixel and the smooth value of each pixel, A noise value NOISE (MB) for each macroblock is calculated as an average value within the macroblock of the sum of absolute differences. Thus, the noise value NOISE (MB) for each macroblock can be appropriately obtained by the calculation unit 19.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the setting unit 20 sets a higher quantization value QP (MB) as the noise value is higher. As described above, for a macroblock having a high noise value, by setting the quantization value QP (MB) high, a high noise removal effect by the quantization process can be obtained. In addition, for a macroblock having a low noise value, by setting the quantization value QP (MB) low, it is possible to avoid deterioration in image quality due to quantization processing.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the calculation unit 20 (second calculation unit) calculates the sum of absolute differences between the smooth value of each pixel and the average value of the smooth value in the macroblock. The activity evaluation value ACT (MB) for each macroblock is calculated as an average value within the macroblock of the sum of absolute differences. As a result, the activity evaluation value ACT (MB) for each macroblock can be appropriately obtained by the calculation unit 20.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the setting unit 24 holds a plurality of types of masks each having an arbitrary multiplier set for each frequency component according to the activity evaluation value ACT (MB). The multiplier set in the mask corresponding to the activity evaluation value ACT (MB) calculated by the calculation unit 20 is set as the correction value. As a result, a correction value corresponding to the activity evaluation value ACT (MB) for each macroblock can be set easily and appropriately.

  In addition, according to the image processing apparatus 1 according to the above-described embodiment, the setting unit 24 sets 1 time as a correction value for a low-frequency component for a macroblock with a low activity evaluation value ACT (MB). For the medium frequency component and the high frequency component, approximately zero times is set as the correction value. For a flat portion where the activity evaluation value ACT (MB) is low and noise is conspicuous, it is possible to obtain a high noise removal effect by correction processing by performing masking for cutting the medium frequency component and the high frequency component. Further, the setting unit 24 sets 1 time as a correction value for the low-frequency component, the medium-frequency component, and the high-frequency component for the macroblock having the high activity evaluation value ACT (MB). For edge portions and texture portions with high activity evaluation value ACT (MB), by performing masking that does not cut all frequency components, it is possible to realize correction processing that sharpens the edge portions and texture portions, and to improve image quality Is possible. The setting unit 24 sets a correction value of 1 for the low frequency component and the medium frequency component as a correction value for a block whose activity evaluation value ACT (MB) is at a medium level, and corrects approximately zero times for the high frequency component. Set as a value. For image portions other than the flat portion, the edge portion, and the texture portion, it is possible to obtain a moderate noise removal effect by the correction processing by performing masking that cuts only high-frequency components.

  Further, according to the image processing apparatus 1 according to the embodiment, the CPU 2 removes noise based on the noise value NOISE (PIC) of the entire input image and the activity evaluation value ACT (PIC) of the entire input image. The necessity of execution of the process is determined. As a result, noise removal processing can be executed for an input image that truly requires noise removal processing, and execution of noise removal processing on an unnecessary input image can be avoided, so that power consumption can be reduced.

  Further, according to the image processing apparatus 1 according to the above-described embodiment, the CPU 2 divides the noise value NOISE (PIC) of the entire input image by the activity evaluation value ACT (PIC) of the entire input image, thereby generating noise. Calculate the intensity. If the noise intensity is less than a predetermined threshold value, it is determined that the noise removal process is not necessary. If the noise intensity is equal to or greater than the threshold value, the noise removal process needs to be performed. judge. When the CPU 2 performs such determination processing, it is possible to specify an input image that truly requires noise removal processing with high accuracy.

  In addition, according to the image processing device 1 according to the above-described embodiment, the transform unit 11, the quantization unit 12, and the correction unit 13 sequentially perform a plurality of macro blocks extracted so as to partially overlap from the input image. To process. In this way, it is possible to suppress the occurrence of block noise by overlapping a part of macroblocks. For an image portion where a plurality of macroblocks overlap in the output image, an optimum value (average value in the above example) obtained from the plurality of overlapping macroblocks is output as image data of the image portion. As described above, by outputting the optimum values obtained from a plurality of macroblocks for overlapping portions, it is possible to achieve a high noise removal effect while avoiding deterioration in image quality.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 11 Conversion part 12 Quantization part 13 Correction | amendment part 17 Setting part 18 Filter 19,20 Calculation part 21-24 Setting part

Claims (21)

An image processing apparatus that generates an output image by performing noise removal processing on an input image,
A conversion unit that converts the first image data of the spatial domain extracted in units of predetermined blocks from the input image into second image data of the frequency domain including a plurality of frequency components;
A quantization unit that generates third image data by performing a quantization process on the second image data;
A calculation unit that calculates an activity evaluation value for each block based on the first image data;
Based on the activity evaluation value for each block, a quantization value setting unit for setting the quantization value for each block used for the quantization process,
A smoothing processing unit that calculates a smooth value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel;
With
The calculation unit calculates a sum of absolute differences between a smooth value of each pixel and an average value of the smooth values in the block, and calculates an activity evaluation value for each block as the average value of the difference sum in the block. An image processing apparatus.   An image processing apparatus that generates an output image by performing noise removal processing on an input image,
  A conversion unit that converts the first image data of the spatial domain extracted in units of predetermined blocks from the input image into second image data of the frequency domain including a plurality of frequency components;
  A quantization unit that generates third image data by performing a quantization process on the second image data;
  A calculation unit that calculates an activity evaluation value for each block based on the first image data;
  Based on the activity evaluation value for each block, a quantization value setting unit for setting the quantization value for each block used for the quantization process,
With
  The transform unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap from an input image,
  An image processing apparatus in which an optimum value obtained from a plurality of overlapping blocks is output as image data of the image portion for an image portion where a plurality of blocks overlap in an output image.   An image processing apparatus that generates an output image by performing noise removal processing on an input image,
  A conversion unit that converts the first image data of the spatial domain extracted in units of predetermined blocks from the input image into second image data of the frequency domain including a plurality of frequency components;
  A quantization unit that generates third image data by performing a quantization process on the second image data;
  A calculation unit that calculates an activity evaluation value for each block based on the first image data;
  Based on the activity evaluation value for each block, a quantization value setting unit for setting the quantization value for each block used for the quantization process,
With
  The transform unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap from an input image,
  For an image portion in which a plurality of blocks overlap in an output image, the output data of the image portion in the process related to the current block is used as input data of the image portion in the process related to the next macroblock .   A correction unit that generates fourth image data by executing correction processing for each frequency component on the third image data;
  Based on the activity evaluation value for each block, a correction value setting unit for setting a correction value for each block used for the correction process;
The image processing apparatus according to claim 1, further comprising:   The quantized value setting unit includes:
  For blocks with a low activity rating, set a high quantization value,
  For blocks with a medium activity rating, set a moderate quantization value,
  The image processing apparatus according to claim 1, wherein a low quantization value is set for a block having a high activity evaluation value.   The correction value setting unit
  Multiple types of masks with arbitrary multipliers set for each frequency component are stored according to the activity evaluation value.
  The image processing apparatus according to claim 4, wherein a multiplier set in a mask corresponding to the activity evaluation value calculated by the calculation unit is set as a correction value.   An image processing apparatus that generates an output image by performing noise removal processing on an input image,
  A conversion unit that converts the first image data of the spatial domain extracted in units of predetermined blocks from the input image into second image data of the frequency domain including a plurality of frequency components;
  A quantization unit that generates third image data by performing a quantization process on the second image data;
  A first calculation unit that calculates a noise value for each block based on the first image data;
  Based on the noise value for each block, a quantization value setting unit for setting the quantization value for each block used for the quantization process;
  A smoothing processing unit that calculates a smooth value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel;
With
  The first calculation unit calculates a sum of absolute differences between a pixel value of each pixel and a smooth value of each pixel, and calculates a noise value for each block as an average value in the block of the difference absolute value sum. , Image processing device.   An image processing apparatus that generates an output image by performing noise removal processing on an input image,
  A conversion unit that converts the first image data of the spatial domain extracted in units of predetermined blocks from the input image into second image data of the frequency domain including a plurality of frequency components;
  A quantization unit that generates third image data by performing a quantization process on the second image data;
  A first calculation unit that calculates a noise value for each block based on the first image data;
  Based on the noise value for each block, a quantization value setting unit for setting the quantization value for each block used for the quantization process;
With
  The transform unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap from an input image,
  An image processing apparatus in which an optimum value obtained from a plurality of overlapping blocks is output as image data of the image portion for an image portion where a plurality of blocks overlap in an output image.   An image processing apparatus that generates an output image by performing noise removal processing on an input image,
  A conversion unit that converts the first image data of the spatial domain extracted in units of predetermined blocks from the input image into second image data of the frequency domain including a plurality of frequency components;
  A quantization unit that generates third image data by performing a quantization process on the second image data;
  A first calculation unit that calculates a noise value for each block based on the first image data;
  Based on the noise value for each block, a quantization value setting unit for setting the quantization value for each block used for the quantization process;
With
  The transform unit and the quantization unit sequentially process a plurality of blocks extracted so as to partially overlap from an input image,
  For an image portion in which a plurality of blocks overlap in an output image, the output data of the image portion in the process related to the current block is used as input data of the image portion in the process related to the next macroblock .   A correction unit that generates fourth image data by executing correction processing for each frequency component on the third image data;
  A second computing unit that calculates an activity evaluation value for each block based on the first image data;
  Based on the activity evaluation value for each block, a correction value setting unit for setting a correction value for each block used for the correction process;
The image processing apparatus according to claim 7, further comprising:   The image processing apparatus according to claim 7, wherein the quantization value setting unit sets a higher quantization value as the noise value is higher.   A smoothing processing unit that calculates a smoothing value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel;
  The second calculation unit calculates a sum of absolute differences between the smoothed value of each pixel and the average value of the smoothed values within the block, and uses the difference absolute value sum as an average value within the block to evaluate the activity for each block. The image processing apparatus according to claim 10, wherein a value is calculated.   The correction value setting unit
  Multiple types of masks with arbitrary multipliers set for each frequency component are stored according to the activity evaluation value.
  The image processing apparatus according to claim 10, wherein a multiplier set in a mask corresponding to the activity evaluation value calculated by the second calculation unit is set as a correction value.   The correction value setting unit
  For blocks with a low activity evaluation value, set 1 times as the correction value for the low frequency component, and set almost zero times as the correction value for the medium frequency component and high frequency component,
  For blocks with a medium activity evaluation value, set 1 times as the correction value for the low frequency component and medium frequency component, and set approximately zero times as the correction value for the high frequency component,
  The image processing apparatus according to claim 6 or 13, wherein for a block having a high activity evaluation value, 1 time is set as a correction value for a low frequency component, a medium frequency component, and a high frequency component.   The determination part which determines the necessity of execution of a noise removal process further based on the noise value of the whole input image, and the activeness evaluation value of the whole input image is provided as described in any one of Claims 1-14. Image processing apparatus.   The determination unit calculates the noise intensity by dividing the noise value of the entire input image by the activity evaluation value of the entire input image. If the noise intensity is less than a predetermined threshold value, a noise removal process is performed. The image processing apparatus according to claim 15, wherein it is determined that the execution of is unnecessary, and if the noise intensity is greater than or equal to the threshold value, it is determined that the noise removal process is required.   The input image has a plurality of color components,
  The image processing apparatus according to claim 1, wherein the image processing apparatus performs a noise removal process independently on each color component.   An image processing apparatus that generates an output image by performing noise removal processing on an input image,
  A conversion unit that converts the first image data of the spatial domain extracted in units of predetermined blocks from the input image into second image data of the frequency domain including a plurality of frequency components;
  A quantization unit that generates third image data by performing a quantization process on the second image data;
  A calculation unit that calculates an activity evaluation value for each block based on the first image data;
  Based on the activity evaluation value for each block, a quantization value setting unit for setting the quantization value for each block used for the quantization process,
With
  The input image has a plurality of color components,
  The image processing apparatus is an image processing apparatus that performs noise removal processing on a plurality of color components in an integrated manner.   An image processing apparatus that generates an output image by performing noise removal processing on an input image,
  A conversion unit that converts the first image data of the spatial domain extracted in units of predetermined blocks from the input image into second image data of the frequency domain including a plurality of frequency components;
  A quantization unit that generates third image data by performing a quantization process on the second image data;
  A first calculation unit that calculates a noise value for each block based on the first image data;
  Based on the noise value for each block, a quantization value setting unit for setting the quantization value for each block used for the quantization process;
With
  The input image has a plurality of color components,
  The image processing apparatus is an image processing apparatus that performs noise removal processing on a plurality of color components in an integrated manner.   An image processing method for generating an output image by performing noise removal processing on an input image,
  Converting the first image data in the spatial domain extracted from the input image in units of predetermined blocks into the second image data in the frequency domain including a plurality of frequency components;
  Generating a third image data by performing a quantization process on the second image data;
  A step (C) of calculating an activity evaluation value for each block based on the first image data;
  A step (D) of setting a quantization value for each block to be used for the quantization processing based on the activity evaluation value for each block;
  Calculating a smooth value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel;
With
  The step (C) calculates the sum of absolute differences between the smoothed value of each pixel and the average value of the smoothed values in the block, and the activity evaluation value for each block is calculated as the average value of the difference absolute value in the block. An image processing method including a step of calculating.   An image processing method for generating an output image by performing noise removal processing on an input image,
  Converting the first image data in the spatial domain extracted from the input image in units of predetermined blocks into the second image data in the frequency domain including a plurality of frequency components;
  Generating a third image data by performing a quantization process on the second image data;
  Calculating a noise value for each block based on the first image data;
  A step (D) of setting a quantization value for each block used for the quantization process based on a noise value for each block;
  Calculating a smooth value of each pixel based on the pixel value of each pixel and the pixel values of a plurality of peripheral pixels adjacent to the pixel;
With
  The step (C) calculates a sum of absolute differences between the pixel value of each pixel and a smooth value of each pixel, and calculates a noise value for each block as an average value in the block of the difference absolute value sum. Including an image processing method.

Images