CN101669142A - A technique for adjusting the effect of applying a kernel to a signal to achieve the desired effect on the signal - Google Patents

Abstract

The invention discloses and a kind ofly be used to adjust the method and the device of core application in the effect of signal.Described adjustment can be desirably in the size of the effect that produces on the described signal matrix based on kernel matrix.Described adjustment can be based on multiple factor, include but not limited to signal data signal to noise ratio (S/N ratio), be used for the attribute of equipment (for example, camera lens) of acquired signal data or the tolerance that derives based on analyzing described signal data.Be that according to embodiment of the present invention image data processing purpose recovering may be owing to being used for the contrast that the attribute of optical device of acquisition of image data loses.

Description

A technique for adjusting the effect of applying a kernel to a signal to achieve the desired effect on the signal

technical field

The present invention relates to digital signal processing. In particular, embodiments of the invention relate to modifying the effect achieved when applying a kernel matrix to process signal data in order to achieve a desired effect.

Background technique

In digital imaging, deconvolution is sometimes used to create better quality images by correcting optical aberrations such as optical blur, optical distortion, chromatic aberration, curvature of field, etc. in imaging instruments. The mathematical operation that defines deconvolution is the same mathematical operation that is performed in convolution - the reason it is called deconvolution is in the context of correcting or compensating for the effects imposed on an image by convolution. In other words, using deconvolution - in this way - is based on the fact that when an image signal passes through an imaging instrument, the output signal can be described as the difference between the image signal and a 2D function called the point spread function (PSF). the convolution. Taking optical blur as an example, PSF describes optical blur, that is, how a single point of light is imaged by an optical system, in terms of the trajectory of a point source of electromagnetic radiation through an instrument. The result of PSF convolution is that the acquired image signal is blurrier (and may also be distorted, and/or contain chromatic aberration) than the object itself. To compensate for this blurring, a deconvolution process can be performed, in which the image signal is convolved with a 2D function whose goal is to produce an unblurred image. This 2D function is often the inverse of the PSF (or a variant of the inverse of the PSF), to the extent that it counteracts the blur introduced by the PSF.

2D deconvolution can be performed in the digital domain with software (SW) or hardware (HW) on the digital representation of the image. Kernel matrices are often used as 2D deconvolution filters, characterized by a desired frequency response. To compensate for PSF blur and enhance image contrast, this kernel matrix is convolved with the signal matrix (the 2D image). For the filter to enhance contrast, it should have a bandpass filter (BPF) or highpass filter (HPF) frequency response such that it matches the PSF in a way that enhances contrast in the desired frequencies to a specified level.

In addition to enhancing contrast in an image, the same convolution operation can be used to reduce noise in an image by applying a kernel with a low-pass frequency response (LPF) to the image, averaging the noise. When performing such denoising operations, care should be taken not to compromise the contrast of the signal.

Typically, the deconvolution process to enhance an image is part of the image signal processing (ISP) chain running in a digital camera (with SW or HW). This series of mathematical operations on the image converts it from a "RAW" format image (sometimes called a "NEF" image) output by a CMOS/CCD sensor (in e.g. "BAYER" format) to a final image that is viewed and saved (e.g., JPEG images, TIFF images, etc.). Within the ISP chain, the deconvolution filter may be applied to the image after performing demosaicing (color interpolation), or before demosaicing. The former approach makes the filter affect the maximum possible frequency in the image (pixel by pixel); however, in order to cover a large spatial support, the 2D kernel should have many coefficients. The latter method implies that the filter is applied while the image is still in BAYER format (before color interpolation). The latter approach has the advantage of covering a larger spatial support with fewer coefficients, but the filter affects only lower frequencies. Since for most optical designs, for deconvolution to be effective, the size of the kernel should be roughly the same size (in pixels) as the PSF, compensating for a large PSF requires a large enough space to support the deconvolution kernel, and thus requires Many coefficients (which means more memory and math calculations). Since if too small a kernel is applied to an image blurred by a large PSF, undesired artifacts may remain after processing, this may be the result of applying said deconvolution kernel to the BAYER image before the demosaicing process the main reason.

Noise may also be undesirably amplified when applying a deconvolution filter with a HPF/BPF frequency response to an image may result in an increase in contrast. This noise amplification is seldom noticed when the signal-to-noise ratio (S/N) is good, and is extremely noticeable under harsh lighting conditions that can result in low S/N. Unfortunately, applying the deconvolution kernel without considering the amount of noise already in the image may result in a noisier image.

Denoising algorithms may be employed to reduce unwanted noise amplification due to application of deconvolution kernels. However, applying a denoising algorithm after the deconvolution process has amplified the noise in the image may require performing strong de-noising, which may lead to unwanted data loss in areas of the image with fine details. Also, applying strong denoising still does not remove all the noise.

The approaches described in this section are ones that could be pursued, but not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that the approaches described in this section are prior art solely by virtue of their inclusion in this section.

Description of drawings

The invention is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings, in which like reference numerals refer to similar elements, and in which:

Figure 1a depicts the ISP chain without the digital autofocus box;

Figure 1b depicts an ISP chain with possible locations for digital autofocus blocks according to an embodiment;

Figure 1c depicts an ISP chain with another possible location for a digital autofocus block according to an embodiment;

2 depicts a block diagram illustrating an exemplary system for updating a kernel matrix and processing image data with the updated kernel matrix, according to an embodiment;

Figure 3 shows an exemplary equation for updating a core according to an embodiment;

Figure 4 depicts a system block diagram in which "shading indicators" are used to modify convolution cores;

Figure 5 depicts a diagram of an exemplary lens shadow profile;

FIG. 6 depicts an equation for updating cores based on shadow parameters, according to an embodiment;

Figure 7 depicts a system block diagram in which the core update logic in the convolution block uses a "metric" to modify the core, according to an embodiment;

Figure 8 shows an exemplary bayer image with red (R), blue (B) and green (G) pixels;

Figure 9a illustrates data for an image window;

Figure 9B illustrates a histogram of the data in Figure 9A;

Figure 10A illustrates a single peak histogram;

Figure 10B illustrates a bimodal histogram;

Figure 11 depicts an equation for updating a core based on a metric, according to an embodiment;

Figure 12 is a flowchart illustrating a process for resizing a core effect, according to an embodiment;

Figure 13 is a diagram of an exemplary mobile device on which embodiments of the present invention may be implemented;

Figure 14 is a diagram of an exemplary computer system on which embodiments of the present invention may be implemented;

15 is a graph depicting a graph for determining parameters to modify a kernel based on image features, according to an embodiment;

Figure 16 is a block diagram of a system for processing signals according to an embodiment;

Figure 17 depicts correcting motion blur according to an embodiment;

Figure 18 depicts correcting motion translation according to an embodiment;

19 is a block diagram depicting a system for adjusting the intensity of core processing based at least on shading parameters, according to an embodiment;

20 is a block diagram illustrating a system for adjusting the intensity of core processing based at least on characteristics of neighboring pixels, according to an embodiment;

Figure 21 depicts a technique in which a core is updated according to one or more parameters, according to an embodiment;

Figure 22 depicts a technique, in accordance with an embodiment, in which "scaling" is applied to the convolution result of an unmodified kernel;

Figure 23 depicts a technique for computing information metrics based on distance, according to an embodiment; and

Figure 24 depicts a separation gain function according to an embodiment.

Detailed ways

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention. It is apparent, however, that the invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

Embodiments are described herein according to the following outline:

1.0 Overview

2.0 Embodiment of image signal processing chain

3.0 Overview of Convolution

4.0 Overview of Adjusting Core Effects Based on Expected Effect Size

4.1 Process overview

4.2 Overview of adjusting core effects by updating the core

4.3 Overview of adjusting core effects by modifying convolution results

5.0 Adjust core effects based on multiple factors

5.1 Adjust core effects based on S/N graphics

5.2 Adjust the core effect according to the spatial position

5.3 Adjust the core effect according to the local image characteristics

6.0 Adjust Core Effects to handle motion blur depending on implementation

6.1 Overview of Correcting Motion Blur

6.2 Implementation of fuzzy point distribution function

6.3 Motion shift vector implementation

6.4 Combining sharpening and motion blur cores

8.0 Hardware Overview

8.1 Mobile Device Embodiment

8.2 Computer System Embodiment

1.0 Overview

Techniques for processing signal data are disclosed herein. The signal data may be represented by a number of different signal matrices, each matrix representing a different portion of the signal. For example, each signal matrix may include elements corresponding to a group of pixels in the image sensor. However, the signal may be eg audio data or other signal data instead of image data. A given signal matrix is processed according to at least one core matrix to achieve some desired effect. More particularly, the size of the desired effect is adjusted in order to obtain the desired effect of said size, said desired effect being obtained due to processing said signal matrix according to the kernel matrix.

In one embodiment, before applying the modified kernel matrix to the signal data, the kernel matrix applied to the signal data is modified one or more times according to one or more parameters that will be listed in subsequent sections. Revise. Modifying the kernel matrix according to the one or more parameters allows to achieve the desired effect of the size. In another embodiment, the final processed result of a certain pixel may be the result of the deconvolution using the unmodified kernel and some other value(s), including but not limited to the original pixel value in the input image. weighted sum. The weight of each summed value can be modified according to several parameters that will be listed in subsequent sections. The modification may be based on factors including, but not limited to, the signal-to-noise ratio of the signal data, properties of the equipment used to acquire the signal data (e.g., lens), spatially related information (e.g., pixel location), or based on the signal data The obtained metrics (eg, pixels in the neighborhood of the pixel currently being processed) are analyzed.

Processing image data according to the "sharpening" embodiment of the present invention aims at restoring lost contrast. Contrast may be lost, for example, due to the properties of the optics used to acquire the image data. However, this sharpening implementation does not require the use of any particular lens. The purpose of the sharpening implementation described is to restore some of the image contrast lost due to the optics properties of the lens, so that the final image has good contrast for both the image of a distant object and the image of a near object, although Paid a reasonable S/N price. The sharpening implementation may be implemented as an image signal processing (ISP) module using hardware, software, or some combination of hardware and software.

In one embodiment, the core used has a high-pass or band-pass frequency response, which allows the core to sharpen the image by enhancing differences in the image. Examples of such differences include edges, shapes, contours, etc. between dark/light areas. However, applying these kernels on relatively uniform surfaces in an image (eg, such as a gray wall or blue sky) can result in amplifying small differences that exist between these pixels. These small differences between pixels can be caused by noise (random noise or fixed pattern noise). Therefore, applying these kernels may lead to undesired noise amplification in "flat", uniform, non-informative regions of the image. This unexpected noise can be especially severe in low light conditions. Several techniques are disclosed herein that avoid amplifying noise in "flat" areas of the image, while preserving contrast in desired areas.

In one embodiment, each core's gain (eg, the amount of sharpening they provide) is adjusted according to the S/N graph. Thus, when processing an image or a portion thereof with a poor S/N, the gain of the core may be reduced. As a specific example, the gain of the core may be reduced when processing images acquired in very low light. In one embodiment, the gain of the kernel is adjusted by modifying the kernel according to the S/N parameter and then applying the modified kernel to the image. In another embodiment, scaling is performed according to the S/N parameter after applying the unmodified kernel to the image.

In one embodiment, the gain of the kernel is adjusted according to the spatial location of the current portion of the image being processed. In one embodiment, the spatial position dependent adjustment compensates for properties of the lens used to capture the image data. Autofocus and standard lenses may suffer from radially reduced illumination profiles. That is, the amount of light passing through the lens is much lower in the high regions of the lens than in the middle of the lens. Therefore, S/N may be lower at the image borders compared to the center. In some designs, the lighting at the border is about 50-55% of the value in the middle area. However, the illumination at the border may be greater or less than 50-55%. In one embodiment, the gain of the kernel is adjusted by modifying the kernel according to shading parameters, and then applying the modified kernel to the image. In another implementation, the scaling is performed according to the shadowing parameters after applying the unmodified kernel to the image. Therefore, according to the embodiment of the present invention, a better S/N is obtained at the image boundary, and a better-looking image is obtained.

In one embodiment, the effect of applying the kernel is modified according to a metric describing some characteristic of the image in the neighborhood of the pixel currently being processed. In one embodiment, the metric is based on a statistical analysis of pixels in the neighborhood of the current pixel. For example, the statistical analysis may be based on the standard deviation between the elements in the image data matrix, the histogram of the elements in the image matrix, or the entropy among the elements in the image matrix. However, the metric can also be calculated based on gradients between specific pixels in the neighborhood of the current pixel. In one embodiment, the core is modified according to the metric. The modified kernel is then applied to the image data. In another embodiment, scaling is performed according to the metric after applying the unmodified kernel to the image.

In some applications, the core may often be used to offset the PSF. That is, the kernel may be the inverse of PSF representing optical aberrations such as blurring in the device that acquired the signal. However, it is not necessary that the core be the inverse of PSF. Additionally, there are embodiments herein where the core is used to enhance image contrast. However, it is not required that the methods described herein be applied to the core to enhance image contrast. Additionally, there are embodiments herein where the cores are used for image processing. More generally, however, the techniques described herein can be applied to a core for any digital signal processing application.

In one embodiment, the same mechanism used to sharpen the image (contrast enhancement) is also used to reduce or eliminate the effect of motion blur or shake in the image. Using the same mechanism for motion blur correction and contrast enhancement results in a better solution based on silicon size, processor cycles, complexity and cost.

2.0 Image Signal Processing Chain Embodiment

In a digital still camera module, the set of algorithms that process the raw image and output the final image (e.g., JPEG image, TIFFany image, etc.) stored in non-volatile memory is called the image signal processing chain, or ISP chain. FIG. 1 a depicts typical stages that occur in an ISP chain 100 . The ISP chain can have other stages and does not have to have all the stages depicted in Figure 1a. The stages can be in a different order.

In one embodiment, digital autofocus is implemented as a stand-alone block that acts as part of the ISP chain. The example ISP chain 100 shows a typical ISP chain without a digital autofocus box. The embodiment ISP chain 100 includes the following stages. Analog gain 101 applies the gain before A/D conversion to compensate for dark images (eg while employing low exposure times). Black level correction 102 removes the lowest values (ie, the "dark current" output from the sensor) from the image. Bad pixel correction 106 locates "burnt" pixels in the image that produce false extremely low or extremely high values, and replaces these pixels with "smart" average values for the environment. In a CMOS sensor with BAYER mode, the Gr/Gb compensation 107 compensates for the difference that may occur between the green pixel value in line with the blue pixel and the green pixel value in line with the red pixel. It is not necessary for the image sensor to use the BAYER format. To balance colors and compensate for lighting conditions and light levels, white balance and digital gain 108 applies gain to each color (green-red, red, green-blue, blue) while maintaining dynamic range.

Lens shading correction 109 compensates for the lens shading profile by enhancing the brightness of pixels far from the center. Noise removal 110 removes random noise and fixed pattern noise from the image. Color interpolation (demosaicing) 112 interpolates the BAYER image into a full RGB image where each pixel has three color channel values.

Sharpening 114 enhances the contrast in the image. As will be described in more detail later, the digital autofocus box is distinct from the sharpening box, where the digital autofocus compensates for the PSF.

The color correction matrix 115 is responsible for the color accuracy and color saturation of the image. Gamma correction 116 applies a gamma (γ) curve to the image. JPEG compression 118 compresses the image from the full BMP image into a JPEG image, which can then be saved to non-volatile memory such as flash memory. Compression other than JPEG may be employed.

There are several possible positions for the digital autofocus box to be located in the ISP chain. Figure 1b depicts an ISP chain 100a incorporating a digital autofocus block 111 according to an embodiment. In the ISP chain 100a, for example, the digital autofocus block 111 is located just before the demosaic block 112, ie it is applied on the BAYER image. Figure 1c depicts an ISP chain 100b which also incorporates a digital autofocus block 111 according to an embodiment. In the ISP chain 100b, the digital autofocus block 111 is located after the demosaicing block 112, ie it is applied on the RGB image. In both the ISP 100a and ISP 100b cases, it is not necessary that the digital autofocus box be located exactly as depicted - it could be located anywhere in the ISP. In one embodiment, the digital autofocus block 111 occurs after the bad pixel correction 106 because the digital autofocus 111 can enhance the effect of bad pixels. In another embodiment, the recommended digital autofocus box appears in the bayer denoising box 110, lens shading correction box 109, white balance box 108, Gr/Gb compensation box 107, bad pixel correction box 106 and black level correction block 102, because these blocks have corrected for sensor and some lens failures. Note that incorporating the digital autofocus block into the ISP may render the sharpening block 114 unnecessary.

The ISP chain 100 as a whole can provide good quality images for low resolution screens without even having to apply the digital autofocus 111 stage. Thus, it is not necessary to activate the digital autofocus 111 in preview mode (where the image is displayed on the phone's preview screen before it is captured).

In one embodiment of the algorithm, the camera module system that captures the image includes a macro/normal feature that allows the user to choose whether to capture a close-up image (e.g., 20-40 cm) or a long-distance image (e.g., , 40cm-infinity). In one embodiment, a mode selection is input into the digital autofocus 111 and can be used to alter the process. For example, different deconvolution kernels may be used, or the kernel may be changed according to the shooting mode. Other parameters from other stages of the ISP chain 100 may also be input into the digital autofocus stage 111 . As an example, a further stage provides a digital autofocus stage 111 with an S/N pattern of the image. The digital autofocus stage 111 adjusts the amount of sharpening according to the S/N graph. In one embodiment, the amount of sharpening is based on pixel location, which may be provided by another stage in the ISP chain 100 . However, it is not necessary to use normal/macro mode to choose between the different cores - a single set of cores can support both close and far distances.

3.0 Deconvolution Overview

In one embodiment, the core is used to perform bayer deconvolution. However, deconvolution is only one example of how a core may be used. A bayer image can be shaped as a 2D matrix of pixel values such that each cell value has its own color—red, green, or blue—according to the color filter on the sensor pixel that each cell value represents. FIG. 8 shows an exemplary bayer image 800 having red (R), blue (B) and green (G) color pixels. The position of the pixel indicates the position of the pixel in the image data convolved with the kernel of that color. Each color has a central pixel 801g, 801r, 801b which identifies said pixel in the image signal whose pixel value is modified by convolving the appropriate color kernel matrix with the corresponding pixel in the image signal.

According to one embodiment, bayer convolution uses three 2D kernels - one for each pixel color - to define the pixels involved in the convolution and the coefficient values to multiply each pixel with. Kernel size may be defined in terms of support, which refers to the number of pixels in the signal image that the kernel will "cover" when the kernel is placed on the image. Referring to Figure 8, the core supported size is 9x9. The core can support many other sizes. In another embodiment, the green pixels are divided into green pixels collocated with blue pixels (G _b ) and green pixels collocated with red pixels (G _r ). In such an embodiment, there are two green cores, one for _Gb and one for _Gr , which are of similar construction to the red or blue pixel cores.

Note that in some implementations, the convolution does not mix different colors. That is, in some embodiments, each central pixel is convolved with surrounding pixels of the same color. However, in other implementations, the convolution does mix colors. For example, when processing a red center pixel, information from blue and/or green pixels can be used in deconvolution. Thus, the kernel matrix for a given color may have non-zero coefficients at locations corresponding to pixels of a different color.

The coefficients of the kernel are multiplied by the corresponding pixels in the acquired image, they are all summed, and the result is written in the appropriate location in the output image. Those corresponding pixels in the image are referred to herein as the "current image window." For example, the core in Figure 8 supports a 9x9 pixel environment. In Figure 8, both red and blue pixels form a 5x5 matrix. When processing the central red pixel 801r, the red pixels surrounding the central red pixel 801r in the captured image are the current image window. Thus, in this embodiment, the current image window only includes pixels of a single color. However, for some types of processing, the current image window may have pixels of more than one color - for example, if the core mixes colors - i.e. contains coefficients multiplied with pixels of a different color than the central pixel on which the processing is performed - then they The image window for the operation must also include these pixels. Note that there are 5x5 patterns and 4x4 patterns for green pixels for a total of 41 pixels. The central pixel of each pattern is the pixel whose value is updated by convolution. In this embodiment, a bayer convolution is used, where for each color only pixels of the same color are used. Thus, the red pattern is used to update the red center pixel 801r from the red center pixel and the neighboring red pixels in the pattern.

Table I depicts the coefficients of an exemplary bayer filter for convolving green pixels with a 3x3 support core. When the center coefficient of this kernel falls on a green pixel, only the green pixel is convolved. Note that it is not necessary to actually perform multiplication on kernel coefficients with a value of 0.

Table I

For example, if the above-described bayer filter were to be applied to the appropriate portion of Figure 8, then the green center pixel 801g and the four nearest green pixels participate in the convolution. Convolution can act as a high-pass filter, or a band-pass filter. However, convolution can also act as a low-pass filter, or another function.

Note that the sum of the core coefficients in Table I is equal to 1. If the coefficients sum to a value other than 1, the kernel will affect the "DC" value of the image (eg, when all participating pixels are the same value), which can affect color or intensity. When applying a core to a surface of uniform color, it may be desirable to preserve the color (or intensity).

When RGB deconvolution is used (i.e. deconvolution is performed after demosaicing), all pixels have 3 values (R, G, B), and thus the kernel for each color can be written as a 2D matrix, Each matrix cell has valid coefficients - because the information for each color in the image is spatially continuous and exists in each pixel. It is convenient to think of an RGB image as a composition of 3 2D matrices - one for red, one for green and a last matrix for blue. In cases where no color blending is performed, each kernel is applied to the 2D matrix containing the colors appropriate for that kernel. However, in one embodiment, the kernel can mix colors, and thus the kernel can have values multiplied with pixels in all 3 R, G and B image matrices.

In one embodiment, a check is made to ensure that there is no overflow or underflow with respect to the bit resolution of the output image. If present, saturation and clipping may be used. In one embodiment, the bit resolution of the output image is the same as that of the input image. However, this is not required. In another embodiment, the output image resolution may be changed relative to the input image resolution. The order in which the pixels are processed is not limited to any particular order. The order may depend on the input interface of the sharpening stage 114 .

4.0 A review of adjusting core effects based on the size of the expected effect

4.1 Process overview

FIG. 12 is a flowchart illustrating a process 1200 for adjusting core effects, according to an embodiment. At step 1202, an initial kernel matrix is accessed. In step 1204, the processing effect of the kernel matrix on the signal matrix is adjusted. The purpose of this adjustment is to modify the magnitude of the effect the core has on the signal. Figures 21 and 22 depict two techniques for adjusting the effect that the kernel matrix has on the signal matrix.

The intended effect might be to enhance image contrast, although it might be another effect. The adjustment of the magnitude of the effect may be based on the S/N pattern of the signal. The adjustment of the size of the effect may be based on the position of the pixel. The adjustment of the size of the effect may be based on properties of the lens used to acquire the signal (eg, lens shading). The adjustment to effect size may be based on characteristics of the data in the neighborhood of the data being processed. For example, the characteristic may be statistical in nature, such as the standard deviation of the data. The core effect can be adjusted by any number of these factors, or by other factors.

4.2 Overview of adjusting core effects by updating cores

FIG. 21 depicts a technique in which a core is updated according to one or more parameters, according to an embodiment. Then, this updated kernel is used for convolution. In FIG. 21 , input interface 2202 receives an image signal from another stage in the ISP chain and passes the image pixels into update core block 2106 . In such an embodiment, the input interface 2206 provides certain data from the ISP chain to the update core block 2106. Examples of this data include, but are not limited to, S/N graphs, image histograms, exposure times, digital to analog gains, denoising information, white balance information, and pixel locations.

The update kernel block 216 updates the kernel 215 used for the convolution of the signal matrix formed by the selected image pixels. Herein, various embodiment parameters are discussed, which are based on factors such as S/N ("α"), spatial location ("β"), local characteristics ("γ"), and the like. The core may be updated according to one or more of these parameters, or other parameters. Thus, the core update is not limited to these parameters. Convolution block 2206 convolves the signal matrix (based on image pixels) with the updated kernel and final result.

The method used to modify the core with a computed metric (e.g. 'α', 'β', 'γ', ...) is as follows - once the metric is computed, it is in the dynamic range of 0 to 1 (If not, it can be converted to such a dynamic range), the effect of the core (eg, a 3x3 core) can be reduced using the following update formula:

This formula creates the following desired effect: if the metric is close to 0, reduce the effect of the core, and if the metric is close to 1, keep the original core. The final core is a linear combination of 2 cores - the initial core and the kronecker-delta core. The convolution of the kronecker-delta kernel with the image remains the same without modification. Thus, combining it with an initial kernel, relying on the metric when the coefficients multiplied by the kernel add up to 1, ensures that the effect of the initial kernel is reduced in strength while ensuring that the DC frequency components of the image are not Any change (since the sum of all coefficients in the final core remains 1).

In another embodiment, the kronecker-delta core is replaced by another core, for example a core with a stronger effect than the original core, and thus, depending on said measure, an effect of increasing the potency of said treatment may be achieved.

4.3 Overview of Adjusting Core Effects by Modifying Convolution Results

Figure 22 depicts a technique, according to an embodiment, in which "scaling" is applied to the convolution result of the kernel without modification. This scaling achieves the same result as updating the core as depicted in FIG. 21 . In FIG. 22 , input interface 2202 receives an image signal from another stage of the ISP chain and passes the image pixels into convolution block 2206 . The convolution block 2206 convolves the signal matrix (based on image pixels) with the unmodified kernel 215 and outputs the convolution result. Convolution block 2206 convolves the current pixel being processed with the original kernel and outputs the result into block 2212 .

The averager block 2212 receives the convolution result from block 2212, the current pixel value before convolution, and other data from the ISP. Examples of this data include, but are not limited to, S/N graphs, image histograms, exposure times, digital-to-analog gains, denoising information, white balance information, and pixel locations, and calculate the required update metrics (alternatively, based on The updated metrics for this information were previously calculated and they were only fed into the averager).

Then the averager 2212 takes an average between the convolution result and the pixel value before convolution, and outputs the final result (FinalRsult) using the following formula in one embodiment.

Equation 1: FinalRsult = ConvRs·metric+(1-metric)·InputPixel

In Equation 1, ConvRs is the convolution result, and InputPixel is the initial value of the current pixel being processed. In one embodiment, InputPixel is used since it is the convolution result of the kronecker-delta kernel and the signal matrix. Where the desired effect is to augment the effect of the initial kernel, replace the InputPixel with the result of convolution with a stronger kernel than the one used to create the ConvRs. The metric can be any update metric with a dynamic range between 0 and 1, and the update metric is constructed such that when it is close to 0, the desired effect is to reduce the effect of the first addend in the equation and The effect of the second addend is enhanced, and if it is close to 1, the opposite (in the final result, the effect of the first addend is enhanced and the effect of the second addend is reduced). Multiple metrics can be used to create the final output, and more than two addends can be used in the equation as long as the multiplication factor adds up to 1. Herein, various embodiments of metrics are discussed that are based on factors such as S/N ("α"), spatial location ("β"), local features ("γ"), and the like. The end result may depend on one or more of these factors, or other factors.

5.0 adjusts core effects based on multiple factors

5.1 Adjust core effects based on S/N graphics

System embodiment for updating the core

FIG. 2 depicts a block diagram illustrating an exemplary system 200 for processing signal matrices to achieve a desired amount of processing based on a kernel matrix. Figure 2 depicts updating a kernel matrix and processing image data with the updated kernel matrix, according to an embodiment. As described anywhere else herein, it is not necessary to modify the kernel itself in order to adjust the effect of applying the kernel. In one embodiment, scaling is performed after processing the image with an unmodified kernel in order to adjust the processing effect according to the kernel. Thus, it is to be understood that the various blocks in FIG. 2 are depicted for ease of illustration, and updating the core does not require multiplying the core by α as depicted in FIG. 2 .

In general, the update kernel block 202 updates the selected kernel matrix according to the noise level estimator. The convolution block 206 applies the updated kernel to the image data, which is output into the output interface 209 . In one embodiment, the update kernel block 202 and the convolution block 206 form part of the autofocus block 111 of Figure 1b or 1c. In one embodiment, the upstream portion of the ISP chain 204a provides the following parameters to the input interface 208:

i. Shooting mode.

ii. Noise level estimator.

iii. Input image.

In one embodiment, capture mode is used to determine which set of cores 215a, 215b to use. The exemplary system of FIG. 2 depicts cores for macro mode and normal mode. However, there may be cores for other modes such as night, landscape, and the like. A capture mode can be any type of scene that requires special handling of the kernel. In one embodiment, the capture mode is based on user selection. For example, the user can choose between macro mode and normal mode. In another embodiment, the shooting mode is determined automatically. In such an embodiment, either the macro kernel 215a or the normal kernel 215b is provided as input to the update initial kernel block 202 based on the capture mode. In one embodiment, the coefficients of the kernel are stored in memory. For example, 16-bit signed resolution can be used to represent the coefficients.

For example, the normal mode kernel 215b can be used for images acquired between 40cm-infinity. The macro mode core 215a can be used to photograph close-up elements. For example, the near-distance element may be in the range of 20cm-40cm, or in the closer range of 10cm-40cm. In one embodiment, there are two sets of three-color cores (red, green, blue): one set for normal mode and one set for macro mode. Different numbers of color cores can be used. There may also be different core sizes. Additionally, the supported dimensions employed may be based on pixel dimensions. For example, if the pixel size is 1.75um, the core support size might be 13x13. For 2.2um pixel size, the core support size may be 9x9. However, it is not necessary to use a larger support size for a smaller pixel size.

Noise Level Estimator is an estimator of the noise level in the acquired image. Noise level estimators can be represented in arbitrary formats. For example, the noise level estimator may be an integer value in the range 0-7, where 0 indicates low noise (good S/N) and 7 indicates a lot of noise (bad S/N). Bad S/N may be associated with poor lighting conditions.

In one embodiment, the upstream portion of the ISP chain 204a applies bad pixel correction (BPC) to the image data before providing the image pixel data as input. In one embodiment, the downstream portion of the ISP chain 204b applies demosaicing to the image data after the image data has been processed by the convolution block 206 .

The update core block 202 possesses alpha calculation logic 212 to calculate a value referred to herein as "alpha" from the noise level estimator. In one embodiment, where the noise level estimator is a number between 0 and the maximum noise value (Max_noise_val), the value of α is calculated using Equation 2.

Equation 2: α=(Max_noise_val-noise_val)/Max_noise_val

Equation 2 indicates that the larger the image noise (the larger the noise value (noise_val)), the lower the value of α (and vice versa). Therefore, if there is almost no noise in the image, the value of α is close to 1. In Equation 2, the relationship between α and noise_val is linear. However, a linear relationship is not required. Depending on the relationship between noise_val and the true amount of noise in the image, α may depend on noise_val in more complex forms, such as quadratic correlations, or other correlations - if noise_val is linear, then α may be a linear function of noise_val. The update core block 202 has core update logic 214 that generates an updated version of the original core based on the value of α.

FIG. 3 illustrates an exemplary equation for updating a core, according to an embodiment. Figure 3 indicates that the α-updated kernel matrix is a linear combination of two kernels: 1) the initial kernel matrix, which is multiplied by α; and 2) the delta kernel matrix, which is multiplied by 1-α. Delta cores may also be referred to as kronecker-delta cores. Note that this causes the sum of the α-update kernel coefficients to remain at 1, thus preserving the "DC" value of the image. Note that implementing the equation in Figure 3 does not require the deconvolution kernel to be multiplied by α. Conversely, as discussed below, after convolving the image matrix with the unmodified deconvolution matrix, alpha "scaling" can be performed to produce the same result.

If α is relatively close to 1, the α-updated kernel closely approximates the original kernel. However, if the value of α is closer to 0 than to 1, the α-update kernel will be more similar to the kronecker-delta kernel. For example, if α = 0.2, the α-update cores will include 80% of the kronecker-delta cores and only 20% of the original cores. In one embodiment, the initial core is essentially a high-pass or band-pass filter. Thus, if α is low, the α-update kernel will produce a much lower amount of sharpening to the image than the original kernel.

Note that if the core is a high-pass or band-pass filter, it may amplify noise; therefore, attenuate such a core when the image has a low S/N without updating it, compared to processing the image with the original core The S/N in the final processed image can be improved. This technique reduces the gain of the initial core. The same technique can be used to strengthen the original core by replacing the kronecker-delta core with a core with a stronger high-pass/band-pass frequency response than the original core. Convolution logic 216 in convolution block 206 convolves the current portion of the input image with an α-update kernel and outputs a value to output interface 209 . In one embodiment, the output image has the same resolution as the input image.

Exemplary system using post-convolution scaling

As discussed earlier, it is not necessary for the core to multiply by α to achieve an α update. Figure 16 depicts an embodiment where "scaling" is applied to the convolution result of an unmodified kernel; this scaling achieves the same result as updating the kernel with α as depicted in Figure 2 . In FIG. 16 , image interface 208 receives image signals from image sensor 1372 and passes the signal matrix into convolution block 206 . The mode select signal from the graphics interface 208 is used to select one or more kernels 215 , which are provided to the convolution block 206 . Embodiments of different modes have been discussed above. The convolution block 206 convolves the signal matrix with the unmodified kernel and outputs the convolution result. Each convolution result includes the value of one of the elements of the signal matrix currently being processed.

In such an embodiment, the graphics interface 208 provides the noise level estimator signal to the scaling 1602 , which utilizes the alpha calculation logic 212 to calculate “α”. The alpha calculation may be performed in the same manner as the embodiment depicted in FIG. 2 . Alpha scaling 1612 inputs the convolution result, alpha, and the initial value of the pixel currently being processed, and outputs the final result for the current pixel. In one embodiment, the current pixel is the center pixel of the signal matrix. In one embodiment, alpha scaling 1612 is performed as follows:

Equation 3: FinalRsult = ConvRs·α+(1-α)·InputPixel

In Equation 3, ConvRs is the convolution result, and InputPixel is the initial value of the current pixel being processed. Thus, with the unmodified kernel and one or more other values, the final processed result for the current pixel is a weighted sum of the convolution results. Those other values include, but are not limited to, the original pixel values in the input image. In such an embodiment, the alpha parameter is used as a weighting (or scaling) parameter.

5.2 Adjust the core effect according to the spatial position

exemplary system

In one embodiment, the effect of applying the deconvolution kernel to the image is modified according to the spatial location. In one embodiment, the spatial location is based on the shadow profile of the lens used to capture the image data. 4 depicts a block diagram of a system 400 in which β-update logic 304 in convolution block 203 uses a "shading indicator" to modify α after the α-update convolution kernel has been modified by update-core block 202. - Update convolution cores. The shading indicator defines the amount of shot-shading and can be described relative to the center in the image data. A beta-update can be performed on the core before or without an alpha-update on the core, if desired. As described anywhere else herein, it is not necessary to modify the kernel itself in order to adjust the effect of applying the kernel. In one embodiment, scaling is performed after processing the image with an unmodified kernel in order to adjust the processing effect according to the kernel. Herein, the adjustment based on the shaded indicator is referred to as "β-adjustment".

FIG. 6 depicts an equation for updating kernels based on shadow parameters, according to an embodiment. Note that the equations in Figure 6 describe a β update on an α-updated core. The initial core coefficients are marked with a label to indicate that the initial core was updated with α. As discussed earlier, a beta update may be performed on the core prior to an alpha update. Additionally, in one embodiment, a β-update may be performed without performing an α-update. If a β-update is performed on the kernel, the convolution is a non-linear convolution (ie, a convolution in which the kernel changes for each pixel depending on more than one condition). Note that implementing the equations in Figure 6 does not require the alpha-updated deconvolution kernel to be multiplied by β. Conversely, as discussed below, alpha and beta "scaling" may occur after convolving the unmodified deconvolution matrix with the image matrix.

After performing a beta update on the associated color kernel, convolution logic 216 convolves the associated color kernel with the image data represented as an image matrix. The relevant color kernel is determined based on the color of the central pixel in the image matrix, which is the current pixel being processed. Convolution involves multiplying the β-updated kernel coefficients by matching pixels in the image matrix. The result is written to the appropriate location in the output image, updating the value of the current pixel. Then, the next pixel in the image data is processed. For example, if bayer processing is being performed, the next pixel in the bayer image is processed. Beta-updating the core may provide better S/N; however, there may be less image sharpening at image boundaries.

Examples of beta and shaded contours

In one embodiment, the β-update attenuates the kernel as a function of distance from the image center in order to compensate for the S/N penalty at image boundaries caused by lens-shadow contours. FIG. 5 depicts a graph 500 of an exemplary lens shading profile. The y-axis is the value of the shading parameter β. The x-axis is the distance ("R") from the center of the lens. In such an embodiment, a larger R results in a smaller β. Thus, in such an embodiment, β is close to 1 near the center of the lens, and β is close to 0 near the outer edge (eg, perimeter) of the lens. Note that in this embodiment, the relationship between R and β is non-linear. The shape of the curve can be modeled according to Equation 4.

Equation 4: β=1−a·(R/(b·max(R))) ²

In Equation 4, β has a value between 0-1. Each pixel is thus processed according to its distance (R) from the lens center of the image. This distance or squared distance can be calculated according to the following equation:

Equation 5: $R=\sqrt{{x\_\mathrm{<figure-callout\; id="2"\; label="index"\; filenames="A2008800134220061H2.png,A2008800134220062H1.png"\; state="\{\{state\}\}">index</figure-callout>}}^2+\mathrm{the\; y}\_{\mathrm{<figure-callout\; id="2"\; label="index"\; filenames="A2008800134220061H2.png,A2008800134220062H1.png"\; state="\{\{state\}\}">index</figure-callout>}}^2}$

Equation 6: R ² =x_index ² +y_index ²

In these equations, x_index and y_index are the indices of the pixels in the image (x is the column and y is the row), and the center pixel in the image has index [0, 0]. The values of "a" and "b" are constants affecting the amount of shadowing.

The values of "a" and "b" may vary and may be provided by the upstream ISP chain 204a. Thus, in one embodiment, the shaded indicators provided by the upstream ISP chain 204a define constants a and b. In another embodiment, the shadow indicator is calculated based on image statistics. For example, a shading indicator may be computed by convolution block 206 based on the coordinates of the pixel data and the S/N associated with the image data. The S/N may be provided by the upstream ISP chain 204a. In another embodiment, a and b are set according to the lens shading profile such that they satisfy the lens shading profile characteristics and are fixed, no shadow indicator is required.

R computation logic 302 in convolution block 206 determines R for at least some of the pixels. Based on the value of R, β-update logic 304 accesses (or determines) the value of β. That is, β can be accessed from a table, or calculated at runtime. Typically, β is accessed from a table based at least in part on the R value of the pixel currently being processed to save processing time. However, β can be calculated at idle time.

Since the R value is only slightly different from one of its neighbors, it is not necessary to calculate a new R value for each pixel. Thus, it is not necessary to perform a β-update on the core for each pixel. Therefore, beta-updates can be performed every few pixels or even more pixels and still achieve the desired effect.

It is possible to choose between different shadow profiles for the same lens; however, for a specific lens design, a constant profile can be selected. In one embodiment, the shading indicator is an integer value between 0-8, where 0 indicates no shading at the boundary and 8 indicates severe shading at the boundary. However, values outside this range may be used.

Alternative Implementations for Determining Beta Coefficients

This section describes an alternative technique to determine β based on the "R" value, which describes the relative position of the pixel data with respect to the lens. In this embodiment, first, the value of ^R2 can be shifted into a smaller dynamic range. Equation 7 is an example of implementing this transfer.

Equation 7: f(R): R ² [0...max(R ² )] → [0...15]

After shifting the R value into a smaller dynamic range, β can be calculated using Equation 8:

Equation 8: β=15-p·f(R)/q

The constants "p" and "q" affect the amount of shadows and thus adjust the strength of the core. These constants thus serve a similar purpose to the constants "a" and "b" discussed in the previous technique for determining β, but are not the same constants. The "p" and "q" values may be provided by the ISP chain 204a (eg, a shaded indicator may define these constants). Alternatively, appropriate p and q values may be determined by convolution block 206 . For example, p and q can be determined based on image statistics. However, p and q can also be determined and kept fixed by the lens shading profile. Note that in this equation, the final value of β can be between 0-15 (ie, 4 bits are used to represent β here). However, depending on the implementation, β may have any desired dynamic range.

The amount of attenuation of cores at image boundaries can be adjusted based on 'p' and 'q'. For example, if p = 0 and q = 1, there will be no core attenuation at image boundaries. If p = 1 and q = 1, then at image boundaries the kernel will become a kronecker-β function. Thus, the core will be weakened.

The shadow profile from which the core gain may be updated may be a function other than R^2. For example, it may be a function of R, or a non-rotationally symmetric function. In these other cases, the formula for calculating β will be somewhat different. For example, polynomials can be used for this function as illustrated in the following equation:

Equation 9: β(R)=a ₀ +a ₁ (R/max(R))+a ₁ (R/max(R)) ² +...+a _n (R/max(R)) ⁿ

In Equation 9, β may have a final value between 0-1.

Likewise, adjustments based on the lens shading profile can be used to strengthen nuclei in image boundaries, rather than weaken them. In this embodiment, the kronecker-delta core is replaced with a core with a more aggressive highpass/bandpass frequency response, and the initial core coefficients are adjusted accordingly. The rationale for wanting to enhance the core is that, in general, for a given lens, the PSF is usually larger at the borders of the field of view (FOV) than at its center.

Spatial Correlation Processing

This embodiment of adjusting core effects based on lens shading profile is a technique for adjusting core effects according to spatial (or pixel) position. However, this is not the only technique used to adjust core effects based on spatial location. In one embodiment, different kernels are saved or retrieved during image processing to process different regions of the image sensor differently to account for position-dependent optical aberrations. This location dependent processing can be as fine grained as a single pixel region, but different regions can have more than one single pixel.

For example, optical aberrations can be calculated for different regions of the image sensor. Each region can be as small as a single pixel in size, but can include more than one single pixel. In one embodiment, a deconvolution kernel ("region kernel") is established for each region to cancel out the optical aberrations of the pixels in that region. Different regional cores may be stored for each region. Alternatively, a single region core can be stored for multiple regions, the core being tuned to allow different processing for different regions.

Adjusting Core Effects with Post-Deconvolution Scaling

As discussed earlier, as depicted in FIG. 4 , it is not necessary that the kernel be multiplied by α and β in order to obtain the result of updating the kernel with α and β. Figure 19 depicts an implementation where alpha and beta "scaling" is applied to the unmodified kernel based deconvolution result. In such an embodiment, the graphics interface 208 provides the noise level estimator signal and the shading indicator to the scaler 1602 which utilizes the alpha calculation logic 212 to calculate “α”. The alpha calculation may be performed in a similar manner to the embodiment depicted in FIG. 2 . In this embodiment, the graphics interface 208 provides the shading indicator signal and the pixel index to the scaler 1602 , which utilizes the beta calculation logic 302 to calculate “β”. The β calculation can be performed in a similar manner to the embodiment as depicted in FIG. 4 . In one embodiment, the shade indicator is not used to determine β. Conversely, β is determined based on the pixel index without having to use a shading indicator.

Alpha/Beta Scaling 1912 inputs the convolution result, alpha, beta, and initial values for the current pixel being processed, and outputs the final result for the current pixel. In one embodiment, the current pixel is the center pixel of the signal matrix. In one embodiment, the alpha scaling 1612 is performed as follows:

Equation 10: FinalRsult = ConvRs·α·β+(1-α·β)·InputPixel

In Equation 10, ConvRs is the convolution result, and InputPixel is the initial value of the current pixel being processed. Thus, with the unmodified kernel and one or more other values, the final processed result for the current pixel is a weighted sum of the convolution results. Those other values include, but are not limited to, the original pixel values in the input image. In such an embodiment, the alpha and beta parameters are used as weighting (or scaling) parameters. The beta parameter can be used instead of the alpha parameter if desired.

5.3 Adjust the core effect according to local image features

In one embodiment, the core effect is adjusted according to local image characteristics. The image features may be local features of the current pixel being processed. Adjusting core effects based on image characteristics can be performed in conjunction with the previously discussed alpha adjustments and/or beta adjustments. However, in one embodiment, the core effects are adjusted according to image characteristics without any alpha or beta adjustments. Furthermore, tuning of the core other than alpha- or beta-tuning can be performed using the techniques of this section.

One aspect of adjusting the kernel effect according to image characteristics is to adjust the processing level (i.e., kernel strength or effect size) according to the center pixel value itself (or, alternatively, according to a weighted average of several pixels around ). Thus, for example, when the center pixel (or average) has a high value, the effect size will be larger, and when the center pixel has a low value, the effect size will be smaller. The opposite can also be used - the lower the value of the center pixel (or the average of several pixels around the center pixel) the stronger the core effect, the higher the value the weaker the effect - depends on the need. One embodiment of varying the magnitude of the effect based on the pixel value is given in Equation 11 below:

Equation 11: FinalRsult=pixel_val/Max_pixel_val·ConvRs+

(1-pixel_val/Max_pixel_val) InputPixel

In Equation 11 above, before convolution, when pixel_val is high, the final result will approximate the convolution result, and when pixel_val is low, the final result will approximate the initial pixel value.

Another aspect of adjusting the kernel effect based on image characteristics is to identify areas in the image where the kernel should be applied with greater or lesser intensity. Thus, the magnitude of the effect to be produced on the signal matrix with respect to the core matrix is determined. The desired effect may be image contrast, in which case, for maximum effect, the kernel may be applied in the image area where the desired image contrast is greatest. Alternatively, in areas where little image contrast is desired, the kernel may not be applied at all, or the effect of applying the kernel may be modified, resulting in less image contrast. In one embodiment, the effects of the application core are modified such that a smooth transition between minimum and maximum effects is achieved. In one embodiment, there is a default kernel that is applied when no adjustment to the effect is desired. The adjustment may enhance the magnitude of the effect of the default kernel, or may decrease the magnitude of the effect of the default kernel. In another embodiment, the kernel used is a low pass filter for denoising purposes, where the same metric is used to determine where to apply the kernel and with what strength.

Structural Survey of Adjusting Core Effects Based on Local Image Features

7 depicts a block diagram of a system 700 in which kernel update logic 704 in convolution block 206 modifies the kernel based on metrics computed from image features in the location of the pixel currently being processed. The metric itself can be modified, resulting in a value referred to herein as "γ". In system 700, the core is first updated based on the alpha- and beta-updates before doing the gamma-updates. However, it is not necessary to use alpha- or beta-updates together with gamma-updates. Also, in this embodiment, the γ update follows the α- and β-updates. However, the order of core updates can be changed. As described anywhere else herein, it is not necessary to modify the kernel itself in order to adjust the effect of applying the kernel. In one embodiment, scaling is performed after processing the image with an unmodified kernel in order to adjust the processing effect according to the kernel.

Metric calculation logic 702 in convolution block 206 calculates the metric based on an analysis of the current pixel being processed and its neighbors. The metric calculation logic 702 examines the current image window (multiplied by the image segment of the convolution kernel) for each pixel, determining a metric for the desired amount of processing for the current pixel. Next, based on the metrics, the convolution kernels are updated by the kernel update logic 704 . Finally, apply the updated kernel to the image data.

Once γ is defined by the metric, logically the method for updating a core can be similar to the method for α- or β-updating a core. For example, from a logical point of view, if one wishes to reduce the size of the process, the updated core can be generated from a linear combination of the initial core and the delta core, or if one wishes to increase the size of the process, with a more aggressive The core of sex arises. Referring again to FIG. 3, the core may be updated with gamma instead of a. Thus, γ affects the kernel coefficient of the linear combination of kernels. Note that if the convolution is performed with a kernel updated with gamma and alpha and/or beta, a non-linear convolution result is obtained. That is, a convolution is performed where the kernel changes for each pixel depending on more than one condition. Note that post-deconvolution scaling as described herein can be used to obtain deconvolution results with updated kernels.

Functional overview of adjusting core effects based on local image features

As discussed earlier, in this embodiment the intensity of processing is adjusted based on local features surrounding the pixel currently being processed. First, check the area around the current pixel to determine things like if there is an edge, if the area around the pixel is relatively planar - i.e. uniform, there is no change in the information in the area (such as a blank gray wall) - etc. class information. Equation 11 describes a method for computing the metric.

Equation 12: $\mathrm{metric}=\sqrt{\frac{1}{N}\underset{j=1}{\overset{N}{\mathrm{\&Sigma;}}}|P_j-\stackrel{\&OverBar;}{P}|}$

In Equation 12, P _j and N are pixels within the window, and P is the average value of N pixels.

Example of Determining the Metric

Below are several exemplary methods for computing the metric. The metric may be calculated using a current image window around the center pixel from which the output value is calculated. However, it is not necessary to use all pixels in the entire current image window (those that are being colored). Furthermore, the metric may be based on pixels outside the current image window. Furthermore, it is not necessary to determine the measure based only on pixels of the same color. That is, for some types of processing it may be appropriate to include pixels of different colors in the metric calculation.

Variance/Standard Deviation Example

In one embodiment, the variance or standard deviation (win_std) of the pixels in the current image window is used as an indicator of the presence and amount of differences in the current image window around the center pixel. If standard deviation is used, the formula in Equation 13A or 13B can be used:

Equation 13A: $\mathrm{win}\_\mathrm{std}=\sqrt{\frac{1}{\mathrm{no}-1}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}{(x_i-\stackrel{\&OverBar;}{x})}^2}$

Equation 13B: $\mathrm{win}\_\mathrm{std}=\sqrt{\frac{1}{\mathrm{no}}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}{(x_i-\stackrel{\&OverBar;}{x})}^2}$

In Equations 13A and 13B, _xi is the pixel in the current image window, "n" is the number of pixels in the current image window, and x is the average value. As previously discussed, in one embodiment, the pixels in the current image window are all the same color. The average value can be determined as indicated in Equation 14.

Equation 14: $\stackrel{\&OverBar;}{x}=\frac{1}{\mathrm{no}}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}{x}_i$

However, in one embodiment, it is not essential that all pixels participate in the average calculation. Furthermore, in one embodiment, rather than a simple average, a weighted average is used, where the pixels participating in the average are weighted according to their spatial position relative to the central pixel, or according to another weighting scheme. The weights are normalized to 1 so that the mean value of the homogeneous region remains the same. The weighted average (where _Wi is the weight) is exemplified in the following equation:

Equation 15: $\stackrel{\&OverBar;}{x}=\frac{1}{\mathrm{no}\&Center\; Dot;\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}{w}_i}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}{w}_i{x}_i$

If _xi is considered to be a random variable, these are the unshifted and shifted estimators for the standard deviation. Next, in one embodiment, two values are calculated - the maximum value of win_std in the current image window (Max_std_val), and the minimum value of win_std in the current image window (Min_std_val). These values are calculated according to Equations 16a and 16b.

Equation 16a: $\mathrm{Max}\_\mathrm{std}\_\mathrm{val}=\underset{x_i}{\max }\left(\right|x_i-\stackrel{\&OverBar;}{x}\left|\right),i=1..\mathrm{no}$

Equation 16b: $\mathrm{Min}\_\mathrm{std}\_\mathrm{val}=\underset{x_i}{\min }\left(\right|x_i-\stackrel{\&OverBar;}{x}\left|\right),i=1..\mathrm{no}$

In Equations 16a and 16b, x is the average value and "n" is the number of pixels in the current image window. These minimum and maximum values define the range of standard deviations from top to bottom. Thus, the dynamic range (dyn_range) of the standard deviation of the current image window can be defined according to Equation 16.

Equation 16: dyn_range = Max_std_val - Min_std_val

The core can be updated according to the equation depicted in FIG. 11 . Note that kernel updates as described in FIG. 11 may be implemented by gamma scaling by post-deconvolution as described herein. That is, the signal matrix is convolved with the unmodified kernel matrix, and then the sum of the convolution result and the original pixel values is scaled according to γ.

Note further that if dyn_range=0, all pixels in the neighborhood of the center pixel are the same. Thus, whether the central pixel has the same value as the remaining pixels in the current image window (ie, win_std=0), or whether the central pixel has a different value than the remaining pixels, the delta kernel is applied to the central pixel. Note that no processing may be expected in any of the above cases. The first case may indicate that the current image window is a "flat" area (eg, wall, blue sky, etc.) where no enhancement of image contrast is expected. The second case may be caused by noise since no pattern is detected; thus enhancing image contrast may not be expected. Further, note that in the equation of FIG. 11 , the sum of the two coefficients multiplied by input_kernel and delta_kernel is 1.

If the current image window is "planar" or nearly "planar", or even if it is a noisy region, but has a relatively low noise standard deviation, then the standard deviation is a low value. Therefore, the coefficient multiplied by input_kernel in Figure 11 is close to 0, and the coefficient multiplied by delta_kernel is close to 1. So, for areas of the image where there is little detail, little processing happens, as desired.

However, if the current image window includes an edge between light and dark regions, then the standard deviation value is large (eg, close to the maximum possible value). Therefore, the coefficient multiplied by input_kernel in Figure 11 will reach a value close to 1, and the coefficient multiplied by delta_kernel will get a value close to 0, which means that final_kernel will sharpen the image and restore the lost contrast. Since a dynamic range of standard deviations is set for each pixel, it does not rely on a fixed threshold, which varies with the scene scene.

However, it is not necessary to calculate the minimum and maximum values in the window and to use the local dynamic range. In one embodiment, a fixed dynamic range is used to normalize the standard deviation to the dynamic range [01]. For example, for pixel values from 0 to 1023 (10 bits per pixel), the standard deviation may receive a value from 0 to 512, thus fixing the dynamic range into that range.

Alternatively, more complex functions can be applied to the standard deviation in order to vary the core in a more sophisticated manner. For example, the dynamic range of the standard deviation can be divided into parts, and in each part, a different function is applied in such a way that the transition between these parts is continuous and achieves the desired effect. For example, the graph 1500 depicted in FIG. 15 illustrates a gamma curve with respect to the metric, according to an embodiment. Referring to FIG. 15 , two thresholds ("low threshold (low_th)" and "high threshold (high_th)") are defined. If the metric is low (eg, below "low_th"), then there is no edge information (or very little edge information) in the current image window. Therefore, the gamma value is kept low. If the metric lies between "low_th" and "high_th", the higher value of γ is assigned. The range proves that there are edges, but note that the range has an upper limit "high_th". In the embodiment of FIG. 15, the value of γ increases above "low_th" until γ reaches 1 at some value between "low_th" and "high_th". However, if the metric is high (eg, above "high_th"), the value of γ gradually decreases from the value of 1 obtained by γ at "high_th". This reduction in gamma reduces the likelihood of over-sharpened edges that are already sharp enough. From each metric value, calculate the γ parameter, where γ is between 0 and 1. This function is continuous over the entire range of the metric. The γ value is multiplied by the core, while (1-γ) is multiplied by the kronecker-delta core (or, alternatively, the more aggressive core - depending on the desired effect ).

Instead of using standard deviation as a measure for gradient-presence identification, variance can be used according to the following formula:

Equation 17: $\mathrm{win}\_\mathrm{var}=\frac{1}{\mathrm{no}-1}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}{(x_i-\stackrel{\&OverBar;}{x})}^2$ or $\mathrm{win}\_\mathrm{var}=\frac{1}{\mathrm{no}}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}{(x_i-\stackrel{\&OverBar;}{x})}^2$

In Equation 17, the notes are the same as in the standard deviation formula. In this alternative, the maximum and minimum values should be determined accordingly in order to create a suitable dynamic range (alternatively, a fixed dynamic range could be used, as listed previously). Also here, the formula in Figure 11 can be used to update the core, or more complex functions can be used as described above.

As another alternative to using standard deviation as a measure for gradient-presence identification, absolute value of differences can be used according to Equation 18:

Equation 18: $\mathrm{win}\_\mathrm{abs}=\frac{1}{\mathrm{no}-1}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}|x_i-\stackrel{\&OverBar;}{x}|$ or $\mathrm{win}\_\mathrm{abs}=\frac{1}{\mathrm{no}}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}|x_i-\stackrel{\&OverBar;}{x}|$

where the notes are the same as in the standard deviation formula. In this alternative, the maximum and minimum values should be determined accordingly in order to create a suitable dynamic range. Also here, the formula in Figure 11 can be used to update the core, or more complex functions can be used as described above.

Histogram Example

An approach similar to the standard deviation method is to compute a histogram of the current image window and determine the size of the kernel based on the histogram. The local histogram of the current image window can be calculated by first defining a certain number of bins. This can be done eg according to Equation 19.

Equation 19: #bins=max(x _i )-min(x _i )+1, i=1..n, x _i ∈ image_window

Alternatively, the number of bins may be a predetermined fixed number. For example, 1024 bins are available for the number of intensity levels corresponding to a 10-bit image. Then, the bins themselves are defined, for example, by dividing the dynamic range of the image window by #bins, as described in Equation 20.

Equation 20: dyn_range=max( _xi )-min( _xi )+1, i=1..n, _xi ∈ image_window

In such an embodiment, there would be #bins bins, each 1 pixel wide. The dynamic range can be divided into wider bins, each bin being several pixel values wide. Once the bins are defined, each bin is filled with the number of pixel values in the image window that fall within that bin range. Figure 9A illustrates data for an image window, and Figure 9B illustrates a histogram for the data.

In one embodiment, once the histogram of the current image window is known, a determination is made of how close the histogram is to a unimodal histogram and how close the histogram is to a multimodal histogram. A unimodal histogram may also be called a uniform histogram. In one embodiment, the multimodal histogram includes a bimodal histogram and a higher order histogram.

The closer the shape of the histogram to a multimodal histogram, the stronger the indication that edges are present in the current image window. It may be desirable to enhance the effect of deconvolution kernels applied to windows with edges. For example, it may be desirable to enhance the contrast of an image. Alternatively, if the core has a low pass frequency response, it may be desirable to reduce their effect size so that they do not blur edges in the image.

The closer the histogram is to a unimodal histogram, the more "flat" (ie, more uniform) the image window will be. Therefore, it may be desirable to modify the deconvolution kernels so as to minimize their effects. For example, in this case it may be desirable not to enhance image contrast. Noise image windows can produce uniform or unimodal histograms. Figure 1OA illustrates a unimodal histogram and Figure 1OB illustrates a bimodal histogram for different green image windows. These histograms were generated with 1024 bins (# of intensity levels of the 10-bit bayer image), and 41 values of the distribution (# of values in the green image window for the 9x9 green kernel).

Thus, in one embodiment, the histogram is analyzed to determine whether the distribution is unimodal or multimodal. One technique to distinguish unimodal from multimodal histograms is to employ a clustering algorithm (eg, k-means), which can be used to detect the presence of two or more clusters. Additionally, the metric may be based on distances between centers of the plurality of clusters. To obtain a measure of the gain for updating a core, the distance is normalized by its dynamic range.

Another technique for distinguishing bimodal from unimodal histograms is based on smoothing the histogram and then finding local maxima therein. A histogram is unimodal if there is only a single local maximum. However, if two local maxima are found, the range between the two local maximum points is used to derive the metric. For example, if the range is small, the core may be tuned to have low gain. However, if the range is large, the core can be tuned to have high gain. These adjustments can be expected to cause the kernel to have the most image contrast enhancement when the range is large. If three or more local maxima are found, the measure may also be based on the range between the local maxima.

There may be a relationship between the standard deviation technique discussed previously and the shape of the histogram. A current window matrix with pixels whose values have a high standard deviation may have a histogram shape with two or more peaks, where pixel values are far from the mean. On the other hand, a current window matrix with pixels whose values have a low standard deviation may have a histogram with a single peak where most of the pixel values are close to the mean.

Example of edge detection

In one embodiment, determining the metric is based on detecting edges in the image. For example, convolution masks can be used to find edges in the current image window. Examples of convolution masks that may be employed include, but are not limited to, Sobel, Laplacian, Prewitt, and Roberts. In one embodiment, an edge detection mask is applied to the current image window to detect edges. A statistical measure such as the standard deviation of the pixels in the window is applied to the result. The metric is then based on the standard deviation result. In another embodiment, an edge detection mask is applied to the current image window to detect edges. In the edge detection implementation, the kernel is applied based on whether an edge is found.

distance example

The information measure can also be constructed based on the "distance" method. Figure 23 illustrates a method 2300 for constructing the information metrics. The method begins at step 2304, where the absolute differences between pixels are summed. The near distance is calculated from the pixels around the middle pixel in the current image window. Far distance is calculated from pixels that are spatially far away from intermediate pixels in the same image. The information metric 2312 is computed from short distances only, while the metric 2316 is computed from both short and long distances. At step 2320, the method 2300 calculates a weighted average of the two information measures with weights determined from the two information measures, which results in a final information measure, albeit without scaling. At step 2324, the final information measure is scaled according to the spatial position within the current image, and the scaled information measure is then provided to process 2400, which is described in more detail in FIG.

In one embodiment, the various distance metrics are computed by dividing the image window into clusters of pixels, and for each cluster, using A distance metric is computed for each cluster as the weighted sum of the absolute values of the differences between matching pixels in . Equation 21 shows the computation of the distance metric for an exemplary single 3x3 cluster example:

Equation 21: $d_k=\underset{i=1}{\overset{3}{\mathrm{\&Sigma;}}}\underset{j=1}{\overset{3}{\mathrm{\&Sigma;}}}W[i,j]\&Center\; Dot;\left|C_k\right[i,j]-C_{\mathrm{middle}}[i,j\left]\right|$

where W[i, j] is a configurable weighting matrix, C _k [i, j] is the value of pixel [i, j] in the 3x3C _k cluster, C _mid [i, j] is the value of [i, j] in the middle cluster ] pixel value.

Edge Enhancement Implementation

Since the coefficients in the edge detection mask sum to 0, the synthesized image window will contain strong positive and negative values if there is an edge in the initial image window. Therefore, image windows containing edges will have high standard deviation values and thus high processing-metric (p metric). On the other hand, if the window has data for "flat", uniform regions in the image, or even for noisy regions with relatively low noise standard deviations, then the edge-filtered image will have pixels close to each other value (typically close to 0). Therefore, the edge-filtered image will have low standard deviation values and low processing metrics. In one embodiment, a fixed predetermined threshold is used to normalize the standard deviation such that the final processing metric is a value between 0 and 1 . The metric calculation can be determined as indicated in Equation 22:

$\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; metric</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; std</span>}}{\mathrm{<span\; class="notranslate">\; edge</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; std</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; threshold</span>}}$

Equation 22: if (p_metric > 1)

p_metric=1

The core can be updated in a manner similar to that depicted in FIG. 3 . However, p_metric replaces α.

Normalizing the standard deviation results in a smooth gradient of the process metric, from low to high values, and thus creates a continuous and smooth process and removes artifacts in the output image.

Table II provides an exemplary value for edge detection on a bayer image processed with green bayer. If the green pixels are divided into Gr (green in red row) and Gb (green in blue row), then Table III can be used for all pixels (Gr, R, B and Gb). Table III provides exemplary values for edge detection masks for red or blue bayer processing.

Table II

Table III

By way of non-limiting example, the edge detection mask used for the processing performed on the image after demosaicing may be the well known Sobel, Prewitt, Roberts and Laplacian masks.

Edge detection implementation

In one edge detection implementation, an edge detection mask (for bayer mode if the processing is performed on a bayer image) is applied to the image window to find edges. Thresholding can be used to determine whether edges are present, considering an a-priority assumption: there is a pattern of how edges appear in natural images. For example, individual pixels that differ from other pixels may be random noise, as opposed to valid image data. The threshold may be updated based on results obtained for neighboring pixels in order to reduce the likelihood of falsely detecting noise as opposed to detecting edges. A possible implementation of this method can be as follows.

First, an edge detection mask is applied to the current image window. If there are absolute values above the current threshold above, it is checked whether they form a line or other pattern. Indicating that a line or other pattern may be, several neighboring pixels have high absolute values. If a line or other pattern is found, the current image window is declared to include edges. If no pixels have an absolute value above the threshold, or if the pixels with a high absolute value do not form a line or other pattern, the current image window is considered to contain a "flat" (possibly noisy) region .

Next, depending on whether an edge is found, a deconvolution kernel is applied to the current image window. For example, if edges are found, a strong deconvolution kernel can be applied. Otherwise, a weak deconvolution kernel may be applied to the window. Additionally, setting the deconvolution kernel gain can be done in terms of the "strength" of the edges. For example, in the case of finding edges, the standard deviation of the pixels in the image window is calculated after applying the edge mask. The core's gain is then measured based on the standard deviation in a manner similar to the standard deviation method described above.

Other edge detection methods may also be used. For example, other edge detection masks including, but not limited to, Roberts, Prewitt, Sobel, or Canny can be used to determine whether an edge exists in a window.

Entropy-based embodiment

In one embodiment, the size of the kernel to be applied to the image window (eg high gain or low gain of the deconvolution kernel) is based on the entropy in the current image window. Higher entropy may indicate that there is a lot of information in the current image window, and thus it may be desirable to apply a stronger core to that window. A "flat" image window may have low entropy; therefore, it may be desirable to apply weaker cores to the window processing metrics. The Entropy calculation may be determined as described in Equation 23.

Equation 23: $\mathrm{Entropy}=-\underset{i=o}{\overset{1023}{\mathrm{\&Sigma;}}}{p}_i\log \left(p_i\right)$

In Equation 23, _pi is the empirical probability of pixel value i in the current image window. If, for example, the image window is "flat" the entropy is 0, if it includes edges (pixels with different intensity values) the entropy will receive a high value. By normalizing the entropy value from Equation 23, the actual processing metric can be determined.

However, since the entropy does not take into account the amount of difference between pixels, high entropy values may be caused by random noise. Therefore, in one embodiment, the S/N of the image is used to determine whether to employ entropy techniques.

Gradient-based embodiment

In one embodiment, the presence of edges is determined by computing several local gradients around the center pixel in the window currently being processed. A metric is then calculated from the gradient magnitudes, for example from a weighted average of the gradient magnitudes. Gradients can be obtained according to the following examples:

Table IV

From Table IV, the possible gradients are: |1-13|, |2-12|, |3-11|, |6-8|, |4-10|, |5-9|.

Table V

From Table V, the possible gradients are: |1-9|, |2-8|, |3-7|, |6-4|.

Table IV shows the pixel locations for the 5x5 support for green in the bayer image, and Table V shows the pixel locations for the 5x5 support for red and blue in the bayer image. When calculating the weighted average of the gradients, the weights may be determined by the spatial distance between pixels participating in a certain gradient.

Once the metrics are calculated, the core update mechanism can proceed as set forth above. That is, the metric may be normalized according to the possible dynamic range, thereby determining "γ". In one embodiment, similar to that described in FIG. 15 , the normalization may be performed using a gain function. Then, techniques disclosed herein that affect deconvolution kernels as used in alpha and beta updates can be used for gamma updates.

In one embodiment, the method for creating the metric consists of a weighted combination of some of the methods described above. For example, the final metric is based on a weighted average of the standard deviation metric and the gradient metric, or any other combination of 2 or more metrics is used to create the final metric.

available light embodiment

It is also possible to have two or more separate gain functions that are used depending on a local estimate of the amount (level) of light in the current image window, instead of always using the same gain function to derive from the information "γ" is determined in the metric. In one embodiment, this estimate may be the value of the intermediate pixel itself or a weighted average of several nearby pixels. An exemplary method 2400 is shown in FIG. 24, where exactly two gain functions are used, although more than two gain functions may be used, FIG. 24 is only an example. At step 2404, the scaled information measure (discussed in connection with FIG. 23) is passed to two gain functions Fg1 and Fg2. In this embodiment, the gain function Fg1 can be applied to high light levels, and the gain function Fg2 can be applied to low light levels. At the same time, at step 2408, the light level of the pixel currently being processed is measured. In steps 2412 and 2416, first and second gain functions Fg1 and Fg2 are applied to the scaled information metric. In step 2420, the outputs of the two gain functions are weighted according to the light level of the pixel measured in step 2408, as specified, and summed to produce a unified metric.

Adjust Core Effects with Post Deconvolution Scaling

It is not essential that image processing based on α, β and γ includes multiplying the kernel by γ. Figure 20 depicts an embodiment in which alpha, beta and gamma "scaling" is applied to the deconvolution result of the unmodified kernel. In this embodiment, the graphics interface 208 provides a noise level estimator signal and a shading indicator to the zoom 1602 , which uses the alpha calculation logic 212 to calculate “α”. The alpha calculation may be performed in a manner similar to the embodiment depicted in FIG. 2 . In such an embodiment, graphics interface 208 provides a shading indicator signal and a pixel index to scale 1602 , which utilizes β calculation logic 302 to calculate “β”. The beta calculation may be performed in a manner similar to the embodiment depicted in FIG. 4 . In such an embodiment, the graphics interface 208 provides the value of the current pixel and its neighbors to the scaling 1602 , which utilizes the gamma calculation logic 702 to calculate “γ”. The gamma calculations may be performed according to any of the techniques disclosed herein.

The α/β/γ scaling 2012 inputs the convolution result, α, β, γ and the initial value of the current pixel being processed, and outputs the final result of the current pixel. In one embodiment, the current pixel is the center pixel of the signal matrix. In one embodiment, the alpha scaling 1612 is performed as follows:

Equation 24: FinalRsult = ConvRs·α·β·χ+(1−α·β·χ)·InputPixel

In Equation 24, ConvRs is the convolution result and InputPixel is the initial value of the current pixel being processed. Thus, with the unmodified kernel and one or more other values, the final processed result for the current pixel is a weighted sum of the convolution results. Those other values include, but are not limited to, the original pixel values in the input image. In such an embodiment, the alpha, beta and gamma parameters are used as weighting (or scaling) parameters. Any combination of alpha, beta and gamma parameters can be used.

6.0 Adjusts Core Effects to handle motion blur depending on implementation

6.1 Overview of Correcting Motion Blur

Motion blur is the blurring of an image due to camera movement during image acquisition. Motion blur often occurs when images are acquired with long exposure times, such as in low light conditions. Typically, motion blur is characterized by a PSF that is directional. Horizontal lines, for example, imply blur due to the horizontal motion of the camera.

Compensating for motion blur generally includes determining motion blur and correcting the determined motion blur. Determining motion blur includes determining the direction, amount and character of motion, which can be specified with motion vectors. Motion vectors may be measured by gyroscopes, inferred from the images themselves, inferred from a set of images, or extracted by other means. A deconvolution kernel may be applied to the image, correcting the determined motion blur.

This paper describes various image processing algorithms in which the effect of applying certain spatial image processing kernels can be determined according to features such as S/N, lens shading profile, optical aberrations depending on pixel position, and the position of the pixel currently being processed. and so on to adjust for various factors. In one embodiment, motion blur can also be corrected in the same IP block, where the aforementioned spatial image processing core is applied. Herein, the spatial image processing core may be referred to as a core for correcting the "lens PSF". Thus, motion blur and lens PSF blur can be corrected simultaneously. Since motion blur and lens PSF blur can be shaped as a PSF convolved with an image, the combined blur can be shaped by simultaneously convolving the motion blur PSF and the lens PSF. Thus, in one embodiment, the deconvolution kernel may be tuned to match the combined motion blur PSF and shot PSF.

If used, motion vectors can be extracted from either a single frame image, or a sequence of frames. By way of non-limiting example, a gyroscope can be used to help define motion vectors. In one embodiment, depending on the motion direction (or directions), a motion blur PSF can be deduced from said motion vectors. In one embodiment, the deconvolution inverse kernel is computed from the motion blur PSF.

However, it is not required to deduce the motion blur PSF from the motion vectors. Conversely, information from the motion vectors themselves may be sufficient to correct motion blur without inferring a motion blur PSF. Alternatively, the motion blur information may already be given in the form of a (motion blur) PSF, wherein motion vectors are not necessary for deriving the motion blur PSF.

In one embodiment, the deconvolution inverse kernel is determined from the motion blur PSF or motion vectors. The deconvolution inverse kernel matches the blur direction and shape, and the image is to be sharpened. In one embodiment, the deconvolution kernel may be selected from a set of kernels that match the possible orientations of the motion blur PSF. These deconvolution cores are kept in memory and include a large number of motion blur possibilities. In one embodiment, the selected kernel matches a PSF that is the convolution of the shot PSF and the motion blur PSF along the direction of the motion vector. The cores may be selected using a matching algorithm.

Once the deconvolution kernel is determined by calculation or selection, the image can be processed by convolving the image with the deconvolution kernel, taking into account the S/N, pixel position, lens shading profile, and image features to adjust the deconvolution kernel.

Note that the deconvolution kernel can handle other optical aberrations besides blur, which kernel can have any desired frequency response. In one embodiment, motion blur correction can be integrated into the sharpening block 114 described herein without increasing silicon size. For example, hardware for implementing sharpening algorithms is used for motion blur correction. Thus, compared to handling motion blur and PSF blur separately in two different ISP blocks, the techniques disclosed herein are more efficient in terms of resources (e.g., silicon size/gate count, processor cycles, power consumption, complexity) It is relatively cost-effective.

6.2 Implementation of fuzzy point distribution function

One way to implement motion blur correction is to estimate the characteristics of the motion blur and create a matching deconvolution kernel ("motion blur kernel") during preview mode. This approach is illustrated in the embodiment depicted in FIG. 17 . This method identifies motion blur by comparing frames during preview mode. Then, upon entering capture mode, the motion blur kernel and other kernels are applied to the image. Other cores can be used for sharpening. In various implementations, the alpha, beta, and/or gamma parameters are used to modify the strength of the effect of processing with other cores, as described herein.

Referring now to the preview mode ISP 1710, in block 1725 a blurred PSF 1721 is estimated based on the motion blurred image 1722. The blur PSF 1721 may be estimated by analyzing the acquired image 1722 using techniques known to those skilled in the art. One technique for estimating the blur PSF 1721 by analyzing the acquired image 1722 involves using a gyroscope to supply motion vectors to the camera. Another method involves using correlation to find the direction of blur, comparing multiple adjacent frames, and extracting the direction of motion based on the comparison. In block 1728, based on the blur PSF 1721, a motion blur deconvolution kernel is created.

Referring now to the acquisition mode ISP chain 1720, the motion blur kernel is provided to logic that performs other kernel processing on the image. In one embodiment, the other logic is a digital autofocus block 111 . Since the convolution operation is associative (i.e., a*(b*image)=(a*b)*image), the convolution kernel from sharpen and the motion blur kernel can be combined to apply A core of the image. If the sharpening is not enabled, but the digital autofocus block 111 is implemented in IP chain 1720, then the digital autofocus block 111 can be used solely for motion blur deconvolution.

6.3 Motion translation vector implementation

The digital autofocus block 111 in the IP chain can be used to solve the problem of shaky images during preview mode or video streaming. In preview mode or video streaming, a spatial offset may occur between adjacent frames, which results in a "shaky" image on the display screen. One way to solve this problem is to translate the image by a predetermined amount in the opposite direction to the motion vector estimated from the previous frame. Once the motion vectors are estimated, this translation can be implemented in the digital autofocus block 111 by convolving the image with a kernel that results in a spatial translation in the desired direction.

Figure 18 depicts correcting for motion translation, according to an embodiment. Frame 1801 in each of images 1802a-c includes the "viewed" image. Note that the frame is moving relative to the house depicted in the three images 1802a-c. The motion vector estimation logic 1810 estimates a motion vector in block 1820 from the current frame (n) and parameters from the previous frame (n-1). For example, extracting features from multiple locations in each frame. Next, the feature locations in the current frame are compared with feature locations in one or more previous frames, thereby calculating motion vectors. Examples of such features include, but are not limited to, edges or other unique marks in a scene that may continue from frame to frame.

Based on the estimated motion vectors, in block 1830 a translation kernel is computed. One technique for computing a translation kernel from the motion vectors includes employing a kronecker-delta kernel that translates from earlier computed motion vectors. In preview/video mode ISP 1850, a panning core is provided to the digital autofocus block 111, where frame(n) is processed based on the panning core and other cores. Other cores can be used for sharpening. In various implementations, the alpha, beta, and/or gamma parameters are used to modify the strength of the effect of processing with other cores, as described herein.

6.4 Combining sharpening and motion blur kernels

As mentioned herein, the sharpening algorithm may use a certain support size for which an implementation-dependent kernel may be used. For example, the core may provide a 9x9 pixel support. However, using the anti-motion-blur together with the sharpening method does not necessarily increase the kernel size. Usually, when convolving two cores - if one is MxM size and the other is NxN size, then their convolution is (M+N-1)x(M+N-1). In order to maintain kernel size, the shape of the sharpen kernel can be changed to more closely resemble and complement the motion blur kernel. Also, note that the deblurring kernel may have an asymmetric shape. For example, the deblurring kernel may include rectangular, diagonal, and even circular supports, and not necessarily square supports. This asymmetry provides compensation not only for "simple" jolt-paths (such as lines), but also for complex jolt-paths (such as cross-shaped, T, or circular paths).

8.0 Hardware Overview

8.1 Mobile Device Embodiment

Figure 13 illustrates a block diagram of an exemplary mobile device 1300 in which embodiments of the present invention may be practiced. Mobile device 1300 includes camera arrangement 1302 , camera and graphics interface 1380 , and communication circuitry 1390 . The camera arrangement 1370 includes a camera lens 1336 , an image sensor 1372 and an image processor 1374 . Camera lens 1336 , comprising a single lens or multiple lenses, collects and focuses light onto image sensor 1372 . Image sensor 1372 captures an image formed from light collected and focused by camera lens 1336 . Image sensor 1372 may be any conventional image sensor 1372, such as a Charge Coupled Device (CCD) or Complementary Metal Oxide Semiconductor (CMOS) image sensor. Image processor 1374 processes the raw image data captured by image sensor 1372 for subsequent storage in memory 1396 , output to display screen 1326 , and/or transmission via communication circuitry 1390 . The image processor 1374 may be a conventional digital signal processor programmed to process image data, as is well known in the art.

Image processor 1374 interfaces with communication circuitry 1390 via camera and graphics interface 1380 . Communications circuitry 1390 includes antenna 1392 , transceiver 1394 , memory 1396 , microprocessor 1392 , input/output circuitry 1394 , audio processing circuitry 1396 , and user interface 1397 . The transceiver 1394 is coupled to the antenna 1392 which receives and transmits signals. Transceiver 1392 is a fully functional cellular radio transceiver that can operate in accordance with any known standard, including commonly known as Global System for Mobile Phones (GSM), TIA/EIA-136, cdmaOne, cdma2000, UMTS, and Wideband CDMA and known standards.

The image processor 1374 may process images acquired by the sensor 1372 utilizing one or more implementations described herein. The image processor 1374 may be implemented in hardware, software, or some combination of software and hardware. For example, the image processor 1374 may be implemented as part of an application specific integrated circuit (ASIC). As another example, to implement one or more embodiments of the invention, the graphics processor 1374 may access instructions stored on a computer-readable medium and execute those instructions on the processor.

Microprocessor 1392 controls the operation of mobile device 1300 , including transceiver 1394 , according to programs stored in memory 1396 . Microprocessor 1392 may further perform some or all of the image processing embodiments disclosed herein. Processing functionality may be implemented in a single microprocessor, or in multiple microprocessors. Suitable microprocessors may include, for example, general and special purpose microprocessors and digital signal processors. Memory 1396 represents the overall memory hierarchy in the mobile communication device and may include random access memory (RAM) and read only memory (ROM). Computer program instructions and data required for operation are stored in non-volatile memory, such as EPROM, EEPROM, and/or flash memory, which may be implemented as discrete devices, as a stack device, or may be integrated with microprocessor 1392 .

Input/output circuitry 1394 interfaces microprocessor 1392 with image processor 1374 of camera device 1370 via camera and graphics interface 1380 . Camera and graphics interface 1380 may also interface image processor 1374 with user interface 1397 according to any method known in the art. Additionally, input/output circuitry 1394 interfaces with microprocessor 1392 , transceiver 1394 , audio processing circuitry 1396 , and user interface 1397 of communications circuitry 1390 . User interface 1397 includes display screen 1326 , speaker 1328 , microphone 1338 , and keypad 1340 . A display screen 1326 disposed on the back of the display area allows the operator to see dialed digits, images, call status, menu options, and other business information. Keypad 1340 includes an alphanumeric keypad and optionally includes a navigation control, such as a joystick control (not shown) as is well known in the art. Further, keypad 1340 may include a full QWERTY keyboard such as those used on handheld or smartphones. Keypad 1340 allows the operator to dial numbers, enter commands, and select options.

The microphone 1338 converts the user's voice into an electronic audio signal. Audio processing circuitry 1396 accepts analog audio input from microphone 1338 , processes these signals, and provides these processed signals to transceiver 1394 via input/output 1394 . Audio signals received by transceiver 1394 are processed by audio processing circuitry 1396 . The base analog input signal processed by the audio processing circuit 1396 is provided to the speaker 1328 . The speaker 1328 then converts the analog audio signal into an audible signal that can be heard by the user.

Those skilled in the art will appreciate that one or more of the elements shown in FIG. 13 may be combined. For example, while the camera and graphics interface 1380 are shown as separate components in FIG. 13 , it is understood that the camera and graphics interface 1380 may be combined with the input/output circuitry 1394 . Further, microprocessor 1392 , input/output circuitry 1394 , audio processing circuitry 1396 , image processor 1374 , and/or memory 1396 may be incorporated into a specially designed Application Specific Integrated Circuit (ASIC) 1391 .

8.2 Computer System Embodiment

Figure 14 is a block diagram illustrating a computer system 1400 on which one or more embodiments of the invention may be implemented. Computer system 1400 includes a bus 1402 or other communication mechanism for communicating information, and a processor 1404 coupled with bus 1402 for processing information. Computer system 1400 also includes main memory 1406 , such as random access memory (RAM) or other dynamic memory device, coupled to bus 1402 for storing information and instructions to be executed by processor 1404 . Main memory 1406 may also be used to hold temporary variables or other intermediate information during execution of instructions to be executed by processor 1404 . Computer system 1400 further includes a read only memory (ROM) 1408 or other static memory device coupled to bus 1402 to hold static information and instructions for processor 1404 . A storage device 1410, such as a magnetic or optical disk, is provided and coupled to bus 1402 for storing information and instructions.

Computer system 1400 can be coupled via bus 1402 to display screen 1412, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 1414 , including alphanumeric and other keys, is coupled to bus 1402 for communicating information and command selections to processor 1404 . Another type of user input device is cursor control 1416 , such as a mouse, trackball, or cursor direction keys, to communicate direction information and command selections to processor 1404 and to control cursor movement on display 1412 . The input device typically has degrees of freedom in two axes, a first axis (eg, X) and a second axis (eg, Y), allowing the device to specify a position on a plane. The computer system 1400 may further include an audio/video input device 1415 such as a microphone or camera to provide audible sound, still images or motion video, any of which may be processed using the above described embodiments.

Various processing techniques disclosed herein may be implemented to process data on computer system 1400 . According to one embodiment of the invention, those techniques are performed by computer system 1400 in response to processor 1404 executing one or more sequences of one or more instructions contained in main memory 1406 . These instructions may be read into main memory 1406 from another machine-readable medium, such as storage device 1410 . Execution of the sequences of instructions contained in main memory 1406 causes processor 1404 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.

The term "machine-readable medium" is used herein to refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented by the computer system 1400, for example, various machine-readable media are included to provide instructions for the processor 1401 to execute. Such a medium may take many forms, including but not limited to storage media and transmission media. Storage media includes non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks such as storage device 1410 . Volatile media includes dynamic memory such as main memory 1406 . Transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 1402 . Transmission media can also take the form of acoustic or light waves, such as those generated in radio-wave and infrared data communications devices. All such media must be tangible such that the instructions carried by the media can be detected by a physical mechanism that reads the instructions into a machine.

Common forms of machine-readable media include, for example, floppy disks, floppy disks, hard disks, magnetic tape, or any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tape, any other physical media having a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chips or cartridges, carrier waves described later, or any other computer-readable medium.

Various forms of machine-readable media may be involved in carrying one or more sequences of one or more instructions to processor 1404 for execution. For example, the instructions may initially be carried on a disk of the remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 1400 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on bus 1402 . Bus 1402 carries the data to main memory 1406 , from which processor 1404 can retrieve and execute instructions. The instructions received by main memory 1406 are optionally stored on storage device 1410 either before or after execution by processor 1404 .

Computer system 1400 also includes a communication interface 1418 coupled to bus 1402 . Communication interface 1418 provides a two-way data communication coupling to network link 1420 , which is connected to local area network 1422 . For example, communication interface 1418 may be an Integrated Services Digital Network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 1418 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 1418 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Network link 1420 typically provides data communication to other data devices through one or more networks. For example, network link 1420 may provide a link through local area network 1422 to a host 1424 or data device operated by an Internet Service Provider (ISP) 1426 . The ISP 1426, in turn, provides data communication services over a world-dimensional packet data communication network now commonly referred to as the "Internet" 1428. Local area network 1422 and Internet 1428 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 1420 and through communication interface 1418 , which carry the digital data to and from computer system 1400 , are exemplary forms of carrier wave transport of the information.

Computer system 1400 can send messages and receive data, including program code, over a network, network link 1420 and communication interface 1418 . In the Internet embodiment, the server 1430 can transmit the request code of the application program through the Internet 1428, the ISP 1426, the local area network 1422 and the communication interface 1418.

The received code may be executed by processor 1404 as it is received, and/or stored in storage device 1410, or other non-volatile storage for subsequent execution. In this manner, computer system 1400 may obtain application code in carrier wave form.

Data processed by embodiments of the program code described herein may be obtained from a variety of sources including, but not limited to, A/V input device 1415 , storage device 1410 and communication interface 1418 .

Claims (39)

1. A method of signal processing, said method comprising the steps of: For each signal matrix in multiple signal matrices, where each signal matrix contains elements describing different parts of the signal: Based on the magnitude of the effect that the kernel matrix expects to have on the signal matrix, adjust The magnitude of the effect achieved due to processing the signal based on the kernel matrix. 2. The method according to claim 1, wherein said based on the size of the effect that the kernel matrix expects to produce on the signal matrix, adjusting the size of the effect due to processing the signal based on the kernel matrix comprises: generating a modified kernel matrix for the signal matrix; and The modified kernel matrix is applied to the signal matrix to determine a new value for a first element of the signal matrix. 3. The method according to claim 1, wherein said based on the size of the effect that the kernel matrix expects to produce on the signal matrix, adjusting the size of the effect due to processing the signal based on the kernel matrix comprises: applying the kernel matrix to the signal matrix to generate an intermediate pixel value; and A weighted average of the intermediate pixel value and a first element of the signal matrix is determined. 4. The method of claim 1, wherein the effect of applying the kernel matrix enhances image contrast in the signal. 5. The method of claim 1, wherein applying the effect of the kernel matrix performs denoising on images in the signal. 6. The method of claim 1, further comprising determining a magnitude of an effect that the kernel matrix is expected to have on a signal matrix based on a signal-to-noise level associated with the signal. 7. The method of claim 1, further comprising determining a magnitude of an effect the kernel matrix is expected to have on the signal matrix based on properties of a lens used to acquire the signal described by the signal matrix. 8. The method according to claim 1, further comprising: based on the data characteristics in the signal matrix, determining the size of the expected effect of the core matrix on the signal matrix. 9. The method of claim 1, wherein determining the magnitude of the desired effect of the kernel matrix on a signal matrix is based on the location of pixels within the image sensor. 10. The method of claim 1, further comprising modifying the kernel matrix according to motion blur characteristics of the image data prior to processing the signal matrix based on the kernel matrix. 11. A device comprising: lens; an image capture device optically coupled to the lens, wherein the image capture device captures an image; and logic configured to process images based on a kernel matrix, wherein the logic is operable to: For each signal matrix in multiple signal matrices, where each signal matrix contains elements describing different parts of the signal: Based on the magnitude of the effect that the kernel matrix expects to have on the signal matrix, Adjusts the size of the effect achieved due to processing the signal based on the kernel matrix. 12. The apparatus of claim 11 , wherein the logic operable to adjust the magnitude of an effect achieved as a result of processing the signal matrix based on a kernel matrix is operable to: generating a modified kernel matrix for the signal matrix; and The modified kernel matrix is applied to the signal matrix to determine a new value for a first element of the signal matrix. 13. The apparatus of claim 11 , wherein the logic operable to adjust the magnitude of an effect achieved as a result of processing the signal matrix based on a kernel matrix is operable to: applying the kernel matrix to the signal matrix to generate an intermediate pixel value; and A weighted average of the intermediate pixel value and a first element of the signal matrix is determined. 14. The apparatus of claim 11, wherein the effect of applying the kernel matrix enhances image contrast in the signal. 15. The apparatus of claim 11, wherein applying the effect of the kernel matrix performs denoising on an image in a signal. 16. The apparatus of claim 11, wherein a magnitude of an effect the kernel matrix is expected to have on a signal matrix is based on a signal-to-noise ratio associated with the signal. 17. The apparatus of claim 16, wherein the magnitude of the effect that the kernel matrix is expected to produce on the signal matrix is further based on properties of a lens used to acquire the signal described by the signal matrix. 18. The apparatus of claim 17, wherein the size of the effect that the kernel matrix is expected to produce on the signal matrix is further based on the characteristics of the elements in the signal matrix. 19. The apparatus of claim 16, wherein the magnitude of the effect that the kernel matrix is expected to produce on a signal matrix is further based on characteristics of elements in the signal matrix. 20. The apparatus of claim 11, wherein the magnitude of the effect that the kernel matrix is expected to produce on the signal matrix is based on properties of a lens used to acquire the signal described by the signal matrix. 21. The apparatus of claim 11, wherein the magnitude of the effect that the kernel matrix expects to have on a signal matrix is based on characteristics of elements in the signal matrix. 22. The apparatus of claim 11, wherein the logic is further operable to: determine a magnitude of an effect that the kernel matrix is expected to have on a signal matrix based on characteristics of elements in the signal matrix. 23. The apparatus of claim 22, wherein the logic is further operable to calculate a metric used to determine a magnitude of an effect that the kernel matrix is expected to have on a signal matrix. 24. The apparatus of claim 23, wherein the logic is further operable to: determine a metric based on differences between values of elements of a signal matrix. 25. The apparatus of claim 24, wherein the logic is further operable to: determine a metric based on a standard deviation between elements of a signal matrix. 26. The apparatus of claim 23, wherein the logic is further operable to: determine a metric based on a histogram of element values of a signal matrix. 27. The apparatus of claim 23, wherein the logic is further operable to: determine a metric by applying an edge detection mask to the signal matrix. 28. The apparatus of claim 23, wherein the logic is further operable to: determine a metric based on an entropy of values in a signal matrix. 29. The apparatus of claim 11, wherein the logic is further operable to: modify the kernel matrix based on motion blur characteristics of the image data prior to processing the signal matrix based on the kernel matrix. 30. A device comprising: lens; an image capture device optically coupled to the lens, wherein the image capture device captures images from the lens; and logic configured to simultaneously process image data to correct both optical aberrations due to the lens and motion blur. 31. The apparatus of claim 30, wherein to correct for the optical aberration, the logic is configured to adjust an amount of sharpening of the image data based on a signal-to-noise ratio of the image. 32. The apparatus of claim 30, wherein to correct for the optical aberration, the logic is configured to adjust an amount of sharpening of the image data based on a lens shading profile of the lens. 33. The apparatus of claim 30, wherein to correct for the optical aberration, the logic is configured to adjust an amount of sharpening of the image data based on local characteristics of the image data. 34. The device of claim 30, wherein the logic configured to process the image data is configured to correct the motion blur based on an estimated aberration point distribution function due to motion of the device. 35. The apparatus of claim 30, wherein the logic configured to process the image data is configured to: correct the motion blur based on an estimated image translation detected between a plurality of frames of the image . 36. The apparatus of claim 23, wherein the logic is further operable to: determine a metric based on a plurality of gain functions. 37. The apparatus of claim 36, wherein the plurality of gain functions are applied to the metric based on a local estimate of an amount of light in an image window. 38. The apparatus of claim 37, wherein the plurality of gain functions are weighted based on a local estimate of the amount of light in an image window. 39. The apparatus of claim 23, wherein the logic is further operable to: determine a metric based on a distance calculation applied to the signal matrix.

Images