JP7387411B2 - Image processing device, image processing method, and imaging device - Google Patents

Description

The present invention relates to an image processing device and an image processing method, and particularly relates to an image processing device and an image processing method provided in an imaging device having a Bayer array, and an imaging device.

Conventionally, solid-state imaging devices such as CCDs and CMOSs used in digital cameras, video cameras, and the like generally output only information regarding the brightness of light. Therefore, in a single-chip camera with one solid-state image sensor, in order to obtain color information, a color filter is placed in front of the CCD that allows only one color component to pass through for each pixel included in the CCD. It is configured so that each pixel portion outputs a corresponding color component. For example, in the case of a primary color filter, there are three types of color components: the three primary colors R (red), G (green), and B (blue). Several types of arrangement of each color component of these color filters have been proposed. A typical example is the Bayer array. In this Bayer array, G, which contributes a large proportion to the luminance signal, is arranged in a checkerboard pattern, and R and B are arranged in the remaining locations in a checkerboard pattern. When a signal is output from a CCD covered with a color filter such as a Bayer array, color information for only one color among RGB color components is obtained for each pixel. Therefore, an interpolation process is performed in which the image processing unit estimates and calculates information for the remaining two colors for each pixel from the color signal values of surrounding pixels to obtain the remaining color information.

An example of the above-mentioned interpolation process is the technique disclosed in Patent Document 1.

Patent Document 1 discloses an image processing device including a data interpolation unit that acquires Bayer data after gamma correction. In the data interpolation section, for each pixel part arranged in a two-dimensional matrix, the presence or absence of an edge (a part where the brightness value changes rapidly) near the pixel of interest and the direction of the edge are determined to be the pixel of interest and the direction of the edge. It is detected by arithmetic processing using color signals of neighboring pixels. Then, the data interpolation section performs data interpolation processing according to the presence or absence of this edge and the directionality of the detected edge, and outputs an RGB signal for each pixel.

There is another technique disclosed in Patent Document 2 that finds edges near the pixel of interest and performs interpolation processing.

Japanese Patent Application Publication No. 2008-125132 (published on May 29, 2008) Japanese Patent Application Publication No. 2009-65671 (published on March 26, 2009)

All of the above-mentioned conventional techniques have problems. For example, in the interpolation process disclosed in Patent Document 1, a technique is disclosed that uses a threshold value to determine whether there is no edge near the pixel of interest and a flat part with a gentle luminance gradient in the edge detection process. be done. Specifically, the maximum value of the edge detection component (in the direction intersecting the edge) is compared with a threshold, and if the edge component is smaller than the threshold, it is determined to be a flat portion. However, the output data from the image sensor includes shot noise and the like even if it is a flat portion. Regarding shot noise, the amount of noise differs between a high-luminance part and a low-luminance part even within the same frame. Therefore, when the brightness average is high, shot noise exceeds the threshold. Therefore, there is a possibility that a flat portion may be erroneously detected as an edge portion.

Furthermore, in the interpolation process disclosed in Patent Document 2, gradient calculation is performed within a reference pixel area of 7×7 pixels in order to calculate edge directionality data. To realize this, a line memory (also called a line buffer) for seven lines is required. Furthermore, in calculating the gradient in the diagonal direction, there is a division by 7 at the end of the calculation formula, but divisions other than powers of 2 cannot be calculated by bit shifting. Therefore, a separate division circuit is required to calculate the gradient in the diagonal direction, which increases the circuit scale.

An object of one aspect of the present invention is to realize an image processing device, an image processing method, and an imaging device that can discriminate flat portions and edge portions more accurately than before and perform color interpolation processing with high precision.

In order to solve the above problems, an image processing device according to one aspect of the present invention generates directionality data that quantifies the directionality of edges in a pixel of interest and neighboring pixels of Bayer array image data for each of a plurality of directions. a direction detection unit that detects an edge direction using the method; a determination unit that determines whether or not there is a difference in the directional data between the directions; and a noise amount calculation unit that calculates the amount of noise of the neighboring pixels; a comparison unit that compares the maximum value of the directional data determined to have the difference with the threshold value including the noise amount; and (i) the maximum value is the threshold value, according to the comparison result of the comparison unit. (ii) If the maximum value is less than or equal to the threshold, the color interpolation of (i) is performed on the pixel of interest. and a color interpolation unit that performs different color interpolation.

In order to solve the above problems, an imaging device according to one aspect of the present invention includes the above-described image processing device and an imaging element having a color filter having a Bayer array.

In order to solve the above problems, an image processing method according to one aspect of the present invention uses directional data that quantifies the directionality of edges in a pixel of interest and neighboring pixels of Bayer array image data for each of a plurality of directions. a direction detection step of detecting an edge direction using the method, a determination step of determining whether there is a difference in the directional data between the directions, and a noise amount calculation step of calculating the amount of noise of the neighboring pixels; a comparison step of comparing the maximum value of the directional data determined to have the difference with a threshold value including the noise amount; and (i) the maximum value is the threshold value, according to the comparison result of the comparison step. (ii) If the maximum value is less than or equal to the threshold, the color interpolation of (i) is performed on the pixel of interest. and a color interpolation step of performing different color interpolations.

According to one aspect of the present invention, it is possible to realize an image processing device, an image processing method, and an imaging device that can perform color interpolation processing with higher precision than before.

1 is a diagram showing the main configuration of an imaging device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a detailed configuration of an image processing unit that is a component of the imaging device shown in FIG. 1. FIG. It is a diagram showing Bayer data shown two-dimensionally and explaining the words "horizontal direction, vertical direction, diagonally upper right direction, and diagonally upper left direction" used in this specification. FIG. 3 is a diagram illustrating a definition regarding the direction of an edge. FIG. 3 is a diagram illustrating a basic provisional determination method of a provisional determination unit provided in the image processing unit shown in FIG. 2; 3 is a diagram illustrating a method of calculating a noise amount performed by a noise amount calculating section provided in the image processing section shown in FIG. 2. FIG. 3 is a diagram illustrating a threshold value used in a determination performed in an edge determination section provided in the image processing section shown in FIG. 2. FIG. 3 is a diagram illustrating a determination procedure performed in an edge determination section provided in the image processing section shown in FIG. 2. FIG. 3 is a diagram illustrating a method of calculating a noise amount performed by a noise amount calculating section provided in the image processing section shown in FIG. 2. FIG. 3 is a diagram conceptually showing color interpolation processing performed in a color interpolation section provided in the image processing section shown in FIG. 2 when the edge direction is diagonally upward to the right. FIG. FIG. 3 is a diagram illustrating a case where the edge direction determined (confirmed) by the edge determination section provided in the image processing section shown in FIG. 2 is the diagonally upper left direction ( _DNE is the smallest in edge detection). 3 is a diagram illustrating a gradient method performed by a color interpolation section 230 provided in the image processing section shown in FIG. 2. FIG. 3 is a diagram illustrating a color correlation method performed by a color interpolation section 230 provided in the image processing section shown in FIG. 2. FIG. 3 is a flowchart showing an operation flow (image processing flow) of the image processing section shown in FIG. 2. FIG. 15 is a detailed flow diagram of a part of the operation flow shown in FIG. 14. FIG. 15 is a detailed flow diagram of a part of the operation flow shown in FIG. 14. FIG. 7 is a flowchart showing an operation flow of an image processing unit that is a component of an imaging device according to another embodiment of the present invention. 18 is a diagram illustrating a part of the operation flow shown in FIG. 17. FIG. 18 is a diagram illustrating a part of the operation flow shown in FIG. 17. FIG. 18 is a diagram illustrating a part of the operation flow shown in FIG. 17. FIG.

Hereinafter, one embodiment of the present invention will be described using FIGS. 1 to 16.

(1) Imaging device FIG. 1 is a diagram showing the main configuration of an imaging device 1 of this embodiment. The main configuration of the imaging device 1 will be explained using FIG. 1. Note that the configuration example of the imaging device 1 according to this embodiment is the same as the schematic configuration diagram of the imaging device in Patent Document 1. This means that although the signal processing in the image processing unit 125 is different from that in Patent Document 1, the technology described in this specification is applicable to general imaging devices.

The imaging device 1 includes an imaging optical system 100 (imaging optical system), an actuator system 110, an electronic flash 115 for object illumination, an AF (autofocus) auxiliary light source 116, a control section 120, and a display section 131. , an operation section 132, and a flash memory 133.

The imaging optical system 100 includes a first lens group 101 , an aperture/shutter 102 , a second lens group 103 , a third lens group 105 , an optical low-pass filter 106 , and an image sensor 107 . A first lens group 101 arranged at the tip of the imaging optical system 100 is held by a lens barrel so as to be movable in the optical axis direction. The diaphragm/shutter 102 not only adjusts the amount of light during photographing by adjusting its aperture diameter, but also functions as a shutter for adjusting the exposure time during still image photographing. The second lens group 103 moves forward and backward in the optical axis direction integrally with the diaphragm/shutter 102, and has a variable magnification effect (zooming function) in conjunction with the movement of the first lens group 101 back and forth. The third lens group 105 is a focus lens that adjusts focus by moving back and forth in the optical axis direction. The optical low-pass filter 106 is an optical element for reducing false colors and moiré in captured images. The image sensor 107 is a solid-state image sensor that includes, for example, a two-dimensional CMOS (complementary metal oxide semiconductor) photosensor (photodiode section) and peripheral circuitry, and is arranged on the imaging plane of the imaging optical system 100.

The actuator system 110 includes a zoom actuator 111, an aperture shutter actuator 112, and a focus actuator 114. The zoom actuator 111 rotates a cam barrel (not shown) to move the first lens group 101 and the second lens group 103 in the optical axis direction, thereby performing a zooming operation. The diaphragm shutter actuator 112 controls the aperture diameter of the diaphragm/shutter 102 to adjust the amount of photographing light, and also controls the exposure time when photographing still images. The focus actuator 114 moves the third lens group 105 in the optical axis direction to perform a focus adjustment operation.

The electronic flash 115 is used during photography, and is a flash lighting device using a xenon tube or a lighting device equipped with a continuous light emitting LED (light emitting diode).

The AF auxiliary light source 116 projects an image of a mask having a predetermined aperture pattern onto the field through a projection lens. This improves focus detection ability for low-luminance objects or low-contrast objects.

The control unit 120 includes a CPU 121, which is a central processing unit, and various other components described below.

The CPU 121 has a control center function that manages various controls. The CPU 121 includes an arithmetic unit, a ROM (read only memory), a RAM (random access memory), an A (analog)/D (digital) converter, a D/A converter, a communication interface circuit, and the like. The CPU 121 drives various components within the imaging device 1 according to a predetermined program stored in the ROM, and executes a series of operations such as AF control, imaging processing, image processing, and recording processing. The CPU 121 controls defective pixel detection and defective pixel correction in this embodiment.

The electronic flash control unit 122 is a circuit that is included in the control unit 120 and controls lighting of the electronic flash 115 in synchronization with the photographing operation according to control instructions from the CPU 121.

The auxiliary light source driving section 123 is a circuit that is included in the control section 120 and controls lighting of the AF auxiliary light source 116 in synchronization with the focus detection operation according to a control command from the CPU 121.

The image sensor driving section 124 is a circuit that is included in the control section 120 and controls the imaging operation of the image sensor 107, and also performs A/D conversion on an image signal acquired from the image sensor 107 and transmits the A/D converter to the CPU 121.

The image processing unit 125 (image processing device) is included in the control unit 120 and performs processing such as gamma correction, scaling, and JPEG (Joint Photographic Experts Group) compression of the image acquired by the image sensor 107 according to control instructions from the CPU 121. This is the circuit that does this. Further, the image processing unit 125 performs a process of generating a captured image acquired by the image sensor 107. The image signal of the captured image is subjected to recording processing and display processing. The image processing unit 125 also performs image processing including color interpolation processing, which will be described later.

The focus drive unit 126 is a circuit that is included in the control unit 120 and drives the focus actuator 114 based on the focus detection result in accordance with the control command from the CPU 121 to move the third lens group 105 in the optical axis direction to adjust the focus. It is.

The aperture shutter drive unit 128 is a circuit that is included in the control unit 120 and drives the aperture shutter actuator 112 in accordance with a control command from the CPU 121 to control the aperture diameter of the aperture shutter 102 .

The zoom drive unit 129 is a circuit that is included in the control unit 120 and drives the zoom actuator 111 in accordance with control commands from the CPU 121 and in response to zoom operation instructions from the photographer.

The above is the configuration of the control unit 120 in this embodiment.

The display unit 131 has a display device such as an LCD (liquid crystal display), and displays information regarding the shooting mode of the imaging device 1, a preview image before shooting, a confirmation image after shooting, and a focus state display image when detecting focus. etc. will be displayed.

The operation unit 132 includes a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, etc. as operation switches, and outputs an operation instruction signal to the CPU 121.

The flash memory 133 is a recording medium that is removably attached to the imaging device 1, and records captured image data and the like.

(2) Image Processing Unit As described above, the imaging device 1 of this embodiment includes the image sensor 107 that can image a subject, and the A/D converter (A/D converter) that converts the image data from the image sensor 107 into A/D. (not shown) and an image processing section 125 serving as an image processing device that performs various signal processing on the digital signal from this A/D conversion section.

In the image sensor 107, subject light passes through a predetermined color filter and enters a photodiode section, and the photodiode section photoelectrically converts the signal into an analog electrical signal representing the size of a color corresponding to each pixel and outputs the signal. Note that the color filter in this case has a Bayer arrangement using the three primary colors of RGB.

The A/D converter converts the analog electrical signal output from the image sensor 107 into a digital signal (hereinafter referred to as Bayer data) and outputs the digital signal.

The image processing section 125 receives Bayer data (image data) output from the A/D conversion section. Then, in addition to various types of processing described later, various image processing such as white balance and gamma correction (not shown) is performed, and then image data including all RGB data for each pixel portion is output.

The image processing section 125 will be explained below using FIG. 2.

(2.1) Configuration of Image Processing Unit FIG. 2 is a block diagram showing a detailed configuration of the image processing unit 125, which is a component of the imaging device 1 shown in FIG. 1.

The image processing unit 125 determines an edge direction to be used as a color interpolation direction, and performs a process of performing color interpolation using the determined edge direction. To this end, the image processing section 125 includes an edge preliminary determination section 210 and an edge determination section 220 that determine the edge direction, and a color interpolation section 230 that performs color interpolation. More specifically, the edge direction is preliminarily determined (tentatively determined) in the edge preliminary determination section 210, the edge direction is determined (determined) in the edge determination section 220, and color interpolation is performed in the color interpolation section 230.

Here, when performing color interpolation, better results can be obtained by performing interpolation using data in a direction along edges (portions where brightness values change rapidly in an image). On the other hand, if color interpolation is performed using data in a direction that intersects an edge, false colors may occur. If the area has no edges, color interpolation can be performed using data of surrounding pixels. Further, in the case of the Bayer array, the contribution rate to the luminance component of the G pixel is twice as large as that of the R and B pixels. Therefore, first, G pixels on R and B pixels are interpolated. Therefore, it is assumed that the edge preliminary determination section 210 and the edge determination section 220 operate when the pixel of interest is an R or B pixel.

<Edge preliminary determination section>
The edge preliminary determination unit 210 preliminarily determines the edge direction.

The edge preliminary determination section 210 includes a directionality data calculation section 211 and a direction detection section 212, as shown in FIG.

The directional data calculation unit 211 calculates edge component values (directional data). The "directional data" here refers to the degree of approximation between the specified direction (multiple directions), including at least diagonal directions, and the actual edge direction near the pixel of interest, for each pixel of the target Bayer data (image data). The larger the value, the smaller the value. In short, the directional data is data that quantifies the directionality of edges in a pixel of interest and neighboring pixels of Bayer array image data for each of a plurality of directions. The directional data calculation unit 211 calculates directional data for each of a plurality of predetermined specified directions. Hereinafter, what is explained with the initial letter D is directional data.

The directional data calculation unit 211 calculates directional data D _H , D _V , D _NE , and D _NW defined below, although detailed calculation processing will be described later.
_DH : An analysis result of the gradient component in the horizontal direction, which is an index for detecting the presence or absence of an edge that intersects with the horizontal direction.D _V : An index that detects the presence or absence of an edge that intersects with the vertical direction. Analysis result of the gradient component in the vertical direction D _NE : Analysis result of the gradient component in the diagonally upper right direction, which is an index for detecting the presence or absence of an edge that intersects with the diagonally upper right direction D _NW : With respect to the diagonally upper left direction Analysis results of the gradient component in the diagonally upper left direction, which is an index for detecting the presence or absence of intersecting edges.

Note that the terms "horizontal direction" and "vertical direction" used here correspond to the horizontal direction and vertical direction of the photodiode array of the solid-state image sensor of the image sensor 107 in FIG. 1. That is, in the two-dimensional _Bayer data 3001 shown in FIG. 3, R ₂₂ is determined as a pixel of interest, the left-right direction of the page corresponds to the "horizontal direction", and the up-down direction of the page corresponds to the "vertical direction". Similarly, in the two-dimensional Bayer data 3002 shown in _FIG . Corresponds to the direction.

Specifically, the directional data calculation unit 211 performs gradient calculation within a reference pixel area of 5 (rows)×5 (columns) pixels. Therefore, line memory for five lines is required. When the pixel of interest is R in the Bayer array, there is G pixel data in the nearest vertical and horizontal directions, and B pixel data in the diagonal direction. Directionality data is obtained by performing gradient analysis for each direction. If the gradient amount is large, it indicates that there is an edge that intersects in that direction, and if the gradient amount is small or zero, it means that the direction is along the edge or it is a flat part with no edges. .

Gradient analysis when the pixel of interest is R is obtained using the following equation (1). In this embodiment, diagonal analysis can be performed only by addition/subtraction and bit shift processing. Note that if the pixel of interest is B, it can be determined using the same method by simply replacing R and B in the formula and text.

In the above equation (1), the horizontal gradient component (D _H ) is the absolute difference between the horizontal G pixels closest to the pixel of interest, and the absolute difference between the pixel of interest and the nearest R pixel in the horizontal direction. (left side/right side) and the absolute difference between the nearest B pixels in the horizontal direction (upper side/lower side). The absolute difference value of the B pixel is divided by 2 in consideration of data components in the diagonal direction, which will be described later.

In the above equation (1), the vertical gradient component (D _V ) is the absolute difference between the G pixels in the vertical direction closest to the pixel of interest, and the absolute difference between the pixel of interest and the R pixel in the vertical direction (upper side).・lower side) and the absolute difference between the closest B pixels in the vertical direction (left side/right side). The absolute difference value of the B pixel is divided by 2 in consideration of data components in the diagonal direction, which will be described later.

In the above equation (1), the gradient component in the diagonally upper right direction (D _NE ) is the absolute difference between the G pixels closest to the pixel of interest in the diagonally upper right direction, and the gradient component between the pixel of interest and the nearest diagonally upper right direction. It is assumed that the absolute difference of the R pixel (upper right and lower left) and the absolute difference of the nearest B pixel in the diagonally upper right direction are added.

In the above equation (1), the gradient component in the diagonally upward left direction (D _NW ) is the absolute difference between the G pixels closest to the pixel of interest in the diagonally upward left direction, and the gradient component between the pixel of interest and the nearest diagonally upward left direction. It is assumed that the absolute difference of the R pixel (upper left and lower right) and the absolute difference of the nearest B pixel in the diagonally upper left direction are added.

Note that when calculating the gradient in the diagonal direction, there is no G pixel in the diagonal direction of the pixel of interest, and the calculation is performed using the nearest G pixel instead, so the accuracy is lower than in the horizontal and vertical directions. . Therefore, since the accuracy of edge detection may deteriorate, the coefficient A is multiplied with respect to D _NE and D _NW , which are the gradient analysis results in the diagonal direction. This controls edge detection accuracy in the diagonal direction. The larger the coefficient A is, the more edges are present in the diagonal direction, which means that color interpolation is performed along the horizontal and vertical directions.

Through the above procedure, the directional data calculation unit 211 calculates directional data (D _H , D _V , D _NE , D _NW ) in each direction.

The direction detection unit 212 acquires the directional data (D _H , _DV , D _NE , D _NW ) calculated by the directional data calculation unit 211 and determines the pixel of interest and its neighboring pixels (nearby pixels passing through the pixel of interest). (hereinafter collectively referred to as the vicinity of the pixel of interest), the presence or absence of an edge and the edge direction are detected. As mentioned above, good results can be obtained by performing color interpolation in the edge direction (direction along the edge). Therefore, the direction detection unit 212 identifies which direction is along the edge from each directional data. The direction along the edge is the direction in which the gradient component is small. In short, the direction detection unit 212 specifies (selects) the direction data indicating the minimum value among D _H , D _V , D _NE , and D _NW . Details will be described later. Note that each directional data may be acquired from the CPU 121 shown in FIG. 1 or from the directional data calculation unit 211.

Here, the "edge direction (direction along the edge)" within the 3 rows x 3 columns pixel area shown in FIG. 4 will be described. Here, the possible values of a pixel will be explained as 0 to 255 (8-bit gradation). In the pixel area A on the left side of FIG. 4, three pixels (L ₀₀ , L ₁₀ , L ₂₀ ) in the first column located on the leftmost side have a pixel value of "0", and a total of 6 pixels in the second and third columns have a pixel value of "0". Assume that the pixels (L ₀₁ , L ₁₁ , L ₂₁ , L ₀₂ , L ₁₂ , L ₂₂ ) have a pixel value of “255”. In this pixel area A, there is an edge between the three pixels in the first column (L ₀₀ , L ₁₀ , L ₂₀ ) and the three pixels in the second column (L ₀₁ , L ₁₁ , L ₂₁ ). That is, in the pixel area A, the direction along the vertical direction of the paper surface is the "edge direction (direction along the edge)" as shown in the figure. On the other hand, the direction along the horizontal direction of the page is the direction that intersects the edge. On the other hand, in pixel region B on the right side of FIG. 4, the direction definition is opposite to that of pixel region A. That is, in the pixel area B on the right side of FIG. 4, the three pixels (L ₀₀ , L ₀₁ , L ₀₂ ) in the first row located at the top have a pixel value of "0", and the pixel values in the second and third rows are "0". It is assumed that a total of six pixels (L ₁₀ , L ₁₁ , L ₁₂ , L ₂₀ , L ₂₁ , L ₂₂ ) have a pixel value of “255”. In this pixel region B, there is an edge between the three pixels (L ₀₀ , L ₀₁ , L ₀₂ ) in the first row and the three pixels (L ₀₁ , L ₁₁ , L ₂₁ ) in the second row. That is, in the pixel region B, as shown in the figure, the direction along the horizontal direction of the paper is the "edge direction (direction along the edge)" and the direction along the vertical direction of the paper is the direction intersecting the edge.

As described above, the edge direction is determined by the edge preliminary determining section 210, but for the reason described later, in this embodiment, the edge determining section 220, which will be explained below, is further provided to determine the previously determined edge direction. It is determined whether the color interpolation unit 230 is suitable for use in color interpolation processing.

<Edge determination section>
The edge determination unit 220 determines whether or not the edge direction preliminarily determined by the edge preliminary determination unit 210 is suitable for use in color interpolation processing by the color interpolation unit 230 at the subsequent stage. For this purpose, the edge determination section 220 includes a temporary determination section 221 (determination section), a noise amount calculation section 222, and an edge determination section 223 (comparison section), as shown in FIG.

The tentative determination unit 221 tentatively determines whether the vicinity of the pixel of interest is a flat portion or an edge portion. Note that in order to determine whether or not it is an edge portion in an edge determination section 223, which will be described later, the determination result in this provisional determination section 221 is regarded as a provisional determination result.

For example, the provisional determination unit 221 acquires directional data in each direction that is the calculation result of the edge preliminary determination unit 210, and if the maximum value and minimum value of these directional data are equal, Tentatively determined to be a flat part. As mentioned above, the direction corresponding to the directional data indicating the maximum value can be regarded as the direction that intersects the edge, and the direction corresponding to the directional data indicating the minimum value can be regarded as the direction along the edge. Can be done. Here, the maximum value may be specified by the edge preliminary determination unit 210, similar to the minimum value.

The basic provisional determination method of the provisional determination unit 221 will be explained using FIG. 5. In FIG. 5, for convenience of explanation, a pixel area of 3 rows x 3 columns is used, and the possible values of the pixels are 0 to 255 (8-bit gradation).

In the example of a) in which the pixel area has edges in the vertical (vertical) direction shown in FIG. 5, the directionality data in each direction is determined by the absolute value difference as shown below.
D _V = |L ₀₁ -L ₂₁ |=0
D _H = |L ₁₀ -L ₁₂ |=255
D _NE =|L ₀₂ -L ₂₀ |=255
D _NW = | L ₀₀ - L ₂₂ | = 255
At this time, since _DV is the smallest, it can be considered that an edge exists along the vertical (vertical) direction of the pixel area, that is, the edge direction is the vertical (vertical) direction. In the example a), since D _H =D _NE =D _NW , any of these three directions may be used to intersect the edge. In an example including edges a) to c) shown in FIG. 5, there is always directional data indicating a minimum value and directional data indicating a maximum value.

On the other hand, in the examples shown in d) to f) in FIG. 5, the above calculation formula cannot find the directional data indicating the minimum value and the directional data indicating the maximum value. However, as will be described later, by expanding the reference pixel area to an area of 5 rows x 5 columns and considering the difference from the pixel of interest (center), the examples shown in d) to f) in FIG. It is possible to detect the edge direction (that is, the fact that an edge is included) even if

Further, in the case of a flat portion shown in examples g) and h) shown in FIG. 5, the directionality data in each direction is determined as follows.
D _V = |L ₀₁ -L ₂₁ |=0
D _H = |L ₁₀ -L ₁₂ |=0
D _NE =|L ₀₂ -L ₂₀ |=0
D _NW = |L ₀₀ -L ₂₂ |=0
At this time, all the directional data show the same value, indicating that there is no difference. In this way, if there is no difference in the directional data depending on the direction, in other words, if the minimum value and the maximum value of the directional data are the same, it can be determined that it is a flat portion.

Note that if it is determined that the area is a flat area, color interpolation is performed using data of surrounding pixels as described later.

In this manner, the tentative determination unit 221 can tentatively determine whether or not the vicinity of the pixel of interest is an edge portion.

Incidentally, the output data from the image sensor 107 in FIG. 1 includes some noise even in a flat portion. For example, values detected at a flat portion may include shot noise values. In this case, even if the vicinity of the pixel of interest is originally a flat portion, the maximum value and minimum value of the calculated directional data may not be equal, and there is a risk that it may be erroneously determined to be an edge portion. If it is erroneously determined to be an edge portion and demosaic processing is performed, there is a risk that noise components will be emphasized.

Therefore, in order to suppress such erroneous determinations, in this embodiment, a value obtained by adding the amount of noise of neighboring pixels and a predetermined determination parameter is compared with the maximum value of the directional data. This determines whether the vicinity of the pixel of interest is (truly) an edge portion. If it is determined that the vicinity of the pixel of interest is an edge portion, demosaicing is performed while leaving edge components in a color interpolation process that uses the edge direction as a color interpolation direction, as will be described later. On the other hand, if it is determined that the vicinity of the pixel of interest is a flat portion, demosaicing may be performed to suppress noise components using a low frequency pass filter (LPF) method such as an average value filter. In this embodiment, the noise amount calculation section 222 shown in FIG. 2 calculates the noise amount described above, and the edge determination section 223 shown in FIG. 2 performs the above comparison.

The noise amount calculation unit 222 calculates the noise amount VAR of neighboring pixels.

Specifically, the noise amount calculation unit 222 calculates the noise amount of neighboring pixels using, for example, the average deviation of the following equation (2).

Since the average deviation is the average value of the absolute difference between the average value of the reference pixels and each pixel, it can be easily compared with the directional data. This is because the directionality data does not use an average value, but uses an absolute difference value between a certain pixel and another pixel.

Note that when the amount of noise is taken as the amount of variation in data, the standard deviation and variance shown in the following equation (3) are often used in statistics.

However, the standard deviation and variance described above require a square calculation and a square root calculation, which increases the amount of calculation. Therefore, in this embodiment, the amount of noise is determined using the average deviation in the above equation (2) instead of using the standard deviation and variance.

The procedure for calculating the amount of noise of neighboring pixels by the noise amount calculation unit 222 will be described below using an example. Incidentally, the amount of noise of neighboring pixels is determined in the direction (direction along the edge) corresponding to the directional data from which the previously determined minimum value was calculated. This is because the direction that intersects the edge contains an edge component amount, which may cause erroneous determination.

(When D _H is minimum)
Below, using FIG. 6 as an example where the pixel of interest is R, we will explain the amount of noise when the direction corresponding to the directional data for which the minimum value is calculated is the horizontal direction (that is, D _H is the minimum). An example of finding .

FIG. 6 shows an image area (R22 located at the center is the pixel of interest) (F) that is referred to when calculating the amount of noise.

The noise amount calculation unit 222 first calculates the average luminance value (B1 _AVE , G2 _AVE , R2 _AVE , B3 _AVE ) of each row using the pixel shown in (P1) in FIG. 6 of the reference image (F). It is determined based on each equation (4) below.

Further, the noise amount calculation unit 222 calculates the absolute value difference between the average value of brightness (B1 _AVE , G2 _AVE , R2 _AVE , B3 _AVE ) and the brightness of each same color pixel as the number of pixels, based on each formula (5) below. The noise amount H _var is obtained by adding all the noise values (B1 _var , G2 _var , R2 _var , B3 _var ) of each row obtained by dividing by .

(When _DV is minimum)
Also, in the reference pixel (F) shown in FIG. 6, when calculating the amount of noise when the direction corresponding to the directional data for which the minimum value has been calculated is the vertical direction (that is, _DV is the minimum), the following is as follows. explain.

The noise amount calculation unit 222 uses the pixels shown in (P2) in FIG. 6 of the reference image (F) to calculate the average luminance value (B1 _AVE , G2 _AVE , R2 _AVE , B3 _AVE ) of each row as follows. It is calculated based on each equation (6).

Further, the noise amount calculation unit 222 calculates the absolute value difference between the average value of brightness (B1 _AVE , G2 _AVE , R2 _AVE , B3 _AVE ) and the brightness of each pixel of the same color as the number of pixels based on each formula (7) below. The noise amount V _var is obtained by adding all the noise values (B1 _var , G2 _var , R2 _var , B3 _var ) of each row obtained by dividing by .

(When _DNE is minimum)
Also, in the reference pixel (F) shown in FIG. 6, when calculating the amount of noise when the direction corresponding to the directional data for which the minimum value has been calculated is the diagonally upper right direction (that is, D _NE is the minimum), This will be explained below.

The noise amount calculation unit 222 uses the pixels shown in (P3) in FIG. 6 of the reference image (F) to calculate the average luminance value (G1 _AVE , B2 _AVE ) of each column arranged diagonally upward to the right. , R2 _AVE , G3 _AVE ) are determined based on the following equations (8).

Further, the noise amount calculation unit 222 calculates the absolute value difference between the average value of brightness (G1 _AVE , B2 _AVE , R2 _AVE , G3 _AVE ) and the brightness of each same color pixel based on the following equations (9) as the number of pixels. The noise amount NE _var is obtained by adding all the noise values (G1 _var , B2 _var , R2 _var , G3 _var ) of each row obtained by dividing by .

(When D _NW is minimum)
In addition, regarding the case where the direction corresponding to the edge directionality data for which the minimum value is calculated is the diagonally upper left direction (that is, the D _NW is the minimum) in the reference pixel (F) shown in FIG. , explained below.

The noise amount calculation unit 222 uses the pixels shown in (P4) in FIG. 6 of the reference image (F) to calculate the average luminance value (G1 _AVE , B2 _AVE) of each column arranged diagonally upward to the right. , R2 _AVE , G3 _AVE ) are determined based on the following equations (10).

Further, the noise amount calculation unit 222 calculates the absolute value difference between the average value of brightness (G1 _AVE , B2 _AVE , R2 _AVE , G3 _AVE ) and the brightness of each pixel of the same color as the number of pixels, based on each formula (11) below. The noise amount NW _var is obtained by adding all the noise values (G1 _var , B2 _var , R2 _var , G3 _var ) of each row obtained by dividing by .

According to the above calculation procedure, the noise amount calculation unit 222 calculates the noise amount VAR (that is, any one of the above-mentioned H _var , V _var , NE _var and NW _var ) of the neighboring pixels.

The edge determination unit 223 compares the sum of the noise amount VAR of the neighboring pixels calculated by the noise amount calculation unit 222 and a predetermined determination parameter α with the maximum value of the directional data, and determines whether the vicinity of the pixel of interest is Determine whether it is an edge part or a flat part.

Specifically, the edge determination unit 223 uses the following relational expression;
Maximum value of directional data>VAR+α
Determine whether the following is satisfied. If the relational expression is satisfied, the area around the pixel of interest is determined (determined) to be an edge portion, and if the relational expression is not satisfied, the area around the pixel of interest is determined (determined) to be a flat area.

As an example, if the direction corresponding to the directionality data for which the maximum value was calculated (the direction intersecting the edge direction) is perpendicular (that is, the _DV is maximum), the edge determination unit 223 calculates the following relational expression;
D _V > H _var + α
Determine whether the following is satisfied. If the condition is satisfied, the pixel of interest is determined to be an edge portion, and if the condition is not satisfied, the pixel of interest is determined to be a flat portion.

Here, the determination parameter α is a threshold value for distinguishing between the amount of noise and the edge component. If the image is an ideal flat image without noise, the determination parameter α may be zero. However, as mentioned earlier, the image data output from the image sensor 107 in FIG. 1 contains noise even if it is a flat part, and if the determination parameter is mistakenly detected as an edge part. Therefore, the edge determination unit 223 uses the sum of the noise amount VAR and the determination parameter α. The determination parameter α will be described later.

In short, the edge determination unit 223 uses the sum of the noise amount VAR and the determination parameter α as a threshold value for determination. In this embodiment, the noise amount VAR is always determined.

To explain this using the image diagram shown in FIG. 7, in this embodiment, the noise amount VAR in the direction along the edge (that is, the direction not intersecting the edge) is calculated, and the value obtained by adding the determination parameter α to this is calculated as follows: Let the threshold value be (=VAR+α). Here, if there is noise in the image, such as the shot noise mentioned above, the amount of noise differs depending on the level of the average brightness value within the screen. Regarding this, in this embodiment, the noise amount VAR is always determined (for example, for each pixel), and the change in the noise amount VAR is reflected in the threshold value. In addition, not only the noise amount VAR but also the determination parameter α is added. From the above, by making the determination by the edge determination unit 223, pixels that are originally a flat portion are not erroneously detected as being an edge portion.

The determination procedure of the edge determination unit 223 will be described below based on a specific example. In the following, for the sake of simplicity, a case where there is no noise at an edge portion, a case where there is no noise at a flat portion, and a case where there is noise at a flat portion will be explained. In each case below, the contents up to the determination by the edge determination unit 223 include the edge preliminary determination unit 210 (directionality data calculation unit 211 and direction detection unit 212), temporary determination unit 221, and noise amount calculation described above. 222.

(Case 1: When there is no noise at the edge part)
FIG. 8 shows the reference image F' in the upper part of the figure. In this reference image F', each pixel has a luminance value shown in parentheses along with a code for identifying the pixel. This reference image F' has an edge in an area corresponding to the upper right of the page. In short, the brightness value of the flat portion is 32, and there is an edge region where the brightness value changes from 32 to 64.

First, the edge direction component is determined based on the following equation (12). At this time, the reference pixel P (D _V , D _{H )} used when calculating D _V and D _H is shown on the lower left side of FIG _. 8, and the reference pixel P (D _NW , D _NE ) are shown on the lower right side of FIG.

From the above, it can be seen that when viewed from the pixel of interest (pixel R ₂₂ at the center of reference image F'), the direction intersecting the edge is the vertical direction or diagonally upper right direction, which is the maximum value D _MAX . On the other hand, it can be seen that the direction along the edge is the horizontal direction or the diagonally upper left direction, which is the minimum value D _MIN .

Next, the amount of noise (H _var , V _var , NE _var , NW _var ) in each direction is determined. FIG. 9 shows a reference pixel P (H _var ) used when calculating H _var , a reference pixel P (V _var ) used when calculating V _var , a reference pixel P (NE _var ) used when calculating NE _var , A reference pixel P (NW _var ) used when calculating NW _var is shown.

The amount of noise in each direction (H _var , V _var , NE _var , NW _var ) is as follows.
H _var =16
V _var =44
NE _var =44
_NWvar =16
For example, the noise amount H _var (or NW _var ) in the direction corresponding to the direction D _MAX intersecting the edge and the direction D _MIN along the edge is compared.
D _MAX > H _var +α?
Here, α is a threshold value for distinguishing between noise amount and edge. If the image is noise-free and has high contrast like in this example, there is no problem with α = 0, but if noise components are included in flat areas as described later, it is better to set α > 0. , the amount of noise and edge components can be distinguished.

In this example, D _MAX = 80, H _var = 16, and when α = 0, the above inequality is satisfied, so the edge determination unit 223 determines that the pixel of interest R ₂₂ is an edge portion. .

(Case 2: When the area is flat and there is no noise)
Although not shown, if each pixel in the reference area has the same brightness value (that is, it is a flat portion), the directional data is zero in all directions, and the amount of noise in all directions is also zero. Therefore, the above-mentioned inequality does not hold, and the edge determining unit 223 determines that the pixel of interest _R22 is a flat portion.

(Case 3: When the area is flat and contains noise)
This example is an example in which a very small amount of noise is applied to an image of R=G=B=32. The following results are obtained by calculating the directional data and the amount of noise.
D _H =1
D _V =5
D _NE =5
D _NW =1
D _MAX = D _V or D _NW
D _MIN = D _H or D _NE
Hvar =1
V _var =2.75
NE _var =2.75
NW _var = 1
The value of D _MAX (direction crossing the edge) is 5, and the amount of noise in the direction along the edge H _var =1. Here, if α=0, the above-mentioned inequality would hold true. Therefore, for example, let α=4. If α=4 is set, the above-mentioned inequality does not hold, so the edge determination unit 223 determines that the pixel of interest R ₂₂ is a flat portion.

(judgment parameter α)
The larger the value of α is, the smaller the region of values determined to be an edge portion becomes. Therefore, the value of the determination parameter is set to a small value to the extent that erroneous determinations due to noise do not increase.

For example, α is determined by preparing a stripe image that can be barely judged by the eye when the image is acquired by the image sensor 107 in FIG. It is best to experimentally find the optimal value that allows you to determine whether it is an edge portion or a flat portion. Note that α is approximately a fixed value for each image sensor.
The determination parameter α is a value of 1 or more.

According to the edge determination unit 220 having each of the above configurations, the edge direction component value of the pixel of interest is compared with the sum of the amount of noise of neighboring pixels and a predetermined determination parameter, and the edge direction component value of the pixel of interest is Determine whether is an edge part or a flat part. In the above-mentioned Patent Document 1, the presence or absence of an edge is determined by comparing the maximum value of edge directionality data (=direction intersecting the edge) with a certain threshold value, so there is a possibility that the above-mentioned erroneous determination may occur. There is. On the other hand, in this embodiment, the amount of noise is determined using the average deviation every time edge determination is performed, and the presence or absence of an edge and the edge direction are determined based on the amount of noise. As a result, in this embodiment, the erroneous determination as described above does not occur.

That is, according to this embodiment, edges and their edge directions can be accurately determined.

<Color interpolation part>
The color interpolation unit 230 performs color interpolation of Bayer data.

Note that the color interpolation process using the edge direction is preferably used particularly when the pixel of interest is an R or B pixel and data of a G pixel in the diagonal direction of the pixel of interest is interpolated. Therefore, in the following, only interpolation of G pixel data when the pixel of interest is an R/B pixel will be described. Regarding interpolation of R and B on G pixels, B on R pixels, and R on B pixels, a conventional example described later may be applied.

The color interpolation unit 230 performs interpolation of G pixel data using the edge direction when the pixel of interest is an R or B pixel in the following procedure. Note that if the pixel of interest is B, it can be determined by the same method by replacing R in the formula/sentence with B.

When the edge direction determined by the edge preliminary determination unit 210 described on the left is diagonally upward to the right regarding the vicinity of the target pixel determined (confirmed) to be an edge portion by the edge determination unit 220 described above (D _NE is the smallest) will be explained using FIG.

(Color interpolation processing for the case where the edge direction is diagonally upward to the right)
FIG. 10 is a diagram conceptually showing color interpolation processing when the edge direction is diagonally upward to the right.

The color interpolation unit 230 performs a first step of determining the G component g _NE in the diagonally upper right direction, and a second step of determining the G component G ₂₂ of the pixel of interest. These steps will be described below with reference to the reference image shown in FIG. 10 ( _R22 is the pixel of interest and _DNE is the smallest).

First, as a first step, the G component g _NE in the diagonally upper right direction is determined. In determining this g _NE , it is assumed that the color correlation of low frequency components of colors in a local region is constant.

In the first step, the average value (low frequency component) g _AVE of the nearest G component of the pixel of interest is determined based on the following equation (13). Also, based on the following equation (13), among the B components closest to the pixel of interest, the B component that exists diagonally upward to the right of the pixel of interest is removed, that is, the B component that exists diagonally upward to the left of the pixel of interest is removed. Find the average value of the components (low frequency component) b _AVE . Furthermore, the average value of the B components present diagonally above and to the right of the pixel of interest is determined, and this is estimated to be the B component b ₂₂ of the pixel of interest. Based on the above assumption, the difference value between g _AVE and b _AVE is the same as the difference value between the G component g _NE in the diagonally upper right direction that is desired to be obtained and the obtained b ₂₂ , so the following formula can be obtained. Based on (13), the G component g _NE in the diagonally upper right direction is obtained by adding b ₂₂ and the difference value between g _AVE and b _AVE .

Incidentally, originally, the _gNE obtained above may be used as is as the G component _G22 of the pixel of interest, but as with edge detection, the accuracy of the G component in the diagonal direction is poor. Therefore, following the first step, the color interpolation unit 230 performs a second step to ensure accuracy.

In the second step, based on the following equation (14), add the average value g _VH of the G components (g _V , g _H ) obtained when horizontal = vertical direction is selected to g _NE obtained above. , find the average. Note that when horizontal=vertical direction, the horizontal G component g _H and the vertical G component g _V can be determined based on the equations described below.

According to the second step, the G component _G22 of the pixel of interest can be interpolated with high precision.

(Color interpolation processing for the case where the edge direction is diagonally upward to the left)
When the edge direction determined by the edge preliminary determination unit 210 described on the left is diagonally upward to the left regarding the vicinity of the pixel of interest determined (confirmed) to be an edge portion by the edge determination unit 220 described above (D _NW is the smallest) will be explained using FIG. 11.

As in the case where the edge direction is the diagonally upper right direction, the color interpolation unit 230 performs color interpolation processing that includes two steps.

Specifically, the color interpolation unit 230 performs a first step of determining the G component g _NW in the diagonally upper left direction, and a second step of determining the G component G ₂₂ of the pixel of interest. These steps will be explained below with reference to the reference image shown in FIG. 11 ( _R22 is the pixel of interest and D _NW is the smallest), but this is different from the case where the edge direction is diagonally upward to the right. I will explain only the points.

The difference is the pixel used for estimating the B component _b22 of the pixel of interest in the first step and the pixel used for calculating b _AVE . If the edge direction is diagonally upward to the left, the average value of the B components existing diagonally upward to the left of the pixel of interest is calculated, and this is estimated to be the B component _b22 of the pixel of interest (using the following formula (15)). Furthermore, pixels present diagonally above and to the right of the pixel of interest are used to calculate the average value (low frequency component) b _AVE of the B component. Other than these, it is the same as above, and using the above assumptions, the G component g _NW in the diagonally upper left direction is determined as shown in the following equation.

In addition, in this case as well, in the second step as described above _{, based on the following equation (16), the G component (g V ,} The average value g _VH of g _H ) is added and the average value is determined.

Thereby, the G component _G22 of the pixel of interest can be interpolated with high accuracy.

(Color interpolation processing when the edge direction is vertical)
Regarding the vicinity of the pixel of interest determined (confirmed) to be an edge portion by the edge determination unit 220 described above, if the edge direction determined by the edge preliminary determination unit 210 described on the left is the vertical direction ( _DV is the smallest), The color interpolation unit 230 can obtain the vertical G component _gV based on the following equation (17).

In this case, the precision of the G component is not poor as in the case of the diagonal direction described above. Therefore, in this case, interpolation is performed by using the obtained _gV as the G component _G22 of the pixel of interest.

(Color interpolation processing when the edge direction is horizontal)
Regarding the vicinity of the pixel of interest that has been determined (confirmed) to be an edge portion by the edge determination unit 220 described above, if the edge direction determined by the edge preliminary determination unit 210 described on the left is the horizontal direction (D _H is the smallest), The color interpolation unit 230 can determine the vertical G component _gH based on the following equation (18).

In this case as well, the precision of the G component is not poor as in the case of the diagonal direction described above. Therefore, in this case, interpolation is performed by using the obtained _gH as the G component _G22 of the pixel of interest.

(Color interpolation processing in cases where the edge direction is horizontal and vertical, or diagonally upward to the right and diagonally upward to the left)
Regarding the vicinity of the target pixel determined (confirmed) to be an edge portion by the edge determination unit 220 described above, if the edge directions determined by the edge preliminary determination unit 210 described on the left are the horizontal direction and the vertical direction (D _H =D _V (and the smallest), or in the diagonally upper right direction and the diagonally upper left direction (D _NE =D _NW and the smallest), the color interpolation unit 230 calculates the horizontal and vertical The average value g _VH of the G components (g _V , g _H ) obtained when the direction is selected is determined.

In this case as well, the precision of the G component is not poor as in the case of the diagonal direction described above. Therefore, interpolation is performed by setting the obtained _gVH as the G component _G22 of the pixel of interest.

Note that if the edge direction determined (confirmed) by the edge determination unit 220 described above is the horizontal direction and the vertical direction, or the diagonally upper right direction and the diagonally upper left direction, it may be processed as a flat portion. In that case, color interpolation may be performed by applying an average value filter using all G pixels in the reference image.
<Color interpolation unit (interpolation of R and B on G pixels, B on R pixels, and R on B pixels)>
For interpolation of R and B on G pixels, B on R pixels, and R on B pixels, a general gradient method may be applied. Alternatively, interpolation can also be performed using a general color correlation method, which provides better results than the general gradient method. However, in the case of the general color correlation method, the circuit scale increases because it is necessary to store G pixel data. The gradient method and color correlation method will be explained below.

(Gradient method)
FIG. 12 shows a reference image 8000 bayer for explaining the interpolation method of B 22 when the pixel of interest is R ₂₂ using the gradient method, and a reference image 8000 _bayer for explaining the interpolation method _of B ₂₂ when the pixel of interest is G ₂₂ using the gradient method. and a reference image 8001 _bayer for explaining the interpolation method of _B22 .

In calculating _B22 using the reference image 8000 _Bayer , the color interpolation unit 230 calculates the absolute value difference based on the following formula as a preliminary calculation.
B _NW = |B ₁₁ -B ₃₃ |
B _NE = | B ₁₃ - B ₃₁ |
And if B _NE =B _NW , then the following formula;
B ₂₂ = (B ₁₁ +B ₁₃ +B ₃₁ +B ₃₃ )/4
If B ₂₂ is found and B _NE <B _NW , then the following formula;
B ₂₂ =(B ₁₃ +B ₃₁ )/2
If B ₂₂ is found and B _NE >B _NW , then the following formula;
B ₂₂ = (B ₁₁ +B ₃₃ )/2
Then _B22 can be found. Note that when the pixel of interest is _B22 , _R22 can be found by replacing R and B in the above explanation.

Next, when the pixel of interest is _G22 , interpolation can be performed as follows using the reference image 8001 _bayer .
B ₂₂ =(B ₁₂ +B ₃₂ )/2
R ₂₂ =(R ₂₁ +R ₂₃ )/2
Note that a G pixel with R pixels on the left and right sides is generally described as G _R , and a G pixel with B pixels on the left and right sides is sometimes described as G _B , and the reference image 8001 _bayer is an example of G _R. In the case of G _B , R and B in the above formula can be replaced.

(color correlation method)
FIG. 13 shows a reference image 9000 _bayer for explaining the interpolation method of B 22 when the pixel of interest is R ₂₂ using the color correlation method, and a bayer image of 9000 for explaining the interpolation method of B ₂₂ when the pixel of interest is G ₂₂ using the color correlation method. This is a reference image 9001 _bayer for explaining the interpolation method of R ₂₂ and B ₂₂ .

If the procedure of this embodiment is followed, the G components at the R and B pixel positions will have already been interpolated. Therefore, in calculating _B22 , the color interpolation unit 230 calculates the average value of B and G as follows as a preliminary calculation.
B _AVE = (B ₁₁ +B ₁₃ +B ₃₁ +B ₃₃ )/4
_GAVE =( _G11 + _G13 + _G31 + _G33 )/4
Then, _B22 is determined based on the color correlation.
B ₂₂ = G ₂₂ + (B _AVE - G _AVE )
Note that in this example, a correlation difference is used, such as B _AVE - G _AVE , but a correlation ratio may be used as described below.
B ₂₂ = G ₂₂ × (B _AVE /G _AVE )
In the above explanation, the average value (AVE) is used, but the original intention is to obtain the low frequency component.

Next, use G ₂₂ to find B ₂₂ . Even in this case, if the procedure of this embodiment is followed, the G components at the R and B pixel positions will have already been interpolated. Therefore, in calculating _B22 , the color interpolation unit 230 calculates the average value of B and G as follows as a preliminary calculation.
_BAVE =( _B12 + _B32 )/2
G _AVE = (G ₁₂ + G ₃₂ )/2
Then, _B22 is determined based on the color correlation.
B ₂₂ = G ₂₂ + (B _AVE - G _AVE )
Next, in calculating R ₂₂ using G ₂₂ , the color interpolation unit 230 calculates the average value of R and G as follows as a preliminary calculation.
_BAVE =( _B21 + _B23 )/2
G _AVE = (G ₂₁ + G ₂₃ )/2
Then, _R22 is determined based on the color correlation.
R ₂₂ = G ₂₂ + (R _AVE - G _AVE )
Color interpolation can be performed using the above method.

(2.2) Operation processing of image processing unit (image processing method)
FIG. 14 is a flowchart showing the operation flow (image processing flow) of the image processing unit 125. The image processing method (color interpolation method) by the image processing unit 125 will be explained using FIG. 14.

When the Bayer data is input to the image processing unit 125, an image processing operation is started, and in step S1, the directionality data calculation unit 211 of the edge preliminary determination unit 210 uses the acquired Bayer data to calculate directionality data (D _H , D _V , D _NE , D _NW ). The calculation method is as described above.

Next, as step S2 (direction detection step), the direction detection unit 212 of the edge preliminary determination unit 210 acquires each directional data (D _H , _DV , D _NE , D _NW ) calculated in step S1. Then, the presence or absence of an edge and the edge direction near the pixel of interest are detected. In step S2, the direction detection unit 212 identifies (selects) the direction (direction along the edge) showing the minimum value among D _H , D _V , D _NE , and D _NW .

Step S2 will be further detailed using FIG. 15.

In step S2, first, as step S21, it is determined whether the pixel of interest is an R pixel or a B pixel. As a result of the determination in step S21, if the pixel of interest is neither an R pixel nor a B pixel, the process ends (No in step S21). In short, in this embodiment, the edge direction is not detected for the G pixel, so if the pixel of interest is G, the process ends. On the other hand, if the pixel of interest is an R pixel or a B pixel, the process moves to step S22 and step S23.

In step S22, comparisons are made in the horizontal and vertical directions. In step S23, a diagonal comparison is performed. Here, for example, the horizontal direction is 0, the vertical direction is 1, the diagonal diagonally upward to the right is 2, the diagonal diagonally upward to the left is 3, the horizontal component = vertical component is 4, and the diagonal component is diagonally upward to the right. = Define an index number with 5 being the upper left diagonal component. In step S22, the horizontal component and the vertical component are compared, and if the horizontal component is small, 0 is stored in the temporary storage variable DirVH, if the vertical component is small, 1 is stored, and if they are equal, 4 is stored. Further, the selected direction component is stored in a temporary storage variable DVH.

In step S23, similarly to step S22, the diagonal upper right component and the diagonally upper left component are compared, and if the diagonal upper right component is smaller, 2 is set, and if the upper left component is smaller, 3 is set. 5 is stored. In addition, each selected direction component is stored in a temporary storage variable DNN.

In step S24, the edge direction is determined. Specifically, in step S24, the horizontal/vertical direction comparison result DVH of step S22 is compared with the diagonal direction comparison result DNN of step S23. As a result of the comparison, if the horizontal and vertical components DVH are small, the direction stored in DirVH is selected (preliminarily determined) as the edge direction. If the diagonal direction component DNN is small, the direction stored in DirNN is selected (preliminarily determined) as the edge direction. If DVH and DNN are equal, for example, the direction stored in DirVH is selected (preliminarily determined) as the edge direction. In this case, since a plurality of minimum values exist in the directional data, it may be determined that this is a flat portion.

When the direction detection unit 212 (preliminarily) determines the edge direction in step S2, the process moves to step S3.

In step S3 (determination step), the provisional determination unit 221 of the edge determination unit 220 provisionally determines whether the vicinity of the pixel of interest is a flat portion or an edge portion. In the tentative determination process in step S3, a tentative determination is made based on the calculation results obtained in steps S1 and S2. The specific provisional determination method is as described above, but will be explained below based on FIG. 16.

In step S3, first, as step S31, directional data (D _V , _DH , D _NE , D _NW ) in each direction is calculated. Then, in the subsequent step S32, it is determined whether there is a minimum value or maximum value in the calculated directional data. If there is a minimum value and a maximum value (Yes in step S32), it is tentatively determined to be an edge portion (step S33), and the process moves to step S4 in FIG. 14. On the other hand, if it is tentatively determined in step S3 that the vicinity of the pixel of interest is a flat portion, the process moves to a color interpolation step in step S6B, which will be described later.

In step S4 (noise amount calculation step), the noise amount calculation unit 222 calculates the noise amount VAR of neighboring pixels. As described above, the amount of noise of neighboring pixels in step S4 is determined along the edge direction determined in step S2. The specific noise amount calculation method is as described above. Once the noise amount calculation unit 222 calculates the noise amount VAR of the neighboring pixels, the process moves to step S5.

Note that the process of calculating the directional data and detecting the edge direction in steps S1 and S2 and the process of calculating the amount of noise in the vertical, horizontal, and diagonal directions in step S4 may be performed in parallel. This makes it possible to compare the amount of noise in the direction of the minimum value with the maximum value of the directional data immediately after the detection of the edge direction is completed, thereby reducing the calculation time.

In step S5 (comparison step), a value obtained by adding the noise amount VAR of the neighboring pixels calculated in step S4 and a predetermined determination parameter α is compared with the maximum value of the directional data calculated in step S1. This comparison process is performed by the edge determination section 223. In step S5, as mentioned above, the following relational expression;
Maximum value of edge directionality data>VAR+α
Determine whether the following is satisfied. Then, if the relational expression is satisfied, the area around the pixel of interest is determined to be an edge part (confirmed), and if the relational expression is not satisfied, the vicinity of the pixel of interest, which was tentatively determined to be an edge part, is determined to be a flat area (confirmed). do.

If it is determined in step S5 that the vicinity of the pixel of interest is an edge portion, the process moves to a color interpolation step in step S6A. If it is determined that the vicinity of the pixel of interest is a flat portion, the process moves to a color interpolation step in step S6B, which will be described later.

In step S6A, the color interpolation unit 230 performs color interpolation processing on the pixel of interest using the edge direction determined in step S2. In step S6A, the interpolation direction selected in step S24 of step S2 is used. That is, in step S6A, interpolation is performed using the direction in which the edge direction component is the smallest. The interpolation method is as described above.

On the other hand, in step S6B, the color interpolation unit 230 suppresses the noise component using a low frequency pass filter (LPF) method such as an average filter using all G pixels in the reference image, so that the color of the pixel of interest is Perform interpolation.

When color interpolation is completed in step S6A or step S6B, the process ends if color interpolation for other pixels of interest has been completed (step S7). If there is a pixel for which color interpolation has not been performed, the process returns to step S1 and color interpolation is performed for that pixel. When color interpolation is completed for all pixels to which color interpolation is to be performed, the process ends.

(3) Effects According to this embodiment, it is possible to perform color interpolation processing with higher precision than before.

In Patent Document 1, when determining the presence or absence of an edge, the maximum value of edge directionality data (=direction intersecting the edge) is compared with a certain threshold value, and if it is smaller than the threshold value, it is considered a flat part. However, the output data from the image sensor includes shot noise and the like even if it is a flat portion. Regarding shot noise, the amount of noise differs between a high-luminance part and a low-luminance part even within the same frame. Therefore, when the brightness average is high, shot noise exceeds the threshold. Therefore, there is a possibility that a flat portion may be erroneously detected as an edge portion. Therefore, it is necessary to appropriately change the threshold value with respect to the average luminance value even within the same frame.

In contrast, in this embodiment, the amount of noise of neighboring pixels is calculated for the vicinity of the pixel of interest that has been tentatively determined to be an edge portion, and the sum of the amount of noise and the parameter is compared with the maximum value of the directional data. . As a result of the comparison, if the maximum value of the directional data is larger than the added value, the vicinity of the pixel of interest is determined to be an edge portion. On the other hand, if it is less than or equal to the added value, the vicinity of the pixel of interest, which was previously tentatively determined to have an edge, is reconsidered to be a flat portion. Thereby, there is no possibility that erroneous color interpolation processing will be performed due to the above-mentioned erroneous determination. Therefore, color interpolation can be performed with high precision, contributing to the generation of high-quality images.

Furthermore, in Patent Document 2, edge directionality data is also calculated. The calculation method is to perform gradient calculation within a reference pixel area of 7×7 pixels. To realize this, a line memory (also called a line buffer) for seven lines is required. Furthermore, in the diagonal gradient calculation, there is a division by 7 at the end of the calculation formula. Since divisions other than powers of 2 cannot be calculated by bit shifting, a separate division circuit is required to calculate the gradient in the diagonal direction, which increases the circuit scale.

In contrast, in this embodiment, as described above, edge directionality data calculation is performed by gradient calculation within a reference pixel area of 5×5 pixels. Therefore, the line memory only requires 5 lines. Furthermore, analysis of diagonal lines is also possible using only addition/subtraction and bit shift processing. Therefore, it is possible to prevent the circuit scale from becoming too large compared to before.

[Embodiment 2]
Other embodiments of the invention will be described below. For convenience of explanation, members having the same functions as the members described in the above embodiment are given the same reference numerals, and the description thereof will not be repeated.

The difference between this embodiment and the first embodiment described above lies in the subsequent processing when a flat portion is determined in the tentative determination. In short, the operation flow (image processing flow) of the image processing method by the image processing unit 125 is different from that of the first embodiment. This embodiment will be described below using FIG. 17.

FIG. 17 is a flowchart showing the operation flow of the image processing method by the image processing unit 125 of this embodiment.

Steps S1 to S3 have been explained in the first embodiment, so their explanation will be omitted.

In the first embodiment described above, if it is tentatively determined in step S3 that the area near the pixel of interest is a flat portion, the process moves to the color interpolation step of step S6B. In contrast, in this embodiment, when it is determined in step S3 that the vicinity of the pixel of interest is a flat portion (step S3A in FIG. 17), it is determined again whether or not it is a flat portion by changing the conditions. be done. This is step S3B in FIG.

Further, in the first embodiment described above, while the image area referred to when tentatively determining whether it is an edge portion in step S3 has a size of 3 rows x 3 columns, in the noise amount calculation process of the subsequent step S4, Calculations are performed based on an image area of 5 rows x 5 columns. On the other hand, in the present embodiment, when the image area of 3 rows x 3 columns is tentatively determined to be an edge portion in step S3A, the reference area is not expanded to 5 rows x 5 columns, and 3 The amount of noise is calculated in an image area of rows and three columns.

Below, details will first be explained regarding the provisional determination step of this embodiment.

Step S3A in FIG. 17 corresponds to step S3 in the first embodiment. In step S3A, the tentative determination unit 221 tentatively determines whether or not there is an edge using the 3 rows by 3 columns pixel area shown in FIG. In the example of a) in which the pixel area has edges in the vertical (vertical) direction shown in FIG. 5, the directionality data in each direction is determined by the absolute value difference as shown below.
D _V = |L ₀₁ -L ₂₁ |=0
D _H = |L ₁₀ -L ₁₂ |=255
D _NE =|L ₀₂ -L ₂₀ |=255
D _NW = | L ₀₀ - L ₂₂ | = 255
At this time, since _DV is the smallest, it can be considered that an edge exists along the vertical (vertical) direction of the pixel area, that is, the edge direction is the vertical (vertical) direction.

On the other hand, in the examples shown in d), e), f), g) and h) of FIG. 5, the minimum value and maximum value of the directional data are the same. Therefore, in step S3A, it is tentatively determined that it is a flat portion. If it is tentatively determined that it is a flat portion, the process moves to step S3B.

In step S3B, the following calculation D _{V (1)} = |2L ₁₁ -L ₀₁ -L ₂₁ | is performed using the 3 rows x 3 columns pixel area shown in FIG.
D _H(1) = |2L ₁₁ -L ₁₀ -L ₁₂ |
D _{NE (1)} = | 2L ₁₁ -L ₀₂ -L ₂₀ |
D _{NW (1)} = | 2L ₁₁ -L ₀₀ -L ₂₂ |
will be carried out.

Subsequently, in step S3C, it is determined whether one or both of D _{V (1)} = D _{H (1) and} D _{NE (1)} = D _{NW (1)} are satisfied based on the calculation result of Step S3B. Determine whether If this is not true (No in step S3C), the process moves to step S3D.

In step S3D, the size of the reference image area is expanded to 5 rows x 5 columns, and the process moves to step S3E.

In step S3E, the edge direction is estimated based on equation (1) described in the first embodiment. At this time, the direction component calculation based on the difference with the pixel of interest in 5 rows x 5 columns is performed using the pixels shown in FIG. 18 as follows:
D _V(2) = |2L ₂₂ -L ₀₂ -L ₄₂ |
D _H(2) = |2L ₂₂ -L ₂₀ -L ₂₄ |
D _{NE (2)} = | 2L ₂₂ -L ₀₄ -L ₄₀ |
D _{NW (2)} = | 2L ₂₂ -L ₀₀ -L ₄₄ |
It is determined by In the example shown in d) of FIG. 5, when the image area is expanded to 5×5 and shown in FIG. 19, by using this calculation,
D _{V (2)} = 0
D _H(2) =D _NE(2) =D _NW(2) =510
Since _DV(2) is the minimum, the direction along the edge can be detected to be the vertical direction.

In step S3G, similarly to step S3 of the first embodiment, it is tentatively determined whether or not it is an edge portion. If it is tentatively determined that it is an edge portion, the process moves to step S4. On the other hand, if it is determined that the area is a flat area, the process moves to step S6B.

On the other hand, in step S3C, if D _V(1) =D _H(1) and D _NE(1) =D _NW(1) (Yes in step 3C), the flat part is defined as a flat part, and henceforth, Embodiment 1 will be described. Process in the same way as .

Here, in step S3A, if there is a minimum value in any of the directional data and it is tentatively determined that it is an edge portion as shown in a), b), and c) of FIG. to move to.

In step S4A, while step S4 of the first embodiment uses an image area of 5 rows x 5 columns to calculate the amount of noise, an image area of 3 rows x 3 columns is used for calculation. FIG. 20 shows an image area of 3 rows x 3 columns. Step S4A will be explained based on FIG. 20. In the image area of 3 rows x 3 columns in FIG. 20, the pixel of interest L ₁₁ is R, and the pixels L ₀₁ , L ₁₀ , L ₁₂ and L ₂₁ that are adjacent to the pixel of interest L ₁₁ on the upper, lower, left and right sides are G, and L ₀₀ , L ₀₂ , L ₂₀ and L ₂₂ that are diagonally adjacent to the pixel of interest L ₁₁ are B. In the image area of FIG. 20, for example, if the direction along the edge is the vertical direction ( _DV is the smallest), the noise amount calculation unit 222 calculates the average luminance value of each row (B1 _AVE , G2 _AVE , B3 _AVE ) as each of the following formulas;
B1 _AVE = (L ₀₀ +L ₂₀ )/2
G2 _AVE = (L ₀₁ +L ₂₁ )/2
B3 _AVE = (L ₀₂ +L ₂₂ )/2
Find it based on. In addition, the noise amount calculation unit 222 calculates a value obtained by dividing the absolute value difference between the average value of brightness (B1 _AVE , G2 _AVE , B3 _AVE ) and the brightness of each pixel of the same color by the number of pixels, based on the following formulas. The noise amount V _var is obtained by adding all the noise values (B1 _var , G2 _var , B3 _var ) of each row.
B1 _var = (|L ₀₀ -B1 _AVE |+|L ₂₀ -B1 _AVE |)/2
G2 _var = (|L ₀₁ -G2 _AVE |+|L ₂₁ -G2 _AVE |)/2
B3 _var = (|L ₀₂ -B3 _AVE |+|L ₂₂ -B3 _AVE |)/2
V _var =B1 _var +G2 _var +B3 _var
Further, when the direction along the edge is the horizontal direction ( _DH is the smallest), the noise amount calculation unit 222 calculates the average value of the brightness of each row (B1 _AVE , G2 _AVE , B3 _AVE ) using the following formulas;
B1 _AVE = (L ₀₀ +L ₀₂ )/2
G2 _AVE = (L ₁₀ +L ₁₂ )/2
B3 _AVE = (L ₂₀ +L ₂₂ )/2
Find it based on. In addition, the noise amount calculation unit 222 calculates a value obtained by dividing the absolute value difference between the average value of brightness (B1 _AVE , G2 _AVE , B3 _AVE ) and the brightness of each pixel of the same color by the number of pixels, based on the following formulas. The noise amount H _var is calculated by adding all the noise values (B1 _var , G2 _var , B3 _var ) of each row.
B1 _var = (|L ₀₀ -B1 _AVE |+|L ₀₂ -B1 _AVE |)/2
G2 _var = (|L ₁₀ -G2 _AVE |+|L ₁₂ -G2 _AVE |)/2
B3 _var = (|L ₂₀ -B3 _AVE |+|L ₂₂ -B3 _AVE |)/2
H _var =B1 _var +G2 _var +B3 _var
Further, when the direction along the edge is diagonally upward to the right ( _DNE is the smallest), the noise amount calculation unit 222 calculates the average value of the luminance of each row (G1 _AVE , B2 _AVE , G3 _AVE ) using the following formulas. ;
G1 _AVE = (L ₀₁ +L ₁₀ )/2
B2 _AVE = (L ₀₂ +L ₂₀ )/2
G3 _AVE = (L ₁₂ +L ₂₁ )/2
Find it based on. In addition, the noise amount calculation unit 222 calculates a value obtained by dividing the absolute value difference between the average value of brightness (G1 _AVE , B2 _AVE , G3 _AVE ) and the brightness of each pixel of the same color by the number of pixels, based on the following formulas. The noise amount NE _var is calculated by adding all the noise values (G1 _var , B2 _var , G3 _var ) of each row.
G1 _var = (|L ₀₁ -G1 _AVE |+|L ₁₀ -G1 _AVE |)/2
B2 _var = (|L ₀₂ -B2 _AVE |+|L ₂₀ -B2 _AVE |)/2
G3 _var = (|L ₁₂ -G3 _AVE |+|L ₂₁ -G3 _AVE |)/2
NE _var = G1 _var + B2 _var + G3 _var
Further, when the direction along the edge is the diagonally upper left direction (D _NW is the smallest), the noise amount calculation unit 222 calculates the average value of the luminance of each row (G1 _AVE , B2 _AVE , G3 _AVE ) using the following equations. ;
G1 _AVE = (L ₀₁ +L ₁₂ )/2
B2 _AVE = (L ₀₀ +L ₂₂ )/2
G3 _AVE = (L ₁₀ +L ₂₁ )/2
Find it based on. In addition, the noise amount calculation unit 222 calculates a value obtained by dividing the absolute value difference between the average value of brightness (G1 _AVE , B2 _AVE , G3 _AVE ) and the brightness of each pixel of the same color by the number of pixels, based on the following formulas. The noise amount NW _var is obtained by adding all the noise values (G1 _var , B2 _var , G3 _var ) of each row.

According to the above calculation procedure, the noise amount calculation unit 222 calculates the noise amount VAR (that is, any one of the above-mentioned H _var , V _var , NE _var and NW _var ) of the neighboring pixels.

On the other hand, if it is tentatively determined in step S3G that it is an edge portion, the processes from step S4 onward in the first embodiment are performed.

In step S5, as in the first embodiment, the edge determination unit 223 calculates the sum of the noise amount VAR of the neighboring pixels calculated by the noise amount calculation unit 222 and the predetermined determination parameter α, and the maximum value of the directionality data. It is determined whether the vicinity of the pixel of interest is an edge portion or a flat portion. Thereafter, processing is performed in the same manner as in the first embodiment.

In this embodiment, as described above, the calculation of the amount of noise does not expand the reference area to 5x5, and the calculation ends in a 3x3 image area, which is the image area for tentative edge determination. Therefore, compared to the first embodiment, the amount of calculation can be reduced. As the number of pixels increases, what used to be a line with a width of one pixel in the low-pixel era is now often expressed using multiple pixels due to higher resolution. In other words, cases d) to f) in FIG. 5 are decreasing. Therefore, in the case of high resolution, the cases a) to c) in FIG. 5 increase, and it is significant that the amount of calculations is reduced in this embodiment. Additionally, it can be said that this contributes to a reduction in power consumption.

[Example of implementation using software]
The control block (particularly the control unit 120) of the imaging device 1 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, the imaging device 1 includes a computer that executes instructions of a program (image processing program) that is software that implements each function. This computer includes, for example, at least one processor (control device) and at least one computer-readable recording medium storing the above program. In the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, in addition to "non-temporary tangible media" such as ROM (Read Only Memory), tapes, disks, cards, semiconductor memories, programmable logic circuits, etc. can be used. Further, the computer may further include a RAM (Random Access Memory) for expanding the above program. Furthermore, the program may be supplied to the computer via any transmission medium (communication network, broadcast waves, etc.) that can transmit the program. Note that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

〔summary〕
The image processing device (image processing unit 125) according to aspect 1 of the present invention uses directionality data that quantifies the directionality of edges in a pixel of interest and neighboring pixels of Bayer array image data for each of a plurality of directions. a direction detection unit 212 that detects an edge direction; a determination unit (temporary determination unit 221) that determines whether there is a difference in the directional data between the directions; and a noise amount that calculates the amount of noise of the neighboring pixels. a calculation unit 222; a comparison unit (edge determination unit 223) that compares the maximum value of the directional data determined to have the difference with a threshold value including the noise amount; ), (i) if the maximum value is greater than the threshold, color interpolation is performed on the pixel of interest using the edge direction; (ii) if the maximum value is less than or equal to the threshold; In this case, the color interpolation unit 230 performs color interpolation different from the color interpolation (i) on the pixel of interest.

According to the above configuration, it is possible to perform color interpolation processing with higher precision than before.

Specifically, according to the above configuration, even if the determination unit (temporary determination unit 221) determines that there is a difference based on the directional data (that is, it is an edge portion), this determination is The suitability is confirmed by the noise amount calculation unit 222 and the comparison unit (edge determination unit 223). For example, if shot noise or the like is included, a flat portion will be determined to be an edge portion if only the determination is based on directional data. However, by performing the above confirmation, even if it is determined to be an edge part based on the directional data, if it is less than a threshold that includes the noise amount of neighboring pixels, it will not be considered an edge part, and in that case, Color interpolation using edge direction is not performed. If color interpolation using the edge direction is applied to a flat portion that includes shot noise, color interpolation will be performed with shot noise remaining, which is not desirable. Therefore, by calculating the noise amount and performing color interpolation according to the comparison result with a threshold value that includes the noise amount as in the configuration of this aspect 1, color interpolation can be performed with high accuracy and high-quality image data can be generated. can contribute to the generation of

In the image processing device (image processing unit 125) according to aspect 2 of the present invention, in the above aspect 1, the noise amount calculation unit 222 calculates the minimum The amount of noise of the neighboring pixels is calculated along the direction corresponding to the directional data indicating the value.

According to the above configuration, the noise amount can be calculated with higher accuracy than when the noise amount is calculated along the direction corresponding to the directional data indicating the maximum value when there is a difference in the directional data. Can be done.

In the image processing device (image processing unit 125) according to aspect 3 of the present invention, in the above aspect 1 or 2, when the noise amount calculation unit 222 determines that there is a difference in the directional data between the directions, The amount of noise in the direction corresponding to the directional data indicating the minimum value of is calculated based on the average deviation of the luminance values of the pixel of interest and neighboring pixels.

According to the above configuration, the amount of noise can be determined by a simpler calculation than in the case where the amount of noise is calculated using the standard deviation and the variance.

In the image processing device (image processing section 125) according to aspect 4 of the present invention, in the above aspects 1 to 3, the comparison section (edge determination section 223) adds the noise amount and a predetermined determination parameter. value is used as the threshold value.

According to the above configuration, it is possible to accurately determine whether or not it is an edge portion.

In the image processing device (image processing section 125) according to aspect 5 of the present invention, in the above aspects 1 to 4, when the color interpolation section 230 is a red (R) or blue (B) pixel, As the color interpolation in (i), diagonal color interpolation is performed on the green (G) pixel data of the pixel of interest, and as the color interpolation in (ii), all green (G) pixels in the neighboring pixels are Color interpolation of the green (G) pixel data of the pixel of interest is performed using an average value filter using the .

The imaging device 1 according to the sixth aspect of the present invention includes the image processing device (image processing unit 125) according to the first to fifth aspects described above, and an imaging element 107 having a color filter having a Bayer array.

An image processing method according to aspect 7 of the present invention detects an edge direction using directionality data obtained by quantifying the directionality of an edge in a pixel of interest and neighboring pixels of Bayer array image data for each of a plurality of directions. a detection step, a determination step of determining whether or not there is a difference in the directional data between the directions, a noise amount calculation step of calculating a noise amount of the neighboring pixels, and a noise amount calculation step of calculating the amount of noise of the neighboring pixels; a comparison step of comparing the maximum value of the directional data with a threshold value including the noise amount; and (i) if the maximum value is larger than the threshold value, the pixel of interest is (ii) if the maximum value is less than or equal to the threshold value, a color interpolation step that is different from the color interpolation in (i) is performed on the pixel of interest; ,including.

According to the above configuration, it is possible to perform color interpolation processing with higher precision than before.

Specifically, according to the above configuration, even if it is determined in the determination step that there is a difference based on the directional data (that is, it is an edge portion), the appropriateness of this determination is determined based on the noise amount calculation. Confirm by step and comparison step. For example, if shot noise or the like is included, a flat portion will be determined to be an edge portion if only the determination is based on directional data. However, by performing the above confirmation, even if it is determined to be an edge part based on the directional data, if it is less than a threshold that includes the noise amount of neighboring pixels, it will not be considered an edge part, and in that case, Color interpolation using edge direction is not performed. If color interpolation using the edge direction is applied to a flat portion that includes shot noise, color interpolation will be performed with shot noise remaining, which is not desirable. Therefore, as in this aspect 6, by calculating the noise amount and performing color interpolation according to the comparison result with a threshold value that includes the noise amount, color interpolation is performed with high accuracy and high-quality image data is generated. can contribute to this.

Further, the image processing device (image processing unit 125) according to one aspect of the present invention may be realized by a computer, and in this case, the computer may be operated as each unit (software element) included in the image processing device. Although the image processing apparatus described above is realized by a computer, an image processing program for the image processing apparatus and a computer-readable recording medium on which the program is recorded also fall within the scope of the present invention.

The present invention is not limited to the embodiments described above, and various modifications can be made within the scope of the claims.

1 Imaging device 107 Imaging element 120 Control unit 121 CPU
124 Image sensor drive section 125 Image processing section (image processing device)
210 Edge preliminary determination unit 211 Directionality data calculation unit 212 Direction detection unit 220 Edge determination unit 221 Temporary determination unit (determination unit)
222 Noise amount calculation unit 223 Edge determination unit (comparison unit)
230 Color interpolation section

Claims (8)

a direction detection unit that detects an edge direction using directionality data that quantifies the directionality of edges at a pixel of interest and neighboring pixels of Bayer array image data for each of a plurality of directions;
a determination unit that determines whether there is a difference between the plurality of directional data quantified for each of the plurality of directions;
a noise amount calculation unit that calculates the noise amount of the neighboring pixels;
a comparison unit that compares the maximum value of the directional data determined to have the difference with a threshold value including the noise amount;
Depending on the comparison result of the comparison unit, (i) if the maximum value is greater than the threshold, color interpolation is performed on the pixel of interest using the edge direction; (ii) if the maximum value is less than or equal to the threshold; If so, a color interpolation unit that performs color interpolation different from the color interpolation in (i) for the pixel of interest;
Equipped with
When the pixel of interest is a red (R) or blue (B) pixel, the color interpolation unit performs diagonal color interpolation of green (G) pixel data of the pixel of interest as the color interpolation of (i). As the color interpolation in (ii) above, color interpolation of the green (G) pixel data of the pixel of interest is performed using an average value filter using all the green (G) pixels in the neighboring pixels. Processing equipment. The noise amount calculation unit calculates the amount of noise along a direction corresponding to directional data indicating a minimum value when it is determined that there is a difference between the plurality of directional data quantified for each of the plurality of directions. The image processing device according to claim 1, which calculates the amount of noise of neighboring pixels. The noise amount calculation unit calculates the amount of noise in a direction corresponding to directional data that indicates a minimum value when it is determined that there is a difference between the plurality of directional data quantified for each of the plurality of directions. , the image processing apparatus according to claim 1 , wherein the image processing device calculates the calculation based on an average deviation of luminance values of the pixel of interest and neighboring pixels. The image processing device according to claim 1 , wherein the comparison unit uses a value obtained by adding the noise amount and a predetermined determination parameter as the threshold value. An image processing device according to any one of claims 1 to 4 ,
an image sensor having a color filter having a Bayer array;
An imaging device equipped with a direction detection step of detecting an edge direction using directionality data obtained by quantifying the directionality of edges at a pixel of interest and neighboring pixels of Bayer array image data for each of a plurality of directions;
a determination step of determining whether there is a difference between the plurality of directional data quantified for each of the plurality of directions;
a noise amount calculation step of calculating the noise amount of the neighboring pixels;
a comparison step of comparing the maximum value of the directional data determined to have the difference with a threshold value including the noise amount;
Depending on the comparison result of the comparison step, (i) if the maximum value is greater than the threshold, color interpolation is performed on the pixel of interest using the edge direction; (ii) if the maximum value is less than or equal to the threshold; If so, a color interpolation step of performing color interpolation different from the color interpolation of (i) on the pixel of interest;
including;
In the color interpolation step, when the pixel of interest is a red (R) or blue (B) pixel, as the color interpolation of (i), color interpolation is performed in a diagonal direction of green (G) pixel data of the pixel of interest. As the color interpolation in (ii) above, color interpolation of the green (G) pixel data of the pixel of interest is performed using an average value filter using all the green (G) pixels in the neighboring pixels. Processing method. 5. An image processing program for causing a computer to function as the image processing apparatus according to claim 1, comprising: the direction detecting section, the determining section, the noise amount calculating section, the comparing section, and the above. An image processing program that allows a computer to function as a color interpolator. A computer-readable recording medium on which the image processing program according to claim 7 is recorded.

Images