JP4934183B2 - Image signal processing method - Google Patents

Description

  The present invention relates to an image signal processing method, and in particular, obtains good results when applied to signal processing in which interpolation is performed using pixel data from existing pixels for non-existing virtual pixels in a digital camera, an image input device, or the like. Is.

  In a digital camera, the resolution of an image obtained by an image pickup device that is two-dimensionally arranged on a solid-state image pickup device, that is, the number of pixels has been influenced. As the number of pixels formed in a certain area of the solid-state imaging device increases, the improvement in resolution is expected. When the number of image sensors is the same, the larger the image sensor size, the less the influence of noise and the like, and a high-sensitivity image quality can be obtained.

  Consider the above resolution and high sensitivity simultaneously. At this time, the relationship in which the size of the image sensor becomes smaller as the number of pixels formed in a certain area increases is contradictory. For example, the resolution improvement effect due to the increase in the number of pixels and the sensitivity reduction effect due to the size reduction act simultaneously. become. This limits the image quality of images obtained with a solid-state imaging device.

  In consideration of this problem, the solid-state image pickup device and the signal processing method described in Patent Literature 1 take an image by shifting the pixels by half the pixel pitch in the matrix (vertical / horizontal) direction with respect to adjacent elements. Elements are arranged and formed. With this pixel arrangement, so-called honeycomb arrangement, the largest number of pixels is formed in a predetermined space. At this time, the image quality is prevented from being deteriorated by making the size of the image sensor large enough for the number of pixels so that sufficient sensitivity of the image sensor is obtained. Then, in order to use the obtained pixel data more effectively, signal processing is performed for generating and interpolating virtual pixels at positions where the image sensor does not exist with respect to the actual pixels where the image sensor exists. By combining the real pixel and the virtual pixel as a pixel, the resolution of the obtained image is improved. In the signal processing, the image quality is also improved by increasing the signal components of the obtained luminance data and color data.

JP 2000-184386 A

  By the way, in the above-described virtual pixel interpolation processing, first, correlation detection of each pixel is performed in the diagonal direction, vertical and horizontal directions, and a direction with high correlation is detected. Next, interpolation processing is performed on the virtual pixels using the pixels located in the direction of high correlation detected. The image quality is improved by this interpolation processing (adaptive interpolation processing).

  However, image quality deterioration may occur in an image obtained by this virtual pixel interpolation processing. The situation in which this image quality degradation occurs is that when the subject includes a line shape of colored and achromatic colors within a specific width, a difference in luminance level occurs in the longitudinal direction associated with this shape of the subject. In addition, in the image when the resolution chart is taken, a phenomenon such as generation of a false signal that crosses the wedge-shaped chart portion serving as an index of resolving power and a straight line appear to be cut off appears.

  In view of such a problem, an object of the present invention is to provide an image signal processing method capable of avoiding image quality degradation caused by virtual pixel interpolation processing.

  In order to solve the above-described problems, the present invention decomposes incident light from a scene into three primary colors, and geometrical images of adjacent imaging elements that are obtained by photoelectric conversion of the obtained incident light of the three primary colors. It is arranged with a half-pitch shift in the row and / or column direction with respect to the center. This image sensor is used as a real pixel, and each image signal obtained from this image sensor is converted into digital pixel data. In an image signal processing method for generating luminance and color difference data by performing interpolation processing for generating a new pixel as a virtual pixel between the pixels, this method uses the green color of the pixel data when the pixel data is displayed two-dimensionally. Are obtained in a square lattice pattern, and a first process for obtaining red pixel data from an actual pixel at one diagonal position across green and blue pixel data from an actual pixel at the other diagonal position, respectively. And when one of the red and blue real pixels is the target pixel, the vertex of the correlation determination area for the target pixel is expanded to at least the area represented by the same color pixel as the target pixel. While determining the correlation accuracy to improve the correlation accuracy, the pixel data of the complementary color of green is used as the luminance data by using the other plurality of pixel data around the target pixel located in the correlation direction and the other color. Using the complementary pixel data and the green pixel data, the luminance data in the virtual pixel is interpolated and generated from the pixel data of the same color located in the horizontal and / or vertical direction by using the complementary color pixel data and the green pixel data. Based on the color attributes of each pixel and virtual pixel, all red, blue and green colors that are different from the color attributes obtained for all real pixels and virtual pixels are interpolated. And adding a signal of a high-frequency component obtained by passing a frequency higher than a predetermined high-frequency band included in the luminance data generated by interpolation to the pixel data of the three primary colors of each pixel, And a fourth step of performing a broadbanding process.

  Further, in order to solve the above-described problems, the present invention decomposes incident light from the object scene into three primary colors, and geometricalities of adjacent image pickup elements that are photoelectrically converted from the obtained incident light of the three primary colors. This pixel data is arranged with a half-pitch shift in the row and / or column direction with respect to the center, and this image sensor is an actual pixel, and each image signal obtained from this image sensor is digital pixel data. In an image signal processing method for generating luminance and chrominance data by performing interpolation processing for generating new pixels as virtual pixels between the pixels using this method, the pixel data is displayed when the pixel data is displayed two-dimensionally. Are obtained in the form of a square lattice, and the red pixel data from the real pixels at one diagonal position sandwiching the green color and the blue pixel data from the real pixels at the other diagonal position are respectively obtained. The second step of creating a green screen as luminance data while interpolating the virtual pixel using the read green pixel data, and the red and blue real pixels of the read pixel data When one of the pixels is the target pixel, the correlation accuracy is improved by extending the vertex of the correlation determination area for the target pixel to at least the area represented by the same color pixel as the target pixel. And generating pixel data of a complementary color of green using a plurality of other pixel data around the target pixel located in the correlation direction and different colors, and using the pixel data of the complementary color Pixels are also interpolated to create a complementary color screen and generate luminance data, and the resulting green screen and complementary color screen are added to create a real and virtual screen. All of the red, blue, and green colors that are different from the color attributes obtained for all the real pixels and the virtual pixels based on the color attributes of the real pixels and the virtual pixels that are obtained. And interpolating and generating a high frequency component signal obtained by passing a frequency higher than a predetermined high frequency band included in the interpolated luminance data to the pixel data of the three primary colors of each pixel, And a fifth step of broadening the pixel data.

  According to the image signal processing method according to the present invention, the red pixel data from the real pixel at one diagonal position sandwiching the green color, and the blue pixel data from the real pixel at the other diagonal position, respectively, When one of the red and blue real pixels is the target pixel, the correlation is determined by expanding the vertex of the correlation determination area for the target pixel to at least the area represented by the same color pixel as the target pixel. In this way, pixel data of the complementary color of green is generated as luminance data using a plurality of other pixel data around the target pixel located in the correlation direction and the other color surrounding the target pixel in the correlation direction. Using the complementary color pixel data and the green pixel data, the luminance data in the virtual pixel is interpolated and generated from the pixel data of the same color located in the horizontal and / or vertical direction. Based on the color attributes of each real pixel and virtual pixel, all the red, blue and green colors that are different from the color attributes obtained for this real pixel and virtual pixel are interpolated and generated, and the predetermined high frequency possessed by the interpolated luminance data By adding the high-frequency component signal obtained by passing a frequency higher than the band to the pixel data of each of the three primary colors of each pixel, the pixel data of each color is widened to suppress the occurrence of image quality degradation and Since it is possible to suppress the occurrence of a phenomenon such as a false signal crossing a wedge shape and a cross or a straight line cut, the phenomenon that occurs when the subject is photographed can be significantly improved compared to the prior art.

  Further, according to the image signal processing method of the present invention, the red pixel data from the real pixel at one diagonal position sandwiching the green color, and the blue pixel data from the real pixel at the other diagonal position, respectively. The green pixel is created as luminance data while interpolating the virtual pixel using the read green pixel data, and either one of the red and blue real pixels of the read pixel data is the target pixel In this case, the vertex of the correlation determination area for the target pixel is expanded to at least the area represented by the same color pixel as the target pixel, and the correlation accuracy is improved by determining the correlation. The pixel data of the complementary color of green is generated using a plurality of other pixel data around the target pixel located in the correlation direction and the different color, and the pixel data of the complementary color is used. Virtual pixels are also interpolated, and this complementary color screen is created to generate luminance data. The resulting green screen and complementary color screen are added to generate luminance data for all real and virtual pixels, Further, based on the color attributes of the actual pixels and virtual pixels obtained, all the red, blue and green colors that are different from the color attributes obtained for all of the real pixels and virtual pixels are interpolated to generate the interpolated luminance. By adding the high-frequency component signal obtained by passing a higher frequency than the predetermined high-frequency band of the data to the pixel data of the three primary colors of each pixel, the pixel data of each color is widened, thereby causing a reduction in image quality. The subject was photographed because it can suppress the occurrence of phenomena such as false signals crossing the wedge shape of the resolution chart and the cutting of straight lines. Phenomenon occurs in the can than conventional greatly improved.

1 is a schematic block diagram of a digital camera to which an image signal processing method according to the present invention is applied. It is the schematic diagram which looked at the image pick-up part used with the digital camera of FIG. 1 from the incident light side. FIG. 2 is a block diagram illustrating a schematic configuration of an interpolation processing unit of the digital camera of FIG. 1. FIG. 2 is a block diagram illustrating a schematic configuration of a broadband processing unit of the digital camera of FIG. 1. 3 is a main flowchart for explaining the operation of the digital camera of FIG. 1. It is a schematic diagram explaining the relationship between the position and color of the real pixel obtained at the time of the main imaging of FIG. 6 is a flowchart illustrating a procedure of still image interpolation signal processing shown in FIG. 5. The high-frequency luminance data Y _H shown in FIG. 7 is a flowchart illustrating a procedure for generating in a checkered pattern. It is a flowchart explaining the procedure of the adaptive process shown in FIG. 10 is a flowchart for explaining the procedure of the diagonal correlation processing shown in FIG. 9. 10 is a flowchart for explaining a procedure of vertical / horizontal correlation processing shown in FIG. 9. FIG. 9 is a schematic diagram illustrating a positional relationship between pixel data Mg generated by the procedure shown in FIG. 8 and pixel data G obtained at the time of imaging. Pixel data G of Figure 12, the Mg is a schematic diagram showing a pattern when treated as a high-frequency luminance data Y _H. It is a schematic diagram represented by a flag indicating the color and signal level obtained from a pixel by imaging and the direction of high-frequency luminance data Y _H generated by a conventional algorithm. FIG. 12 is a schematic diagram showing a correction part by a newly added vertical / horizontal correlation determination process in the processing procedure of FIG. FIG. 9 is a schematic diagram illustrating a positional relationship of pixels used in the flag pattern comparison correction process of FIG. 8. It is a schematic diagram which shows the result of the comparison correction process of the flag pattern of FIG. It is a flowchart explaining the procedure of the brightness | luminance interpolation process of FIG. FIG. 19 is a schematic diagram for explaining the principle of LPF processing used in the luminance interpolation processing of FIG. FIG. 20 is an interpolation pattern obtained by performing an interpolation process in the horizontal direction in the LPF process of FIG. FIG. 20 shows an interpolation pattern obtained by performing interpolation processing in the vertical direction in the LPF processing of FIG. It is a flowchart explaining the procedure of the color interpolation process of FIG. 6 is a flowchart for explaining a procedure of a still image widening process in FIG. 5. It is a schematic diagram which shows the relationship between a thin line | wire, the color of each pixel data, a signal level, and a flag in case the thin line space | interval of an object field is comparatively large. FIG. 12 is a schematic diagram showing a correction part by a newly added vertical / horizontal correlation determination process in the processing procedure of FIG. It is a schematic diagram which shows the relationship between the color of a thin line | wire, each pixel data, a signal level, and a flag in case the thin line space | interval of a field is comparatively small. FIG. 12 is a schematic diagram showing a correction part by a newly added vertical / horizontal correlation determination process in the processing procedure of FIG. It is a schematic diagram which shows the result of the comparison correction process of the flag pattern of FIG. FIG. 6 is a schematic diagram illustrating a pixel pattern focused on a color G. It is a schematic diagram which shows the pixel pattern which paid its attention to color Mg. FIG. 4 is a schematic diagram showing a plain screen obtained by performing interpolation with a color G 1. FIG. 4 is a schematic diagram showing a plain screen obtained by performing interpolation with a color G 1.

  Next, embodiments of the image signal processing method according to the present invention will be described in detail with reference to the accompanying drawings. Referring to FIG. 1, the embodiment of the image signal processing method according to the present invention will be described when applied to a digital camera 10, and the operation procedure of the applied image signal processing method is the still image interpolation of the main flowchart shown in FIG. Operate according to signal processing (SUB1) and wideband processing (SUB2) for still images. The signal processing procedure for a still image includes red pixel data from an actual pixel at one diagonal position sandwiching green in the still image interpolation signal processing (SUB1) of FIG. 5 and blue from an actual pixel at the other diagonal position. When the pixel data is obtained and each of the red and blue real pixels is set as the target pixel, the vertex of the correlation determination area for the target pixel is at least a pixel of the same color as the target pixel. A pixel of complementary green color using a plurality of other pixel data around the target pixel located in the correlation direction and the other plurality of pixel data while making a determination by improving the correlation accuracy by expanding the correlation to the area to be represented. Data is generated as luminance data, and using the complementary pixel data and green pixel data, luminance data in the virtual pixel is obtained from pixel data of the same color located in the horizontal and / or vertical direction. Interpolating and generating all the red, blue and green colors that are different from the color attributes obtained for all the real and virtual pixels based on the color attributes of the real and virtual pixels obtained, The high-frequency component signal obtained by passing a frequency higher than the predetermined high-frequency band of the luminance data interpolated and generated by the still image broadening process (SUB2) is added to the pixel data of the three primary colors of each pixel, By subjecting the pixel data to broadband processing, it is possible to suppress the occurrence of image quality degradation and the occurrence of phenomena such as false signals crossing the wedge shape of the resolution chart and the cutting of straight lines. Phenomena that occur when shooting can be greatly improved compared to the prior art.

  In the present embodiment, illustration and description of portions not directly related to the present invention are omitted. Here, the reference number of the signal is represented by the reference number of the connecting line that appears.

  As shown in FIG. 1, the digital camera 10 includes an optical lens system 12, an operation unit 14, a system control unit 18, a timing signal generation unit 20, a driver unit 22, a mechanical shutter 24, an imaging unit 26, a preprocessing unit 28, A signal processing unit 30, a compression / decompression processing unit 32, a storage unit 34, and a monitor 36 are provided. Each of these parts will be described sequentially.

  The optical lens system 12 is configured by combining a plurality of optical lenses, for example. Although not shown in the figure, the optical lens system 12 adjusts the position where these optical lenses are arranged, and adjusts the angle of view of the screen according to the operation signal 14a from the operation unit 14, and the distance between the subject and the camera 10. It includes an AF (Automatic Focus) adjustment mechanism that adjusts the focus according to the focus. As will be described later, the operation signal 14a is supplied to the system controller 18 via the system bus 16 and the data bus 18a. The system control unit 18 supplies a control signal 18b to the system bus 16. The control signal 18c is supplied from the system bus 16 to the timing signal generator 20 and the driver unit 22 as the control signal 18c. The timing signal generator 20 and the driver unit 22 operate according to the supplied control signal 18c and output a drive signal 22a to the optical lens system 12.

  Although not shown, the operation unit 14 has a release shutter button, a selection function for displaying various items on a monitor screen, for example, and selecting from the displayed items using a cursor. The operation unit 14 includes a still image / moving image setting unit among these various function selections. The still image / moving image setting unit outputs the set result as another operation signal (not shown). The operation signal 14a is notified to the system control unit 18 via the system bus 16 for the operation selected by the operation unit 14.

  The system control unit 18 includes, for example, a CPU (Central Processing Unit). The system control unit 18 includes a ROM (Read Only Memory) in which the operation procedure of the digital camera 10 is written. The system control unit 18 controls the operation of each unit using, for example, information supplied from the operation unit 14 in response to a user operation via the signal line 18a, that is, the operation signal 14a and information stored in the ROM. Signal 18b is generated. The system control unit 18 supplies the generated control signal 18b to the timing signal generation unit 20 and the driver unit 22 in addition to driving the optical lens system 12 described above via the system bus 16, and is not shown. The data is also supplied to the processing unit 28, the signal processing unit 30, the compression / decompression processing unit 32, the storage unit 34, and the monitor 36.

  The timing signal generator 20 includes an oscillator (not shown) that generates a basic clock (system clock) for operating the digital camera 10. For example, a VCO (Voltage Controlled Oscillator) method or the like is used as the oscillator. The timing signal generator 20 supplies the basic clock to almost all the necessary blocks such as the system controller 18 and the signal processor 30, and also divides the basic clock to generate various signals.

  In particular, the timing signal generator 20 includes a circuit that generates timing signals 20a and 20b that use the basic clock to adjust the timing of the operation of each unit based on the control signal 18b. The timing signal generator 20 supplies the generated timing signal 20a to the driver unit 22. The timing signal generation unit 20 also generates and supplies a timing signal 20b so as to be supplied as an operation timing of the preprocessing unit 28 and the like. In addition, as shown in FIG. 1, various timing signals are supplied to the respective parts, although not shown.

  The driver unit 22 generates desired drive signals 22a, 22b, and 22c using the supplied timing signal 20a, and supplies them to the optical lens system 12, the mechanical shutter 24, and the imaging unit. The drive signal 22c is supplied corresponding to the still image mode and the moving image mode.

  The mechanical shutter 24 operates in response to the pressing operation of the release shutter button of the operation unit 14. For example, when the release shutter button is pressed, the operation signal 14a is supplied to the system control unit 18 via the system bus 16, and the control signal 18b from the system control unit 18 is supplied to the system bus 16 and the signal line 18c. Is supplied to the driver unit 22 via The mechanical shutter 24 operates according to the drive signal 22b supplied from the driver unit 22. The mechanical shutter 24 is operated and controlled in this order.

  The imaging unit 26 is a single-plate color CCD sensor in which an optical low-pass filter 26a and a color filter 26b are integrally provided on the incident light side of a solid-state imaging device (Charge Coupled Device: CCD) 26c. This outputs an output signal 27 corresponding to the amount of light that the optical image formed by the optical lens system 12 reaches each image sensor of the light receiving unit of the solid-state image sensor 26c.

  In the imaging unit 26, an optical low-pass filter 26a, a color filter 26b, and a solid-state imaging element 26c are sequentially arranged from the incident light side, and are integrally formed. The color filter 26b is a single plate. The color filter segment 260 of the color filter 26b and the image sensor 262 have a one-to-one correspondence. In the color filter 26b, for example, a color filter segment 260 of the three primary colors RGB as shown in FIG. 2 is arranged. The arrangement pattern of the color filter segments 260 is a pattern arranged in a completely checkered pattern in which the color G is arranged in a square lattice and the same color R or B is arranged diagonally across the color G 1.

  This color pattern is hereinafter referred to as a honeycomb-type G square lattice RB complete checkered pattern. The number of pixels shown in FIG. 2 is 6 of the 14 original colors G and 4 each of the colors R and B. The above-mentioned square lattice shape of the color G does not indicate the shape of the pixel but indicates the arrangement shape of the pixel. The shape of the pixel includes polygonal data such as a quadrangle, a hexagon, and an octagon.

  A CCD or a metal oxide semiconductor (MOS) type solid-state imaging device shown later is applied to the imaging device. In the imaging unit 26, the signal charge obtained by photoelectric conversion in accordance with the supplied drive signal 22c is read out to the vertical transfer path by a field shift, for example, in the vertical blanking period as a predetermined timing, and this vertical transfer path The signal charge obtained by shifting the line is supplied to the horizontal transfer path. The signal charge that has passed through the horizontal transfer path is converted into an analog voltage signal 27 by charge / voltage conversion by an output circuit (not shown) and output to the preprocessing unit 28. For the CCD type, the solid-state imaging device 26c uses thinning readout or all-pixel readout according to the signal charge readout mode.

  Although not shown, the preprocessing unit 28 includes a correlated double sampling unit (hereinafter referred to as CDS) and an A / D conversion unit. The CDS unit removes noise by contributing to the reduction of 1 / f noise and reset noise contained in the analog voltage signal. Further, the preprocessing unit 28 may perform gamma correction here. The output signal from which the noise component has been removed is sent to the A / D converter. The A / D converter includes an A / D converter that quantizes the signal level of the supplied analog signal with a predetermined quantization level and converts it into a digital signal 29. The A / D converter outputs the digital signal 29 converted by the timing signal 20b such as a conversion clock supplied from the timing signal generator 20 to the signal processor 30.

  The signal processing unit 30 includes a data correction unit 30a having a frame memory function, an interpolation processing unit 30b, and a broadband processing unit 30c. Further, although not shown, the data correction unit 30a includes a frame memory and a correction processing unit as a buffer function. The interpolation processing unit 30b has a function of performing adjustments such as gamma correction and white balance, although not shown. Here, the gamma correction also makes it possible to reduce the number of bits for signal processing by reducing power consumption and circuit scale. If this gamma correction processing is already performed by the preprocessing unit 28, for example, the description is omitted. The image data 29 digitized by A / D conversion is supplied to the frame memory of the data correction unit 30a and stored therein. Since the frame memory repeatedly reads out pixel data while shifting the reading area, it is advantageous to use a non-destructive memory. The frame memory is supplied with signals relating to control such as a write / read enable signal and a clock signal included in the control signal 18b from the system control unit 18 as a control signal 18d via the system bus 16. For example, the data correction unit 30a also performs gain adjustment for each color filter in order to adjust the level of the image data corresponding to each color filter to a level suitable for subsequent signal processing. Further, the data correction unit 30a outputs the stored image data to the interpolation processing unit 30b in a predetermined order.

In the present embodiment, the interpolation processing unit 30b has a function of performing still image interpolation and moving image interpolation, and switches the supply destination of the supplied image data 38 according to the user's request and supplies it to each unit. Among these, a configuration for performing still image interpolation will be described. The interpolation processing unit 30b includes a luminance interpolation developing unit 40 and a color interpolation developing unit 42 as shown in FIG. Interpolation processing unit 30b has a function for interpolating generates a high-frequency luminance data Y _H and the three primary colors data including high-frequency component in the virtual pixel located in the middle of the actual pixel and existing pixels. Image data 38 is supplied from the data correction unit 30a to the luminance interpolation development unit 40 and the color interpolation development unit 42, respectively. The image data 38 is pixel data from actual pixels that exist in the image sensor.

The luminance interpolation development unit 40 includes a high-frequency luminance data creation unit 400 and a luminance data interpolation processing unit 402. The high-frequency luminance data creation unit 400 generates high-frequency luminance data Y _H (404) including a high frequency component at the position of the real pixel or the virtual pixel using the supplied pixel data. The high frequency luminance data creation unit 400 of the present embodiment calculates the high frequency luminance data Y _H at the position of the actual pixel. In this calculation, for example, the direction of the correlation with the supplied color is detected using the supplied pixel data 38, and high-frequency luminance data Y _H (404) corresponding to the detection result is generated. More detailed calculation will be described later. The luminance data interpolation development unit 402 generates the high frequency luminance data Y _H at the position of the virtual pixel using the supplied high frequency luminance data Y _H (404). The luminance interpolation development unit 40 outputs the high frequency luminance data Y _H (44) at the positions of all the real pixels and virtual pixels to the wideband processing unit 30c by generating and interpolating the high frequency luminance data Y _{H in} this way.

Incidentally, the luminance interpolator expansion unit 40 creates a high-frequency luminance data Y _H corresponding to the position of the virtual pixel, a high-frequency luminance data Y _H corresponding to the position of existing pixels by using the high-frequency luminance data Y _H of the virtual pixel You may make it create.

  The color interpolation development unit 42 has a function of performing interpolation processing on colors that do not correspond to virtual pixels and real pixels in consideration of the color arrangement of real pixels and generating three primary color data of the entire screen. For this reason, the color interpolation development unit 42 has an interpolation development unit for each color. This is an R interpolation development unit 420, a G interpolation development unit 422, and a B interpolation development unit 424. Each interpolation development unit 420, 422, 424 inputs color data 38a, 38b, 38c for each color from the supplied image data 38, and performs interpolation development processing on each input color data, thereby realizing the actual pixel. Color data is generated at the positions of all virtual pixels. As a result, all three primary colors RGB are prepared, and RGB simultaneous processing is performed to output the three primary color data 46, 48, and 50 to the broadband processing unit 30c. The interpolation processing of these three primary color data RGB will also be described in detail later.

The broadband processing unit 30c includes a high-pass filter circuit 52, an adding unit 54, a color difference matrix unit 56, a distortion prevention processing unit 58, an aperture adjustment unit 60, and color difference gain adjustment units 62 and 64 (see FIG. 4). ). A high-pass filter circuit (hereinafter referred to as HPF) 52 is a filter that extracts high-frequency components from the supplied high-frequency luminance data 44. The HPF 52 outputs the extracted signal 52 a of the high frequency component Y _h to the adding unit 54. Further, a new changeover switch (not shown) may be disposed between the HPF 52 and the adder 540 for the color G so that the high frequency component is not directly supplied to the color G. In this case, a change control signal is supplied from the system control unit 18 to the changeover switch.

The adder 54 includes three adders 540, 542, and 544 corresponding to the three primary colors. The three primary color data RGB are supplied to one ends 540a, 542a, and 544a of the adders 540, 542, and 544, respectively. In addition, high frequency components are supplied to the other ends 540b, 542b, and 544b of the adders 540, 542, and 544. Addition unit 54, to widen the frequency band of the three primary color data by adding the high frequency components Y _h from HPF 52 to each of the three primary RGB. By the way, since the pixel data of the color G itself contains a high frequency component, it can be considered that the band is widened.

The color difference matrix unit 56 generates luminance data Y and color difference data C _r , C _b using the three primary color data 54a, 54b, 54c having a wide band. The color difference matrix unit 56 uses the conventional calculation formulas used so far for the matrix calculation performed here. The color difference matrix unit 56 supplies the generated luminance data Y (56a) and color difference data C _r (56b), C _b (56c) to the distortion prevention processing unit 58.

The distortion prevention processing unit 58 includes low-pass filters (hereinafter referred to as LPFs) 580, 582, and 584 that have a band up to a high band so as not to cause aliasing distortion without impairing the band of the three signals that are supplied. . Among them, the LPF 580 has a characteristic of passing up to the highest frequency according to the luminance data Y. Further, the distortion prevention processing unit 58 limits the overlapping frequency bands in one direction with respect to the overlapping region of the horizontal and vertical frequency bands. This avoids image quality degradation due to frequency overlap. The luminance data Y (58a) and the color difference data C _r (58b) and C _b (58c) processed in this way are supplied to the aperture adjustment unit 60 and the color difference gain adjustment units 62 and 64, respectively.

The aperture adjustment unit 60 processes and outputs so as to eliminate the decrease in the high frequency component by LPF processing, for example. As a result, the image has the same effect as the contour (edge) enhancement processing. Further, the color difference gain adjusting units 62 and 64 perform gain adjustment on the supplied color difference data C _r (58b) and C _b (58c), and align them to a predetermined level. In this way, the signal processing unit 30 supplies the generated luminance data Y (66a) and color difference data C _r (66b) and C _b (66c) to the compression / decompression processing unit 32.

Returning to FIG. 1, the compression / decompression processing unit 32 includes a frame memory that temporarily stores luminance data Y and color difference data C _r and C _b supplied from the signal processing unit 30, for example, orthogonal transformation And a circuit that performs compression in accordance with the JPEG (Joint Photographic Experts Group) standard using the image and a circuit that decompresses the compressed image to the original data again. Here, the frame memory may also be used as the frame memory of the signal processing unit 30. Here, possessing a plurality of frame memories is advantageous in recording a moving image, that is, in processing such as continuous shooting. The compression is not limited to JPEG, and there is a compression method such as MPEG (Moving Picture coding Experts Group) or motion JPEG that applies JPEG to each frame of a moving image.

The compression / decompression processing unit 32 supplies the compressed data to the storage unit 34 via the system bus 16 during recording under the control of the system control unit 18. The compression / decompression processing unit 32 passes the supplied luminance data Y (66a) and color difference data C _r (66b), C _b (66c) to the system bus 16 under the control of the system control unit 18, Alternatively, the signal from the signal processing unit 30 can be supplied to the monitor 36 via the system bus 16. When the compression / decompression processing unit 32 performs the decompression processing, the data read from the storage unit 34 is conversely taken into the compression / decompression processing unit 32 via the system bus 16 and processed. After the data processed here is stored in the frame memory, the compression / decompression processing unit 32 reads the data in the frame memory in the required order under the control of the system control unit 18 and supplies it to the monitor 36 for display.

  The storage unit 34 includes a recording processing unit for recording on a recording medium and a reproduction processing unit for reading image data recorded from the recording medium (both not shown). Examples of the recording medium include a semiconductor memory such as a so-called smart media (registered trademark), a magnetic disk, and an optical disk. In the case of using a magnetic disk or an optical disk, there is a head for writing the image data together with a modulation unit that modulates the image data.

The monitor 36 considers the screen size and adjusts the timing of luminance data Y and color difference data C _r , C _b or three primary color RGB data supplied via the system bus 16 according to the control of the system control unit 18. The display function.

  In this way, the digital camera 10 is configured to perform signal processing on the captured image and record it. Also, the recorded image data is reproduced and displayed on the monitor 36.

  Next, the operation of the digital camera 10 will be described. The drawings used in the above-described configuration are also referred to as necessary. The digital camera 10 operates, for example, according to a main flowchart in photographing shown in FIG. This operation procedure has been described focusing on still image shooting as described above. Although the digital camera 10 is not shown, after the power is turned on, a mode to be operated is selected from among a plurality of modes of the camera 10. When the photographing mode is selected, the image of the object scene taken through the optical lens system 12 is digitally processed and continuously displayed on the monitor 36 (step S10). In particular, when a release shutter button (not shown) of the operation unit 14 is pressed in one step, the camera 10 not only displays the captured image of the object scene on the monitor 36 but also sets the exposure conditions before the actual imaging. The system control unit 18 adjusts the focal length with respect to the processing and exposure target, and prepares for the main imaging. The parameter setting process related to the setting of the exposure conditions may be performed by the signal processing unit 30.

  The user presses the release shutter button up to two stages at a desired timing to perform the main imaging (step S12). When the main imaging is performed, the system control unit 18 is supplied from the operation signal 14a. The system control unit 18 sends a control signal 18b set in advance at this timing to the timing signal generation unit 20 and the driver unit 22 via the system bus 16 and the signal line 18c. The camera 10 operates according to the timing signal and the drive signal from the timing signal generation unit 20 and the driver unit 22. As shown in FIG. 2, the solid-state imaging device 26 c converts the signal charge read from the pixel corresponding to the color filter segment RGB into the analog signal 27 through the color filter 26 b of the honeycomb type G square lattice RB complete checkered pattern. The read analog signal 27 is supplied to the preprocessing unit 28.

  In the preprocessing unit 28, noise included in the analog signal 27 supplied in accordance with the timing signal 20b is removed, and the signal from which noise has been removed is subjected to A / D conversion processing to be converted into a digital signal 29 (step S14). . The digital signal 29 is supplied to the signal processing unit 30 as captured image data.

  The signal processing unit 30 sequentially performs still image interpolation signal processing (subroutine SUB1) and still image widening processing (subroutine SUB2) on the supplied image data. Assume that the supplied image data 29 is, for example, pixel data related to pixel arrangement shown in FIG. The octagonal solid line represents a real pixel, and the octagonal broken line represents a virtual pixel. Symbols R, G, and B written inside the octagon indicate colors generated by each pixel. Subscripts attached to symbols R, G, and B indicate pixel positions in a matrix display.

  The image data 29 is subjected to various corrections such as gamma correction and white balance correction on the pixel data corresponding to each pixel by the data correction unit 30a before the still image interpolation signal processing. The data correction unit 30a may include a frame memory (not shown). In this case, the image data is temporarily stored when supplied, and then read out for each pixel and subjected to the above-described correction. The corrected image data 38 may be stored again in the frame memory or may be supplied to the interpolation processing unit 30b.

The interpolation processing unit 30b performs interpolation signal processing for still images based on the supplied image data 38. As will be described in detail later, with this interpolation signal processing, effective resolution enhancement processing is performed in the honeycomb-type G square lattice RB complete checkered pattern. The generated high-frequency luminance data Y _H (44) and plain color data 46, 48, 50 are output to the broadband processing unit 30c.

The broadband processing unit 30c performs broadband processing on the supplied high-frequency luminance data Y _H (44) and plain color data 46, 48, and 50, respectively, to generate broadband luminance data Y (66a) and color difference data. (BY) (66b), (RY) (66c) are generated.

  Next, compression processing is performed on the image data 66a, 66b, 66c obtained by the signal processing (step S16). Although the compression processing is not shown in the figure, in the moving image mode, the processing is performed by motion JPEG (Joint Photographic coding Experts Group) or MPEG (Moving Picture coding Experts Group) which performs compression processing for each frame. In the still image mode, JPEG or the like is applied as compressed signal processing. The compressed image data is recorded and stored in a recording medium attached to the recording / reproducing apparatus of the storage unit 34 (step S18).

  Finally, it is determined whether or not to end the operation of the digital camera 10 (step S20). When the operation is still continued (NO), the process returns to step S10 to continue the series of processes. When the operation is finished (YES), the power of the digital camera 10 is turned off. It is made to operate in this way.

Next, the interpolation process will be schematically described (subroutine SUB1: see FIG. 7). The interpolation processing includes subroutines SUB3, SUB4, and SUB5, which correspond to the processing in the high-frequency luminance data creation unit 400, luminance data interpolation processing unit 402, and color interpolation development unit 42 in FIG. In the present embodiment, the high-frequency luminance data creation processing creates the high-frequency luminance data Y _H for the positions of the existing pixels in a checkered pattern by the high-frequency luminance data creation unit 400. After this processing, the interpolation of the luminance data using the high-frequency luminance data Y _H created. The interpolation here is processing performed on the virtual pixel. Further, when the high-frequency luminance data creation process is performed on the virtual pixel, the luminance interpolation process is performed on the actual pixel.

The high-frequency luminance data Y _H (44) is generated and color interpolation processing is performed (subroutine SUB5: RGB synchronization processing). With this process, pixel data 46, 48, and 50 of primary colors R, G, and B are generated not only on the actual pixels but also on the entire pixel surface including the virtual pixels.

Further, the procedure of high-frequency luminance data creation processing (subroutine SUB3) will be described (see FIGS. 8 to 11). First, in this embodiment, adaptive processing for generating high-frequency luminance data Y _H while considering the presence or absence of correlation between pixel data using pixel data obtained from each pixel as shown in FIG. It is determined whether or not to perform (sub step SS30). Whether or not to execute the adaptive processing may be set in advance by the operation unit 14. For example, the default of this setting is set to perform adaptive processing. If adaptive processing is to be performed (YES), the process proceeds to subroutine SUB6. The adaptive processing will be described in further detail later. If you do not adaptive processing (NO), and the average pixel data of the four surrounding pixels are arranged in a square lattice and generates a high-frequency luminance data Y _H by (substep SS32). The calculation formula is shown later.

For example, when the generation of the high-frequency luminance data Y _{H at} a certain pixel is completed, it is determined whether or not the interpolation processing for the existing pixels for one frame has been completed at this time (substep SS34). If the interpolation process for one frame has not been completed yet (NO), the process returns to the top of the subroutine SUB3. When this interpolation processing is completed (YES), comparison correction is performed on each pixel data in the obtained image data of one frame (sub step SS36). After the comparison correction, the process proceeds to return and the subroutine SUB3 is terminated. This comparison correction will be described again later.

Next, adaptive processing will be described. Oblique direction adaptive processing according to the user's setting, and calculates the high-frequency luminance data Y _H based on the correlation determination processing in the horizontal direction, and / or vertical direction. First, as shown in FIG. 9, it is determined whether or not the diagonal correlation determination process is to be performed (substep SS60). In this case as well, whether or not the process is possible is determined using the on / off of the flag set in the camera 10. Good. When oblique correlation processing is performed (YES), high-frequency luminance data Y _H is actually calculated according to the correlation determination processing (subroutine SUB7). When the diagonal correlation determination process is not performed (NO), the process proceeds to the calculation process of the high-frequency luminance data Y _H based on the vertical / horizontal correlation (subroutine SUB8).

  After the diagonal correlation determination process, it is determined whether or not to perform another correlation determination process on the pixel data obtained from the target pixel. When other correlation determination processing is performed (NO), the process proceeds to subroutine SUB8. If no other correlation determination process is performed (YES), the process proceeds to return and the subroutine SUB6 is terminated.

Next, the diagonal correlation determination process will be described (see FIG. 10). Comparison data is calculated in the diagonal correlation determination process (substep SS70). The comparison data is used, for example, to determine in which direction the pixel data around the pixel data to be subjected to adaptive processing is correlated. For example, when the target pixel data is the color B ₄₆ shown in FIG. 6, the comparison data AG uses surrounding pixel data G ₃₅ , G ₅₅ , G ₃₇ , G ₅₇ ,
_{AG = | G 35 + G 57} - (G 37 + G 55) | ··· (1)
Obtained from. Even when the pixel data is color R, it is calculated from the surrounding pixel data G. By this calculation, a larger value having a slope on either the left or right side is obtained as the comparison data AG.

After this calculation, it is determined whether or not there is a correlation (that is, a diagonal correlation) in pixel data located diagonally across the target pixel data (sub-step SS70). In this determination, J1 is set as a determination reference value. When comparing data AG is greater than the determination reference value J1 (YES), and generates the high-frequency luminance data Y _H is determined that there is a diagonal correlation (substep SS74). Generating high-frequency luminance data Y _H is performed by averaging the four pixels data G used for calculation of the comparison data AG. When the comparison data AG is smaller than the determination reference value J1 (NO), other correlation determination processing flags are turned on (substep SS76).

By the way, even in this case, there is a possibility of generating false colors. Therefore, if the image quality in the pixel data R located near the boundary of the pixel having this fear is calculated by the above-described calculation of the high-frequency luminance data Y _H , the occurrence of false color at the color boundary can be satisfactorily viewed as the entire image. Can be suppressed. Although not specific description can create an adaptive high-frequency luminance data Y _H based on the presence or absence of the calculated slant correlation similarly compare data against the pixel data B = B _24. After this processing, the process proceeds to return and the subroutine SUB7 is terminated.

Next, the vertical / horizontal correlation determination process will be described (see FIG. 11). With utilizing arrangement pattern also vertical / horizontal correlation determination process shown in FIG. 6, illustrating a target pixel to be processed in the color B _46. The vertical / horizontal correlation determination includes eight actual pixels (R ₂₆ , G ₃₅ , G ₃₇ , R ₄₄ , R ₄₈ , G ₅₅ , G ₅₇ , and R ₆₆ ) centered on the target pixel B ₄₆ and the periphery thereof. Region 68 is used.

In this embodiment, a region 70 surrounding the region 68 is also considered in order to further improve the accuracy of the vertical / horizontal correlation determination. The pixels actually used for the correlation determination are the four real pixels (B ₀₆ , B ₄₂ , B ₄₁₀ , B ₈₆ ) in the region 70 that have the same color as the target pixel B ₄₆ and are positioned at the four corners of the region 70. That is, the data of the same color pixel located in the cross direction that is horizontal and vertical with respect to the target pixel B ₄₆ is used.

In this embodiment, the comparison data is calculated for the area 68 and area 70 (substep SS800). The comparison data in the area 68 is ARB _H , ARB _V , ACB _H , ACB _V , AGB _H and / or AGB _V , and the comparison data in the area 70 is ACBB _H , ACBB _V. The first character “A” representing the comparison data is an arithmetic operation, the second character “R” or “G” is the color of the pixel used for the arithmetic operation, or “C” is a comparison operation with the target pixel, the third The letter “B” is the color of the pixel used for the arithmetic operation, or the color of the target pixel, the fourth letter is the color of the target pixel, and the subscripts H and V for the third or fourth letter represent horizontal and vertical, respectively. ing.

For example, the comparison data for the target pixel B ₄₆ is calculated as follows:
ARB _H = | R ₄₄ −R ₄₈ | (2a)
ARB _V = | R ₂₆ −R ₆₆ | (2b)
ACB _H = | R ₄₄ −B ₄₆ | + | R ₄₈ −B ₄₆ | (2c)
ACB _V = | R ₂₆ −B ₄₆ | + | R ₆₆ −B ₄₆ | (2d)
AGB _H = | G _35- G ₃₇ | + | G _55- G ₅₇ | (2e)
_{AGB V = | G 35 -G 55} | + | G 37 -G 57 | ··· (2f)
ACBB _H = | B ₄₂ −B ₄₆ | + | B ₄₁₀ −B ₄₆ | (2g)
ACBB _V = | B _06- B ₄₆ | + | B _86- B ₄₆ | (2h)
To obtain. Further, the comparison data may be performed before the correlation determination in each vertical / horizontal direction. For the correlation determination in the vertical / horizontal direction, determination reference values J2, J3, and J4 are set in advance for each pair of vertical and horizontal. These values J2, J3, and J4 are set empirically.

Then correlation pixel data positioned vertically across the pixel data B = B ₄₆ subjects (i.e., the vertical correlation) a determination whether there is a (substep SS802). This determination is performed when a correlation value (ARB _H −ARB _V ) is further calculated using the values of the comparison data ARB _V in the vertical direction and the comparison data ARB _{H in} the horizontal direction. When the value is smaller than the value J2 (NO), it is considered that there is no vertical correlation and the process proceeds to the horizontal correlation determination. When the correlation value (ARB _H −ARB _V ) is equal to or greater than the determination reference value J2 (YES), it is considered that there is a vertical correlation, and the process proceeds to generation of the high-frequency luminance data Y _H. This is because the correlation means that the values of the pixel data are close to each other.

When it is determined that there is a vertical correlation, the high-frequency luminance data Y _H is calculated using the target pixel B ₄₆ and the pixels R ₂₆ and R ₆₆ (substep SS804). Frequency luminance data Y _H is basically the pixel data G and the color G complementary, i.e. can be represented by either of the magenta Mg. It is known that magenta Mg can be calculated by an average of pixel data R and pixel data B (0.5 * R + 0.5 * B). In this case, the high frequency luminance data Y _H46 is
_{Y H46 = B 46/2 +} (R 26 + R 66) / 4 ··· (3)
Is obtained. Flag v is attached to a vertical correlation frequency luminance data Y _H. Thereafter, it is considered that the calculation of the high-frequency luminance data Y _H in the pixel B ₄₆ is finished, and the process proceeds to return.

Next, it is determined whether or not there is a correlation (that is, horizontal correlation) in the pixel data positioned horizontally across the target pixel data B = B ₄₆ (substep SS806). When the correlation value (ARB _V −ARB _H ) is calculated and is smaller than the determination reference value J2 (NO), it is considered that there is no horizontal correlation, and the process proceeds to the next vertical correlation determination. When the correlation value (ARB _V −ARB _H ) is equal to or greater than the determination reference value J2 (YES), it is considered that there is a horizontal correlation, and the process proceeds to generation of the high-frequency luminance data Y _H.

When it is determined that there is a horizontal correlation, the high-frequency luminance data Y _H is calculated using the target pixel B ₄₆ and the pixels R ₄₄ and R ₄₈ (substep SS808). In this case, the high frequency luminance data Y _H46 is
_{Y H46 = B 46/2 +} (R 44 + R 48) / 4 ··· (4)
Is obtained. It is denoted by the flag h in the horizontal correlation frequency luminance data Y _H. Thereafter, it is considered that the calculation of the high-frequency luminance data Y _H in the pixel B ₄₆ is finished, and the process proceeds to return.

Next, vertical correlation determination is performed using real pixels located around the target pixel B ₄₆ for calculating the high-frequency luminance data Y _H (sub-step SS810). When the vertical correlation determination is performed by reducing the distance between the target pixel B ₄₆ and the surrounding pixels, the comparison data ACB _H and ACB _V and a predetermined determination reference value J3 are used. That is, when the correlation value (ACB _H −ACB _V ) is smaller than the determination reference value J3 (NO), it is determined that there is no vertical correlation, and the process proceeds to the next horizontal correlation determination. When the correlation value (ACB _H −ACB _V ) is equal to or greater than the determination reference value J3 (YES), it is determined that there is a vertical correlation, and the process proceeds to calculation of the high-frequency luminance data Y _H.

Note that the vertical correlation may be determined using a pixel closest to the target pixel B ₄₆ in the region 68, that is, a pixel of color G. In this case, comparison data AGB _H and AGB _V are used. The correlation value (AGB _H −AGB _V ) and the determination reference value J3 may be compared and determined. The size criteria for comparison determination are the same as described above.

When it is determined that there is a vertical correlation, the high-frequency luminance data Y _H is calculated using the target pixel B ₄₆ and the pixels R ₂₆ and R ₆₆ (substep SS812). In this case, the high frequency luminance data Y _H46 is obtained by the equation (3). Flag v is attached to a vertical correlation frequency luminance data Y _H. Thereafter, it is considered that the calculation of the high-frequency luminance data Y _H in the pixel B ₄₆ is finished, and the process proceeds to return.

Next, horizontal correlation determination is performed using the existing pixels located around the target pixel B ₄₆ for calculating the high-frequency luminance data Y _H (sub-step SS814). When the horizontal correlation determination is performed by reducing the distance between the target pixel B ₄₆ and the surrounding pixels, the comparison data ACB _H and ACB _V and a predetermined determination reference value J3 are used in this case as well. The correlation value is (ACB _V −ACB _H ). When the correlation value (ACB _V −ACB _H ) is smaller than the determination reference value J3 (NO), it is determined that there is no horizontal correlation, and the process proceeds to the next vertical correlation determination. When the correlation value (ACB _V −ACB _H ) is equal to or greater than the determination reference value J3 (YES), it is determined that there is horizontal correlation, and the process proceeds to the calculation of the high-frequency luminance data Y _H.

Here, as described above, the horizontal correlation may be determined in a narrower region using the pixel closest to the target pixel B ₄₆ in the region 68, that is, the pixel of the color G. In this case, comparison data AGB _H and AGB _V are used. The correlation value (AGB _V −AGB _H ) and the determination reference value J3 may be compared and determined. The size criteria for the comparison determination are as described above.

When it is determined that there is a horizontal correlation, the high-frequency luminance data Y _H is calculated using the target pixel B ₄₆ and the pixels R ₄₄ and R ₄₈ (substep SS816). In this case, the high frequency luminance data Y _H46 is obtained by the equation (4). It is denoted by the flag h in the horizontal correlation frequency luminance data Y _H. Thereafter, it is considered that the calculation of the high-frequency luminance data Y _H in the pixel B ₄₆ is finished, and the process proceeds to return.

In order to enhance detection of vertical / horizontal correlation, detection determination of vertical / horizontal correlation is performed in two stages as described above. This two-stage correlation determination is performed in a direction to narrow the correlation detection range with respect to the target pixel. However, for example, when capturing a longitudinal line-shaped object with a specific width, suitable high frequency luminance data Y _H is in some cases compromise the quality of can not be obtained. The range of correlation detection is performed in a region 70 wider than the region 68 so as to prevent such deterioration in image quality at a specific width.

Next, vertical correlation determination is performed using real pixels located around the target pixel B ₄₆ for calculating the high-frequency luminance data Y _H in the region 70 (substep SS818). In this case, the target pixel B ₄₆ and the four surrounding pixels B ₀₆ , B ₄₂ , B ₄₁₀ , B ₈₆ are used. The surrounding pixels to be used have the same color as the target pixel. When comparing data ACBB _H, correlation value calculated from ACBB _V (ACBB _H -ACBB _V) is smaller than the determination reference value J4 (NO), the process proceeds to the next horizontal correlation determination. When the correlation value (ACBB _H −ACBB _V ) is equal to or greater than the determination reference value J4 (YES), it is determined that there is a vertical correlation, and the process proceeds to calculation of the high-frequency luminance data Y _H.

When it is determined that there is a vertical correlation, the high-frequency luminance data Y _H is calculated using the target pixel B ₄₆ and the pixels R ₂₆ and R ₆₆ (substep SS820). In this case, the high frequency luminance data Y _H46 is obtained by the equation (3). Flag v is attached to a vertical correlation frequency luminance data Y _H. Thereafter, it is considered that the calculation of the high-frequency luminance data Y _H in the pixel B ₄₆ is finished, and the process proceeds to return.

Next, in the same manner as described above, vertical correlation determination is performed using real pixels located around the target pixel B ₄₆ for calculating the high-frequency luminance data Y _H in the region 70 (sub-step SS822). Also in this case, four surrounding pixels B ₀₆ , B ₄₂ , B ₄₁₀ , B ₈₆ having the same color as the target pixel B ₄₆ are used. Comparative data ACBB _H, calculates a correlation value (ACBB _V -ACBB _H) from ACBB _V. When the correlation value (ACBB _V −ACBB _H ) is smaller than the determination reference value J4 (NO), it is determined that there is no correlation, and the process proceeds to the surrounding four-pixel averaging process. When the correlation value (ACBB _V −ACBB _H ) is equal to or greater than the determination reference value J4 (YES), it is determined that there is a horizontal correlation, and the process proceeds to calculation of the high-frequency luminance data Y _H.

When it is determined that there is a horizontal correlation, the high-frequency luminance data Y _H is calculated using the target pixel B ₄₆ and the pixels R ₂₆ and R ₆₆ (substep SS824). In this case, the high frequency luminance data Y _H46 is obtained by the equation (4). It is denoted by the flag h in the horizontal correlation frequency luminance data Y _H. Thereafter, it is considered that the calculation of the high-frequency luminance data Y _H in the pixel B ₄₆ is finished, and the process proceeds to return.

When it is determined that there is no correlation, the surrounding four-pixel averaging process is performed (sub-step SS826). The high-frequency luminance data Y _H of the target pixel B ₄₆ is calculated using pixel data obtained from all the existing pixels in the region 68. The high frequency luminance data Y _H is
_{Y H46 = B 46/2 +} (R 26 + R 44 + R 410 + R 66) / 8 ··· (5)
Is obtained. The high-frequency luminance data Y _H, flag indicating that it has been calculated without correlation + is attached. The flag + means that magenta Mg is generated by adding different colors to the target pixel color and the target pixel color. That is, the flag + is the addition of all surrounding pixels. Thereafter, it is considered that the calculation of the high-frequency luminance data Y _H in the pixel B ₄₆ is finished, and the process proceeds to return. The subroutine SUB8 is terminated via return. As a result, pixel data of the color Mg is newly calculated at the positions of the colors R and B together with the actual pixels of the color G obtained by imaging (see FIG. 12). These color G, Mg pixel data from the handle as the high frequency luminance data Y _H, the high-frequency luminance data Y _H is generated corresponding to all positions of existing pixels shown in FIG. 13.

By the way, as described above, the luminance may be obtained in a long line shape with a specific width on the subject. At this time, a luminance level difference occurs in the longitudinal direction. As a specific example, as shown in FIG. 14, a luminance level difference occurs in a portion of a subject where yellow ye is sandwiched between white W 1. However, the color G is omitted. The white W region is obtained, for example, at the same level of pixel data of three primary colors RGB. In addition, the yellow ye area 72 (b +, R +, b +, R +, b +) represents, for example, that the pixel data of the color R and the color G have the same level, and the color B represents the yellow with the pixel data of the small level b. Yes. The generation of the high-frequency luminance data Y _H in the yellow region 72 is obtained by the averaging of four surrounding pixels as indicated by the flag +. When the high-frequency luminance data Y _H for the pixels R + and b + is Y _H (R +) and Y _H (b +), respectively, the equations (6a) and (6b)
Y _H (R +) = R / 2 + (B + b + B + b) / 8 = R / 2 + (B + b) / 4 (6a)
Y _H (b +) = b / 2 + (R + R + R + R) / 8 = R / 2 + R / 2 (6b)
Is obtained. The level difference of the luminance data in the longitudinal direction is defined as ΔY _{H when} the level difference between the high frequency luminance data Y _H (R +) and Y _H (b +) is
ΔY _H = Y _H (R +) − Y _H (b +) = (B−b) / 4 (7)
Is obtained. The luminance level difference ΔY _H is a cause of unevenness.

As a countermeasure against this, a vertical / horizontal correlation determination process in the area 70 shown in FIG. 6 is provided. The newly added vertical / horizontal correlation determination process is performed using pixels of the same color including the target pixel. The pixel data to be used shown in FIG. 14 is the previous pixel data indicated by the arrow 74. 11 indicates that the correlation determination processing is performed on the pixel of color B drawn by the thick line in FIG. 15 from the flag + to the flag h by the procedure after sub-step SS818 shown in FIG. In this way, the calculation processing of the high-frequency luminance data Y _H for one frame is performed. Performing comparison correction flag pattern for one frame of the high-frequency luminance data Y _H (see sub-step SS36 of FIG. 8).

  The comparison correction uses 13 real pixels including the color B of the target pixel (see FIG. 16). In the area 76 including 13 real pixels, the pixels in the top row are u1, u2, u3, the two pixels in the middle row are m1, m2, and the pixels in the bottom row are d1, d2, d3. . In this way, it is determined whether or not to correct the flags at the positions of the colors R and B using the surrounding eight pixels for the comparison correction target pixel.

The determination condition is that the flag of the pixel for comparison correction is the flag of the peripheral pixel when the flags of the pixels in the top and bottom rows are the same, or when the flags of the pixels in the left and right columns are the same with respect to the pixel for comparison correction To correct. For example, if each pixel has a u1 = u2 = u3 = d1 = d2 = d3 or u1 = m1 = d1 = u3 = m2 = d3 flag indicating a horizontal correlation h, the flag for the pixel b for comparison correction is Set to h. See color Bh in FIG. 17 for a more specific example. This flag corrections and simultaneously calculates the high-frequency luminance data Y _H based on the horizontal correlation.

In addition, when each pixel has a vertical correlation v indicating that u1 = u2 = u3 = d1 = d2 = d3 or u1 = m1 = d1 = u3 = m2 = d3 Set to v and calculate high-frequency luminance data Y _H based on the vertical correlation. The high-frequency luminance data Y _H calculated based on the comparison correction is used. Thereby, the luminance level difference ΔY _H is eliminated.

Next, luminance data interpolation processing will be described (subroutine SUB4: see FIG. 18). And it generates the high-frequency luminance data Y _H at the position of the virtual pixel based on the high-frequency luminance data Y _H placement obtained in shown in FIG. 13 (substep SS40). At this time, interpolation processing is performed in the vertical and horizontal directions for each of the virtual pixels. This interpolation processing is performed by, for example, an LPF (Low Pass Filter) shown in FIG.

This LPF will be briefly described. Real pixels d _(-3) , d _(-1) , d ₍₁₎ , d ₍₃₎ are indicated by solid lines, virtual pixels are indicated by broken lines, and virtual pixels are arranged between four real pixels. ing. Virtual pixels d _{n (-4)} , d _{n (-2)} , d _{n (0)} , d _{n (2)} , d _{n (4)} have no data in consideration of correspondence with real pixels. It should be treated as the same relationship as the state that does not enter. That is, for example, zero is preset in these pixels (zero embedding). For example, when the pixel d _{n (0)} is interpolated in the horizontal direction as shown in FIG. 19 (a), the tap coefficients of the digital filter are k ₀ , k ₁ , k ₂ , k ₃ , k ₄ ,. _When arranged as _n , the high-frequency luminance data Y _{H (0)} is _expressed by equation (8)
Y _{H (0)} = k ₀ * d _{n (0)} + k ₁ * (d ₍₁₎ + d _(-1) ) + k ₂ * (d _{n (-2)} + d _{n (2)} ) + k ₃ * (d _(-3) + d ₍₃₎ )
+ k ₄ * (d _{n (-4)} + d _{n (4)} ) + ... k _n * (d _{n (-n)} + d _{n (n)} ) ... (8)
Is obtained as However, in this case, as apparent from FIG. 19 (a), zero data alternately enters, so the coefficient is doubled. This relationship also applies to the other pixels dn _(-4) , dn _(-2) , dn ₍₂₎ , and dn ₍₄₎ to be interpolated in FIG. 19 (a). By performing these interpolation processes, high-frequency luminance data Y _{H (-4)} , Y _{H (-2)} , Y _{H (2)} , Y _{H (4)} are obtained (see FIG. 19 (b)). ).

Then, it is determined whether the above-described virtual pixel interpolation processing for one frame is completed (sub-step SS42). When this interpolation process is incomplete (NO), the interpolation process is performed on the virtual pixels that have not been subjected to the interpolation process (return to sub-step SS40). Further, when the interpolation processing for all the virtual pixels in one frame is completed (YES), the interpolation processing is completed and the process returns to return. As a result, the high frequency luminance data Y _H of the real pixel and the virtual pixel is obtained in the relationship represented by the colors G and Mg as shown in FIG. LPF processing is also performed in the vertical direction. In this process, as shown in FIG. 21, colors G and Mg are generated for each column in each column direction.

Note that the interpolation processing of the virtual pixels is not limited to the method using the LPF processing, but may be performed in the vertical and horizontal directions in consideration of the correlation with the actual pixels around each target pixel. The generated high-frequency luminance data Y _H may be subjected to a duplication prevention process to eliminate one of the common band in the horizontal and vertical directions.

  Next, color interpolation processing will be described (see FIG. 22: subroutine SUB5). The color interpolation process is performed by the color interpolation development unit 42 shown in FIG. The color interpolation development unit 42 is supplied with pixel data 38 read from the data correction unit 30a, that is, three primary color data, to the R interpolation development unit 420, the G interpolation development unit 422, and the B interpolation development unit 424, respectively. Interpolation generation of pixel data of real pixels and virtual pixels for each color using these supplied pixel data will be described according to a flowchart (see FIG. 22). In this case, the interpolation processing of the pixel data G is first performed in substep SS500. At this time, as shown in FIG. 6, since the pattern of the honeycomb type G square lattice RB complete checkered pattern is used, the existing pixels having the existing pixel data G 1 are represented by solid line octagons. A pixel having no pixel data G, that is, a pixel having a virtual pixel and existing pixel data but having a color different from the color G is represented by a dotted octagon. Here, a pixel that does not have this pixel data G is also called a virtual pixel. In the interpolation process, four pieces of existing pixel data are used.

This relationship is specifically shown in FIG. As the pattern in Fig. 20 shows, if half of one row is real pixel data of color G, the other half of G ₁₂ , G ₁₄ , G ₁₆ , G ₁₈ , G ₁₁₀ , G ₁₁₂ is a virtual pixel, and color G is completely When rows that do not include are alternately interpolated with G ₃₂ , G ₃₄ , and G ₃₆ , the interpolation processing is performed for every four adjacent pixel data G ₁₁ , G ₁₃ , G ₃₁ , G ₃₃ and pixel data G ₁₃ , G ₁₅ , G ₃₃ , G ₃₅ etc. are used. For example, interpolated pixel data G ₂₁ virtual pixel is interpolated using the data of actual pixels corresponding to two pixels of the same column
G ₂₁ = (G ₁₁ + G ₃₁ ) / 2 (9)
Obtained from. When the pixel data G ₁₃ and G ₃₃ are applied to the calculation formula (9), the virtual pixel G ₂₃ can be interpolated. Also, the interpolation of the virtual pixel G _12, using the data of actual pixels corresponding to two pixels of the same row
G ₁₂ = (G ₁₁ + G ₁₃ ) / 2 (10)
Obtained from. When the pixel data G ₃₁ and G ₃₃ are applied to the calculation formula of Expression (10), the virtual pixel G ₃₂ can be interpolated. The pixel data G ₂₂ located at the center of each of the four pieces of pixel data G ₁₁ , G ₁₃ , G ₃₁ , G ₃₃ is obtained by using the pixel data at these four positions.
G ₂₂ = (G ₁₁ + G ₁₃ + G ₃₁ + G ₃₃ ) / 4 (11)
Obtained from. If the surrounding pixel data G ₁₃ , G ₁₅ , G ₃₃ , G ₃₅ is used in the calculation formula of Expression (11), the virtual pixel G ₂₄ can be interpolated. Since the four pixel data G ₁₃ , G ₁₅ , G ₃₃ , and G ₃₅ of the existing pixels are regarded as a set of data and interpolated, and the pixel data G ₂₃ has already been calculated, the remaining pixel data G ₁₄ , G ₃₄ , the G ₂₅ may be calculated by the calculation described above. By repeating the interpolation process using the real pixels, a plain image of the pixel data G is created. However, since the outermost edge of the plane image does not have such a relationship, it is preferable to set it as a boundary value when performing interpolation strictly. Further, considering the effective screen, the data in the peripheral portion is outside the range of the effective screen and may not be calculated.

Next, pixel data R is calculated (sub-step SS502). The actual pixel data in the pixel data R is R ₀₀ , R ₀₄ , R ₀₈ , R ₀₁₂ , R ₂₂ , R ₂₆ , R ₂₁₀ , R ₄₀ , R ₄₄ , R ₄₆ , every other line as shown in FIG. ... In this case, pixel data that is obliquely adjacent to the interpolation target virtual pixel is used. For example, the pixel data of the virtual pixel R ₁₁ is obtained by using the pixel data of the pixels R ₀₀ and R ₂₂ .
R ₁₁ = (R ₀₀ + R ₂₂ ) / 2 (12)
Is calculated by Similarly, the virtual pixels R ₃₁ and R ₃₃ are calculated by applying the pixel data R ₄₀ and R ₂₂ and the pixel data R ₄₄ and R ₂₂ to the same relationship as the equation (12), respectively. If calculation is performed in consideration of the existing pixel data R ₂₆ , virtual pixels R ₁₅ and R ₃₅ can also be created by the adjacent oblique interpolation processing.

Next, a pixel surrounded by pixels calculated in the previous process is set as a pixel to be interpolated, and interpolation processing is performed using these four pixel data calculated in the interpolation (substep SS504). For example, as can be seen from FIG. 6 with the virtual pixel R ₂₄ to be calculated as the center, using the data of the positions of the surrounding pixel data R ₁₃ , R ₁₅ , R ₃₃ , R ₃₅ ,
R ₂₄ = (R ₁₃ + R ₁₅ + R ₃₃ + R ₃₅ ) / 4 (13)
Is calculated by When a relationship equivalent to Expression (13) is obtained from surrounding pixels, pixel data R ₀₂ , R ₂₀ , and R ₄₂ are obtained by performing interpolation. In other words, when viewed from the pixel to be interpolated, all the pixel data used for interpolation are located obliquely.

Next, the pixel data obtained so far is used, and among these pixels, interpolation is performed from pixel data positioned vertically and horizontally with respect to the interpolation target pixel (substep SS506). For example, using four of the pixel data of the vertical and horizontal centering on the pixel data R ₁₂
R ₁₂ = (R ₀₂ + R ₁₁ + R ₁₃ + R ₂₂ ) / 4 (14)
Is calculated by Pixel data R ₁₄ , R ₃₂ , R ₁₄ , R _{34 and the} like having the same positional relationship can be calculated by substituting peripheral pixel data R 1 corresponding to equation (14).

In addition, since uninterpolated virtual pixels remain in the peripheral portion, for example, three pixels surrounding the virtual pixel may be interpolated. Also in the case of this interpolation, if the above-described interpolation method is used, the pixel data R ₀₁ of the virtual pixel is
R ₀₁ = (R ₀₀ + R ₀₂ + R ₁₁ ) / 3 (15)
Is calculated by In this manner, pixel data R ₀₃ , R ₀₅ , R ₁₀ , R ₃₀ , R ₄₁ , R ₄₃ , R ₄₅ are also interpolated. Finally, the entire plane screen related to the pixel data R is interpolated.

  Next, interpolation processing is performed on pixel data B (sub-steps SS508, SS510, SS512. These processings are adjacent diagonal interpolation processing on pixel data B, central interpolation processing using four interpolation data, and upper, lower, left, and right, respectively. The central interpolation processing is performed by the four pixels of the pixel data R. The interpolation processing is based on the above-described interpolation processing of the pixel data R (that is, sub-steps SS502, SS504, SS506). It is obvious from the relationship of the pixel arrangement of B. That is, the pixel arrangement of the pixel data B is shifted from the matrix display represented by the subscript of each color to the entire pixel data R by two columns in the horizontal (ie, row) direction. For this reason, when virtual pixels are interpolated by applying the above-described calculation formulas, the pixel data is added on the right side of the matrix display where the column number is 2 or more. It is better to calculate by adding only +2 to the number in the column of 1. If interpolation processing is performed with attention to this relationship, plane interpolation development for pixel data B can be performed. Proceed to determine whether or not to end.

  It is determined whether or not the plane interpolation development has been completed for each color (substep SS514). When the series of processes has not been completed yet (NO), the process returns to sub-step SS500 and is repeated. Note that this confirmation processing may be performed for each color. When a series of processing is completed (YES), the process proceeds to return. After this transition, the processing of subroutine SUB5 is terminated.

Next, the broadband processing will be described (see FIG. 23). The supplied high-frequency luminance data Y _H (44) is subjected to a high-pass filter (HPF) process (substep SS200). That is, the high frequency luminance data Y _H (44) is subjected to processing for passing a frequency higher than a predetermined high frequency band. By this processing, the high frequency component of the high frequency luminance data is extracted as a signal. The high frequency component and Y _h. High-frequency component Y _h is outputted to the adder 54 of FIG. Plain pixel data R (46), G (48), and B (50) are supplied to one end sides 540a, 542a, and 544a of the adders 540, 542, and 544, respectively, and are supplied to the other end sides 540b, 542b, and 544b. Is supplied with a high-frequency component signal Y _h corresponding to the supplied pixel data.

The addition unit 54, the high-frequency component Y _h and the pixel data R, G, is added to the B for each pixel is subjected to be supplied (substep SS202). In general, the position (i, j) of each pixel is displayed in a matrix, and each pixel data of the three primary colors RGB has a wide band. This broadening is represented by the subscript H. At this time, the broadband pixel data R _Hij , G _Hij , B _Hij are
R _Hij = R _ij + Y _hij (16a)
G _Hij = G _ij + Y _hij (16b)
B _Hij = B _ij + Y _hij (16c)
It becomes. The pixel data R _Hij , G _Hij , and B _Hij calculated here are supplied to the color difference matrix unit 56.

Next, using the supplied pixel data R _Hij , G _Hij , and B _Hij , the luminance data Y and the color difference data C _r are _widened by _performing color difference matrix calculation processing on the position of each real pixel and virtual pixel. , C _b (substep SS204).

Next, the obtained luminance data Y and color difference data C _r and C _b are subjected to distortion prevention processing (substep SS206). The frequency data of the luminance data Y and the color difference data C _r and C _b extended with a wide band are passed, and low-pass filter processing is performed so that aliasing distortion does not occur in these three data. Further, as the distortion prevention process, the overlapping frequency bands in one direction are limited with respect to the overlapping area of the frequency bands in the horizontal and vertical directions. This avoids image quality degradation caused by frequency overlap.

Next, aperture adjustment is performed on the luminance data Y that has been subjected to distortion prevention processing (substep SS208). Aperture adjustment is a process corresponding to the contour enhancement process. Then, gain adjustment is performed on the color difference data C _r and C _b subjected to the distortion prevention processing (sub-step SS210). It is determined whether or not processing for one frame has been completed in this way (sub-step SS212). If the processing for one frame has not been completed (NO), the process returns to the beginning (substep SS200) and the series of processing is repeated. If processing for one frame has been completed (YES), the process proceeds to return to end this subroutine SUB2, and proceeds to the main routine via return.

  Since it can be operated in this way and the accuracy of correlation processing can be improved, for example, even if a subject including a situation in which a yellow thin line is sandwiched between white is imaged, a step appears in the yellow thin line portion of the captured image, It is possible to avoid a reduction in image quality, and a high-quality image can be obtained.

  The situation where a thin line having such a step is obtained is not limited to the case where the above-described object scene is imaged. For example, a step is generated even when the fine line interval in a dark region sandwiched between bright regions is large or the fine line interval is small. FIG. 24 shows the flags obtained by the processing up to sub-step SS816 of the subroutine SUB8 in the former case among the steps generated in the thin line in the dark area.

In FIG. 24, pixel data having a high signal level is represented by capital letters, and pixel data having a low signal level is represented by small letters. The three primary colors RGB in the bright area are large, and the pixel data of the pixels R and B sandwiched between the colors G are calculated from the average of the surrounding four pixels. The pixel data of the pixels R and B at the border between light and dark indicates that there is a horizontal correlation. Further, the pixels r and b in the dark region between the colors g are calculated from the average of four pixels. These high-frequency luminance data Y _H (r +) and Y _H (b +)
Y _H (r +) = r / 2 + (B + b + B + b) / 8 = r / 2 + (B + b) / 4 (17a)
Y _H (b +) = b / 2 + (R + r + R + r) / 8 = b / 2 + (R + r) / 4 (17b)
Is obtained. The level difference of luminance data in the longitudinal direction is the difference in level between high frequency luminance data Y _H (R +) and Y _H (b +) as ΔY _H.
ΔY _H = Y _H (r +) − Y _H (b +) = (r−R + B−b) / 4 (18)
Is obtained.

Based on this current situation, if the processing of substeps SS818 to SS824 is performed, it is determined that the flag (+) indicating that the average of the surrounding four pixels sandwiched between the color G and the color g is correlated in the horizontal direction As a result, the flag (h) is obtained (see the bold frame in FIG. 25). When the high-frequency luminance data Y _H is calculated using the pixel data based on the flag (h), the level difference of Expression (18) is zero. Since the relationship of the flags to the pixels R, B, r, and b is also the same, the flag pattern comparison correction need not be performed in this case.

When the latter fine line interval is narrow, the pixels r and b in the dark region sandwiched between the bright regions are determined to be the average of four pixels as indicated by the flag (+) (see FIG. 26). At this time, the high-frequency luminance data Y _H (r +) and Y _H (b +) are obtained by equations (17a) and (17b), respectively. Therefore, the level difference ΔY _H is expressed by equation (18). The above-described sub-steps SS818 to 824 are performed so as to eliminate this level difference. However, in this case, no improvement is made in the same situation as in FIG. 26 even if the flags of the pixels r and b in the dark region are processed to improve the correlation accuracy (see FIG. 27).

Based on this situation, the post-adaptive processing flag pattern is compared and corrected (substep SS36). Comparing the pixel data B and R located on the upper and lower lines with respect to the target pixel, the pixel data B and R located on the upper and lower lines are flag (h), so the target pixel Correct the flag in the horizontal direction (h). When the high-frequency luminance data Y _H is generated in correspondence with this correction, no level difference is generated in the high-frequency luminance data Y _H at the pixel position surrounded by the thick line frame (see FIG. 28).

By generating the high-frequency luminance data Y _H by eliminating the level difference that has occurred in the subject in which such a bright and dark region has a specific width and is in the longitudinal direction, the output image can be improved in image quality.

By the way, in the luminance interpolation processing described above, the mixed colors G and Mg are subjected to LPF interpolation in the horizontal and vertical directions to generate high-frequency luminance data Y _H on the entire surface. However, interpolation is performed by paying attention to each of the colors G and Mg. May be. When attention is paid to the color G 1, the pattern shown in FIG. 29 is obtained, and when attention is paid to the color Mg, the pattern shown in FIG. 30 is obtained. When the respective patterns are interpolated, the plain pixel data G and Mg shown in FIGS. 31 and 32 are obtained. Furthermore, the luminance interpolation example, may be plain pixel data G, the data obtained by addition processing for each pixel corresponding to Mg used as the high-frequency luminance data Y _H.

  By configuring as described above, the luminance level in the longitudinal direction can be obtained by performing correlation detection of a pixel of the same color as the target pixel over a wide area, for example, for a chromatic / achromatic long object in the longitudinal direction. The correlation accuracy is improved by preventing the difference from occurring and taking into account the pattern of the flag indicating the correlation, thereby improving the correlation accuracy. Therefore, the false signal or straight line that intersects the wedge shape of the resolution chart and the cross Occurrence of phenomena such as cutting can be suppressed. As a result, the phenomenon that occurs when such a subject is photographed can be significantly improved as compared to the conventional case.

10 Digital camera
12 Optical lens system
14 Operation unit
16 System bus
18 System controller
26 Image sensor
30 Signal processor
30a Data correction unit
30b Interpolation processor
30c Broadband processing section
32 Compression / decompression processor

Claims (14)

A plurality of image sensors that photoelectrically convert the incident light of the three primary colors is separated in half in the row and / or column direction with respect to the geometric center of the adjacent image sensor. Disposed at a pitch, the image sensor is a real pixel, each image signal obtained from the image sensor is digital pixel data, and a new pixel is generated as a virtual pixel between the pixels using the pixel data In the image signal processing method for performing the interpolation processing to generate luminance and color difference data, the method includes:
When the pixel data is displayed two-dimensionally, the green color of the pixel data is obtained in a square lattice pattern, and the red pixel data from the actual pixels at one diagonal position sandwiching the green color and the other diagonal position A first step of obtaining blue pixel data from real pixels of
When one of the red and blue real pixels is set as a target pixel, the vertex of the correlation determination area for the target pixel is expanded to at least a region represented by a pixel of the same color as the target pixel. While determining the correlation accuracy to improve the correlation accuracy, the pixel data of the complementary color of green is used as luminance data by using the other plurality of pixel data around the target pixel located in the correlation direction and the other color. A second step of generating;
Using the complementary color pixel data and the green pixel data, luminance data in the virtual pixel is generated by interpolation from pixel data of the same color located in the horizontal and / or vertical direction, and the obtained real pixel and virtual pixel are obtained. A third step of interpolating and generating all the red, blue and green colors different from the color attributes obtained for all the real pixels and virtual pixels based on the color attributes;
A high-frequency component signal obtained by passing a frequency higher than a predetermined high-frequency band included in the luminance data generated by the interpolation is added to the pixel data of the three primary colors of each pixel, and the band processing of the pixel data of each color is performed. The image signal processing method characterized by including these processes.   2. The method according to claim 1, wherein the second step includes a fifth step of performing horizontal / vertical correlation determination using pixel data of the same color as the target pixel and positioned in the horizontal and vertical directions. An image signal processing method. 3. The method according to claim 2, wherein, before performing the correlation determination, the fifth step includes pixel data having the same color as the target pixel located in a left horizontal direction of the target pixel data and the target pixel data. Is the first horizontal difference absolute value, and the difference absolute value between the target pixel data and the pixel data of the same color as the target pixel located in the right horizontal direction of the target pixel data is the second horizontal difference The difference absolute value is set, and the added value of the first and second horizontal difference absolute values is set as horizontal comparison data. The pixel data of the same color as the target pixel data located in the upper vertical direction of the target pixel data and the target The difference absolute value from the pixel data is defined as a first vertical difference absolute value, and the difference absolute value between the target pixel data and the pixel data of the same color as the target pixel located in the lower vertical direction of the target pixel data is 2 vertical difference When the absolute value is obtained, the addition value of the first and second vertical difference absolute values is previously calculated as vertical comparison data, and the result obtained by subtracting the vertical comparison data from the horizontal comparison data is greater than or equal to a predetermined value. A first determination step of determining that there is a vertical correlation;
A second determination step of determining that there is a horizontal correlation when the result of subtracting the horizontal comparison data from the vertical comparison data is greater than or equal to a predetermined value;
And a third determination step of determining that there is no correlation when different from both of the first and second determinations. The method according to claim 3, wherein when there is a correlation in the correlation determination as a result of the first or second determination step, the method is different in color from the target pixel data across the target pixel data. And a sixth step of calculating a sum obtained by multiplying an average of a pair of pixel data located in the vicinity of the correlation direction and pixel data of the target pixel by a half weighting factor,
In the case where there is no correlation in the correlation determination, a seventh step of taking the sum obtained by multiplying the target pixel data and the addition average of pixel data of different colors located around the target pixel data by a half weight coefficient, respectively And an image signal processing method. The method according to claim 3, wherein the second step is to perform correlation determination on the real pixel, and then, for the target pixel, one of the surrounding red and blue pixel data different from the target pixel. Among them, an eighth step of comparing using a flag whether the upper and lower pixel data or the left and right pixel data sandwiching the target pixel is correlated in which direction;
As a result of expanding the area and determining the correlation, if all the compared pixel data are correlated in the same direction, the correlation of the target pixel is determined to be correlated in the same direction as the surrounding pixels, and corrected. According to the corrected result, the target pixel data and the target pixel data are different in color from the target pixel data, and an addition average of a pair of pixel data located in the vicinity of the correlation direction and the target pixel A ninth step of generating luminance data by taking the sum of each pixel data multiplied by a half weighting factor;
As a result of correlation determination by expanding the region, if even one of the determined pixel data has a correlation in a different direction, the correlation assigned to the target pixel is used as it is for luminance data generation. An image signal processing method comprising: a tenth step of generating data.   6. The method according to claim 5, wherein, in the correlation of the peripheral pixels with respect to the target pixel, the ninth step corrects the target flag for comparison correction obtained in each of the areas to be subjected to correlation determination to the flag of the peripheral pixel. Then, it is determined that there is a correlation in the same direction as the surrounding correlation direction, and the target pixel data and the target pixel data are sandwiched between the target pixel data according to the determination result, and the correlation is different from the target pixel data. An image signal processing method, wherein luminance data is generated by taking a sum obtained by multiplying an average of a pair of pixel data located in the vicinity of a direction and pixel data of the target pixel by a half weighting factor.   The method according to any one of claims 1 to 6, wherein the third step performs low-pass filter processing in the vertical direction and / or the horizontal direction after predetermined data is inserted into the virtual pixel in advance. An image signal processing method. A plurality of image sensors that photoelectrically convert the incident light of the three primary colors is separated in half in the row and / or column direction with respect to the geometric center of the adjacent image sensor. Disposed at a pitch, the image sensor is a real pixel, each image signal obtained from the image sensor is digital pixel data, and a new pixel is generated as a virtual pixel between the pixels using the pixel data In the image signal processing method for performing the interpolation processing to generate luminance and color difference data, the method includes:
When the pixel data is displayed two-dimensionally, the green color of the pixel data is obtained in a square lattice pattern, and the red pixel data from the actual pixels at one diagonal position sandwiching the green color and the other diagonal position A first step of obtaining blue pixel data from real pixels of
A second step of creating a green screen as luminance data while interpolating the virtual pixel using the read green pixel data;
Among the read pixel data, when any one of the red and blue real pixels is set as a target pixel, the vertex of the correlation determination area for the target pixel is at least a pixel of the same color as the target pixel The target pixel located in the correlation direction and the other plurality of pixel data around different colors are used to determine the correlation color of the green color. A third step of generating pixel data, interpolating the virtual pixel using the complementary color pixel data, creating the complementary color screen, and generating luminance data;
The obtained green screen and the complementary color screen are added to generate luminance data for all the real and virtual pixels, and the real pixels and the virtual pixels are obtained based on the color attributes of the real pixels and the virtual pixels, respectively. A fourth step of interpolating and generating all the red, blue and green colors that differ from the color attributes obtained for all virtual pixels;
A fifth step of widening the pixel data of each color by adding the high-frequency component signal obtained by passing a frequency higher than the predetermined high-frequency band of the luminance data interpolated to each pixel data of the three primary colors of each pixel. And an image signal processing method.   9. The method according to claim 8, wherein the third step includes a sixth step of performing horizontal / vertical correlation determination using pixel data of the same color as the target pixel and positioned in the horizontal and vertical directions. An image signal processing method. 10. The method according to claim 9, wherein, before performing the correlation determination, the sixth step includes pixel data having the same color as the target pixel located in a left horizontal direction of the target pixel data and the target pixel data. Is the first horizontal difference absolute value, and the difference absolute value between the target pixel data and the pixel data of the same color as the target pixel located in the right horizontal direction of the target pixel data is the second horizontal difference The difference absolute value is set, and the added value of the first and second horizontal difference absolute values is set as horizontal comparison data. The pixel data of the same color as the target pixel data located in the upper vertical direction of the target pixel data and the target The difference absolute value from the pixel data is defined as a first vertical difference absolute value, and the difference absolute value between the target pixel data and the pixel data of the same color as the target pixel located in the lower vertical direction of the target pixel data is 2 vertical difference When the absolute value is obtained, the addition value of the first and second vertical difference absolute values is previously calculated as vertical comparison data, and the result obtained by subtracting the vertical comparison data from the horizontal comparison data is greater than or equal to a predetermined value. A first determination step of determining that there is a vertical correlation;
A second determination step of determining that there is a horizontal correlation when the result of subtracting the horizontal comparison data from the vertical comparison data is greater than or equal to a predetermined value;
And a third determination step of determining that there is no correlation when different from both of the first and second determinations. 11. The method according to claim 10, wherein if there is a correlation in the correlation determination as a result of the first or second determination step, the target pixel data is different in color from the target pixel data and the correlation direction A seventh step of calculating a sum obtained by multiplying an average of a pair of pixel data located in the vicinity of the pixel data and pixel data of the target pixel by a half weighting factor,
In the case of non-correlation in the correlation determination, an eighth step of taking the sum obtained by multiplying the target pixel data and the average of pixel data of different colors located in the vicinity of the target pixel data by a half weight coefficient, respectively. And an image signal processing method. 11. The method according to claim 10, wherein after performing correlation determination on the real pixel, the third step is to calculate one of the surrounding red and blue pixel data different from the target pixel with respect to the target pixel. Among them, a ninth step of comparing using a flag whether the upper and lower pixel data or the left and right pixel data sandwiching the target pixel is correlated in which direction;
As a result of expanding the area and determining the correlation, if all the compared pixel data are correlated in the same direction, the correlation of the target pixel is determined to be correlated in the same direction as the surrounding pixels, and corrected. According to the corrected result, the target pixel data and the target pixel data are different in color from the target pixel data, and an addition average of a pair of pixel data located in the vicinity of the correlation direction and the target pixel A tenth step of generating luminance data by taking the sum of each pixel data multiplied by a half weighting factor;
As a result of correlation determination by expanding the region, if even one of the determined pixel data has a correlation in a different direction, the correlation assigned to the target pixel is used as it is for luminance data generation. An image signal processing method comprising: an eleventh step of generating data.   13. The method according to claim 12, wherein in the correlation of the peripheral pixels with the target pixel, the eleventh step corrects the target flag of the comparison correction obtained in each of the correlation determination regions into the peripheral pixel flag. Then, it is determined that there is a correlation in the same direction as the surrounding correlation direction, and the target pixel data and the target pixel data are sandwiched between the target pixel data according to the determination result, and the correlation is different from the target pixel data. An image signal processing method, wherein luminance data is generated by taking a sum obtained by multiplying an average of a pair of pixel data located in the vicinity of a direction and pixel data of the target pixel by a half weighting factor. 9. The method according to claim 8, wherein the virtual pixel interpolation processing performed in the second and third steps is performed in a vertical direction and / or a horizontal direction after predetermined data is previously inserted into the virtual pixel in each step. An image signal processing method, wherein the green and the complementary color of green are generated in accordance with a low-pass filter process or a correlation determination of pixel data located around the virtual pixel.

Images