JP3577354B2 - Interpolated image data generation apparatus and method - Google Patents

Description

[0001]
【Technical field】
BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for creating image data of a frame missing due to frame skipping by interpolation processing based on image data of preceding and succeeding frames.
[0002]
[Prior art and its problems]
A moving image is represented by images of a number of continuous frames. Image data compression processing is performed to efficiently transmit or store image data representing a moving image. As part of this compression processing (compression processing in a broad sense), so-called frame dropping is sometimes performed, in which a suitable number of frames are periodically thinned out (separate frames every other frame) from consecutive frames. The image data of the frame remaining after the frame is dropped is encoded (data compression) and transmitted or stored.
[0003]
FIG. 13 shows a plurality of continuous (here, seven) frames. From these consecutive frames, the second, fourth and sixth frames are dropped. The remaining first, third, fifth and seventh frames are encoded.
[0004]
When reproducing a moving image represented by a plurality of continuous frames, the first frame, the third frame, the fifth frame, and the seventh frame are decoded (data decompression). The frame lost due to frame drop is substituted by the immediately preceding decoded frame. That is, first, the image data of the first frame is output as the reproduction output, and then the image data of the first frame is output again instead of the second frame. Similarly, the image data of the third frame is output twice consecutively, the image data of the fifth frame is output twice consecutively, and finally the image data of the seventh frame is output.
[0005]
Although the frame drop has been described for the frame image, the field image may be dropped in some cases.
[0006]
Thus, upon reproduction of a moving image, a field or frame that is lost due to frame drop is replaced by a field or frame that has not been dropped immediately before and the continuity of the moving image is lost. In particular, in the case of a large number of dropped frames or a moving image having a large movement, there is a problem that the moving image is awkward and unnatural.
[0007]
In order to solve this problem, it is necessary to generate the image data of the frame image lost due to frame dropping by simple interpolation using the image data of the frame images before and after the frame image when reproducing the moving image. It has been proposed (see, for example, JP-A-6-237444). In the example shown in FIG. 13, the image data of each pixel of the second frame is generated by simple interpolation using the image data of the corresponding pixels of the first and third frames. Similarly, the image data of each pixel of the fourth frame is generated by simple interpolation using the image data of the corresponding pixels of the third and fifth frames. The image data of the sixth frame is generated by simple interpolation between the corresponding pixels of the fifth and seventh frames.
[0008]
However, since this method is implemented by simple interpolation between corresponding pixels using the image data of the previous and subsequent frame images that have not been dropped, it does not consider the motion of the moving image. Therefore, it did not make the movement sufficiently smooth.
[0009]
DISCLOSURE OF THE INVENTION
SUMMARY OF THE INVENTION It is an object of the present invention to generate image data capable of representing a smooth motion by performing an interpolation process in consideration of a motion.
[0010]
In the interpolation image data generating apparatus according to the present invention, the first image, the second image, and the third image, each constituting one frame, are continuous in this order, and the second image is missing. When the first image data representing the first image and the third image data representing the third image are given, the second image data representing the second image between them is given by: An apparatus for generating using the first image data and the third image data, wherein the window on the second image is set in windows respectively set at corresponding positions on the first image and the third image. Unit area image data for extracting image data of a unit area located at a point symmetric position with respect to a center unit area at a position corresponding to the center of the window while changing the positions of the unit areas on the first image and the third image Extraction means, the unit area image data Correlation value calculating means for calculating a correlation value between the unit area image data of the first image and the unit area image data of the third image extracted by the means, and the correlation value calculated by the correlation value calculating means. An interpolating means for interpolating the image data of the central unit area of the second image based on a pair of unit area image data that has generated the correlation value representing the strongest correlation is provided.
[0011]
In the interpolation image data generation method according to the present invention, the first image, the second image, and the third image, each of which forms one frame, are continuous in this order, and the second image is missing. When the first image data representing the first image and the third image data representing the third image are given, the second image data representing the second image between them is given by: This is a method of generating using the first image data and the third image data. In the window set at the corresponding positions on the first image and the third image, respectively, The image data of the unit area located at a point symmetric position with respect to the center unit area at the position corresponding to the center of the window is extracted while changing the position of the unit area on the first image and the third image, and is extracted. Unit area image data of the first image and the third image The center unit of the second image is calculated based on a pair of unit area image data that has calculated a correlation value with the unit area image data of each pair and generated a correlation value representing the strongest correlation among the calculated correlation values. This is for performing an interpolation operation on the image data of the area.
[0012]
While sequentially shifting the position of the window in the horizontal and vertical directions by the horizontal and vertical sizes of the unit area on the first image and the third image, respectively, All the second image data representing the second image can be obtained by repeating the extraction of the area image data, the above-mentioned correlation operation and the above-mentioned interpolation operation.
[0013]
There are various modes for taking out the image data of the unit area.
[0014]
One is to set the unit area to the size of one pixel. In this case, the size of the window is odd pixels (excluding one pixel) in the horizontal and vertical directions. The unit area from which the image data is extracted will be shifted by one pixel (or by a plurality of pixels).
[0015]
Second, the unit area is set as a block including a plurality of pixels. Since the calculation of the correlation value is performed for each block, it is strong against noise.
[0016]
In this case, the position of the unit area (block) may be changed for each block, or may be changed for one pixel or a plurality of pixels.
[0017]
According to the present invention, the unit area in the window on the first image and the third image located at a point symmetrical position with respect to the center unit area at the position corresponding to the center of the window on the second image. Extracting image data. The unit area image data of the first image and the third image is extracted while changing the position of the unit area, and the correlation value between them is calculated. It can be said that a pair of unit areas that have generated the correlation value representing the strongest correlation represent substantially the same image. The direction connecting the pair of unit areas in the first image and the third image that has generated the correlation value representing the strongest correlation is the direction of image movement, and the distance between these unit areas (distance in space and time) is This indicates the degree of movement.
[0018]
As described above, the image data representing the second image is calculated by the interpolation operation based on the image data of the pair of unit areas of the first image and the third image most strongly related to the movement of the image. The image data of the second image represents an image connecting the first image and the third image at an intermediate position between the first image and the third image.
[0019]
Therefore, when the first image, the second image, and the third image are continuously reproduced, if there is a motion of the image in these images, the motion is smoothly expressed. .
[0020]
In this way, according to the present invention, the interpolation processing is performed in consideration of the movement of the image in the image, so that image data representing a smooth movement when reproduced is obtained.
[0021]
[Explanation of the embodiment]
FIG. 1 shows a first embodiment of the present invention, and is a block diagram showing an electrical configuration of an apparatus for generating image data which is missing due to frame drop by interpolation. This embodiment performs inter-pixel interpolation.
[0022]
As shown in FIG. 13, a case will be described in which frame image data of an even-numbered frame is lost due to frame drop and frame image data of an odd-numbered frame is provided. Based on the preceding and following odd-numbered frame image data, image data of the even-numbered frame image between them is generated.
[0023]
One frame of image data is input to one of the frame memories 11 and 12 via the changeover switch 10 and stored. The changeover switch 10 is switched every time one frame of image data passes. Therefore, the input image data for one frame is alternately applied to the frame memory 11 or 12.
[0024]
For example, if the first frame image data is provided to the frame memory 11, the next third frame image data is provided to the frame memory 12. With the first and third frame image data stored in the frame memories 11 and 12, respectively, the image data of the second frame is generated using these image data.
[0025]
Thereafter, the fifth frame image data is provided to the frame memory 11. In this state, the image data of the fourth frame is generated using the image data of the third and fifth frames stored in the frame memories 12 and 11.
[0026]
Thereafter, the seventh frame image data is provided to the frame memory 12 and stored. The image data of the sixth frame is generated using the image data of the fifth and seventh frames stored in the frame memories 11 and 12.
[0027]
When the generation of image data by interpolation is performed in real time for displaying a moving image, image data for one frame is generated during one frame period (1/30 second in the case of the NTSC system).
[0028]
For example, with the first and third frame image data stored in the frame memories 11 and 12, the second frame image data is generated using the first and third frame image data during one frame period. Is done. The fifth frame image data is read into the frame memory 11 during the subsequent one frame period. Further, during the subsequent one frame period, fourth frame image data is generated using the third and fifth frame image data. In this way, the reading of the frame image data into the frame memory 11 or 12 and the generation of the frame image data by interpolation are alternately repeated in the frame cycle. In this case, the changeover switch 10 is alternately switched every two frames.
[0029]
Although a contact switch is shown as the changeover switch 10, it is of course actually realized as a semiconductor contactless switch (gate circuit or the like). By switching the designated addresses in the frame memories 11 and 12, a function equivalent to switching by the changeover switch 10 can be achieved (when the frame memories 11 and 12 have different address spaces).
[0030]
First and third frame image data are stored in frame memories 11 and 12, respectively, and a process of generating image data of a second frame by interpolation using the first and third frame image data. Will be described with reference to FIGS.
[0031]
FIGS. 2 and 3 show a pixel group of 3 × 3 pixels extracted from the first frame image data and the third frame image data through a window, respectively. Also shown between these pixel groups are the corresponding pixels of the image data of the second frame to be generated by interpolation. FIG. 2 is a perspective view for clarifying the symmetric relationship between pixels, and each pixel is represented by a dot (black circle). FIG. 3 is drawn in a plan view to make the pixel arrangement easy to understand. Image data is generated for the pixels of the second frame corresponding to the pixels at the center of the window.
[0032]
First, the correlation value between the pixel of the first frame and the pixel of the third frame, which are in a point-symmetric positional relationship with each other with respect to the pixel (2, 5) of the second frame for which image data is to be generated by interpolation, calculate. That is, the correlation value s between the pixel (1, 1) of the first frame and the pixel (3, 9) of the third frame ₁ , The correlation value s between pixel (1, 2) and pixel (3, 8) ₂ , The correlation value s between pixel (1, 3) and pixel (3, 7) ₃ , The correlation value s between pixel (1, 4) and pixel (3, 6) ₄ , The correlation value s between pixel (1,5) and pixel (3,5) ₅ , The correlation value s between pixel (1, 6) and pixel (3, 4) ₆ , The correlation value s between pixel (1,7) and pixel (3,3) ₇ , The correlation value s between pixel (1, 8) and pixel (3, 2) ₈ , And the correlation value s between pixel (1, 9) and pixel (3, 1) ₉ Is calculated. It is also assumed that the pixel (1,5) and the pixel (3,5) corresponding to the pixel (2,5) for which image data is to be generated have a point-symmetric relationship with respect to the pixel (2,5).
[0033]
As the correlation value, for example, the absolute value of the difference between the image data of the two pixels, the value of the square of the difference, and the like are used.
[0034]
Next, these correlation values s _i Find the smallest value (the one with the strongest correlation) among (i = 1 to 9).
[0035]
An interpolation operation using the image data of two pixels (in the first frame and the third frame) that produces the minimum correlation value is performed, and the result of the interpolation operation is the image data of the pixel (2, 5) in the second frame. It becomes. The interpolation operation may be a known operation, for example, a method of calculating an average value of image data of two pixels is used.
[0036]
It is assumed that the object represented by the image data has moved in a certain direction, and has moved by two pixels during the time from the first frame to the third frame (two frame periods). For example, assuming that this movement is in an oblique direction from the lower left to the upper right, the image represented by the pixel (1, 7) in the first frame reaches the position of the pixel (3, 3) in the third frame. Since the image data of the pixel (1,7) and the image data of the pixel (3,3) show almost the same value, the correlation value s ₇ Is zero or a value close to zero.
[0037]
The minimum value of the correlation value between the pixel of the first frame and the pixel of the third frame, which are point-symmetrical to each other with respect to the pixel for which the image data of the second frame is to be generated, is found. Finding a pair) means detecting the direction of motion of the image represented by the image data as described above.
[0038]
An interpolation operation is performed along the direction of the detected motion, and image data of the pixel (2, 5) of the second frame is calculated. Therefore, the calculated image data of the pixel (2, 5) is an image which has moved to the position of the pixel (2, 5) at a time point exactly between the time point of the first frame and the time point of the third frame. It will represent.
[0039]
Therefore, if the object images represented by the image data from the first frame to the third frame, including the image data of the second frame generated by the above-described processing, are continuously reproduced at a frame cycle, Will be expressed smoothly.
[0040]
The size of the window from which the image data should be cut out for the interpolation processing may be determined according to the degree of movement of the object. The size of the window is not limited to 3 pixels × 3 pixels, but can be arbitrarily determined as 5 pixels × 5 pixels, 7 pixels × 7 pixels, 9 pixels × 9 pixels. The larger the window, the larger the movement. Of course, it can handle small movements.
[0041]
Windows are not limited to squares. If the movement in the horizontal direction is large, the window is a rectangle long in the horizontal direction such as 3 pixels (vertical direction) x 5 pixels (horizontal direction), 3 pixels x 7 pixels, 5 pixels x 7 pixels, and 5 pixels x 9 pixels. Should be set to. When the vertical movement is regarded as important, a vertically long window such as 5 pixels × 3 pixels, 7 pixels × 3 pixels, 7 pixels × 5 pixels, and 9 pixels × 5 pixels may be set.
[0042]
The influence of noise is likely to appear significantly in the image data of each pixel, and the correlation value is also likely to vary. Therefore, it is preferable to add a determination of the validity of the found minimum correlation value. For example, a predetermined threshold value is set, and if the minimum correlation value is smaller than this threshold value, the correlation value is regarded as valid. As the threshold value, an average value of the image data of the pixel (1, 5) at the center of the window and the image data of the pixel (3, 5) can be adopted. The threshold value can also be used when the above-described correlation value calculation and detection of the minimum correlation value are performed by software. For example, the correlation value of a pixel pair is calculated in a certain order, and the calculated value is compared with a threshold value each time the correlation value is calculated. When a correlation value smaller than the threshold value is found, the process is terminated. The found correlation value smaller than the threshold value is regarded as the minimum.
[0043]
Many types of images have no or little motion. It is not always necessary to always perform the above-described processing for such an image. Therefore, the degree of movement is first calculated, and the above processing is performed only when the degree of movement exceeds a predetermined threshold. In other cases, the image data of the pixel (2, 5) is windowed. May be calculated based on the image data of the pixel (1,5) and the pixel (3,5) at the center of the image. The degree of the movement may be represented by, for example, a correlation value between the image data of the pixel (1, 5) at the center of the window and the image data of the pixel (3, 5).
[0044]
Returning to FIG. 1, the window circuits 13 and 14 are for setting the above-described window in the image data stored in the frame memories 11 and 12, respectively. The image data of the pixels included in each window set by the circuits 13 and 14 are supplied to a correlation operation circuit 15, where a correlation value between pixels located at point-symmetric positions is calculated. In the interpolation reference pixel selection circuit 16, the minimum value of the calculated correlation values is detected, and a pair of pixels that generate this minimum value is selected. The image data of the selected pair of pixels is output from the window circuits 13 and 14. Based on these image data, the image data of the pixel to be interpolated is calculated and output by the interpolation pixel value calculation circuit 17.
[0045]
The windows set by the window circuits 13 and 14 are scanned in the horizontal and vertical directions one pixel at a time. Interpolated image data is calculated at each scanning position. Therefore, image data is generated for all pixels of the second frame (interpolation based on sufficient motion detection is not necessarily performed for peripheral pixels).
[0046]
If the apparatus shown in FIG. 1 is realized by hardware, the window circuits 13 and 14 may be composed of, for example, a plurality (for example, three) of cascade-connected line memories. The correlation operation circuit 15 may be constituted by a plurality of sets (for example, 9 sets) of a subtraction circuit and an absolute value circuit or a subtraction circuit and a squaring circuit. Of the cells (for example, 9 × 2 = 18) in the windows of the plurality of line memories of the window circuits 13 and 14, a pair of cells having a point-symmetrical positional relationship described above is connected to one subtraction circuit. The interpolation reference pixel selection circuit 16 includes a MIN circuit and a plurality (for example, nine) of comparison circuits. The MIN circuit detects a minimum value of the outputs of the plurality of absolute value circuits or the square circuits. A threshold value obtained by adding a minute value to the minimum value is given to the plurality of comparison circuits. The output of the absolute value circuit or the output of the squaring circuit is given to each comparison circuit. An output is generated from a comparison circuit to which an input smaller than the threshold is applied. The pixel pair corresponding to the comparison circuit that generated this output is the one that generated the minimum interpolation value. The interpolation pixel value calculation circuit 17 includes a gate circuit connected to the output side of the cells (9 × 2 = 18) in the window in the window circuits 13 and 14. The gate circuits are controlled by a pair of corresponding comparison circuits. The image data output through the two gate circuits controlled by the comparison circuit that generates the output is the image data of the pixel pair that has generated the minimum value. The arithmetic circuit 17 further includes an average value circuit, and the image data output from the gate circuit is given to the average value circuit. The output of the average circuit becomes the interpolated image data.
[0047]
Image data is sequentially read from the frame memories 11 and 12 in the order of horizontal scanning, and applied to window circuits 13 and 14, respectively. Each of the circuits 13, 14, 15, 16, and 17 operates in synchronization with the pixel clock, and interpolated image data is sequentially output for each pixel clock.
[0048]
Some or all of the functions of the apparatus in FIG. 1 can be realized by software. In this case, one or more of the above-described window setting, calculation of the inter-pixel correlation value, detection of the minimum correlation value, and calculation of the interpolation value using the image data of the pixel that generated the minimum correlation value are performed. , Or all are realized by a programmed computer.
[0049]
FIG. 4 shows a second embodiment. In this embodiment, processing between blocks is performed.
[0050]
The image data of each pixel is easily affected by noise as described above. In the second embodiment, in order to minimize the influence of noise or the like, motion is detected in units of blocks.
[0051]
As in the first embodiment, the odd-numbered frame image data is alternately applied to the frame memories 11 and 12 via the changeover switch 10 and stored. Windows for extracting frame image data stored in the frame memories 11 and 12 are set in the window circuits 13A and 14A, respectively.
[0052]
FIG. 5 shows pixels of the first frame image cut out by the window set in the window circuit 13A and pixels of the second frame image cut out by the window set in the window circuit 14A. This corresponds to FIG. 3 of the embodiment. FIG. 5 also shows pixels of the second frame in which image data is generated by interpolation using the first frame image data and the third frame image data.
[0053]
One block has a size of 8 pixels × 8 pixels in this embodiment. The window is set to a size including 3 × 3 blocks of pixels. [1, 1], [1, 2],..., [1, 9] are assigned to the blocks of the first frame, and [3, 1], [3, 2], , [3, 9]. A window of a second frame corresponding to each window of the first frame and the third frame is imagined, and the center block of the window of the second frame is [2, 5]. Image data of pixels in the block [2, 5] is generated by interpolation.
[0054]
The size of one block can be arbitrarily set as 2 pixels × 2 pixels, 3 pixels × 3 pixels, 4 pixels × 4 pixels, 3 pixels × 5 pixels, 5 pixels × 4 pixels. The blocks included in the window can also be set arbitrarily, such as 4 blocks × 4 blocks, 3 blocks × 4 blocks, and 5 blocks × 3 blocks.
[0055]
A correlation value between the block of the first frame and the block of the third frame, which are point-symmetric with respect to the block [2, 5] of the second frame, is calculated in the correlation calculation circuit 15A. That is, the correlation value S between the block [1,1] of the first frame and the block [3,9] of the third frame ₁ , The correlation value S between the block [1, 2] and the block [3, 8] ₂ , The correlation value S between the block [1,3] and the block [3,7] ₃ , The correlation value S between the block [1, 4] and the block [3, 6] ₄ , The correlation value S between block [1,5] and block [3,5] ₅ , The correlation value S between the block [1, 6] and the block [3, 4] ₆ , The correlation value S between block [1,7] and block [3,3] ₇ , The correlation value S between the block [1, 8] and the block [3, 2] ₈ , And the correlation value S between the block [1, 9] and the block [3, 1] ₉ Is calculated.
[0056]
These correlation values S _i The smallest value among (i = 1 to 9) is detected in the interpolation reference block selection circuit 16A. The pair of blocks that produces the smallest correlation value is the one with the strongest correlation.
[0057]
Correlation value S between blocks _i Is calculated, for example, as follows. Correlation value S between block [1,1] and block [3,9] ₁ Will be described as an example.
[0058]
Each of these blocks [1, 1] and [3, 9] includes 8 × 8 = 64 pixels. Correlation value s between corresponding pixels in blocks [1,1] and [3,9] _j Is calculated. That is, the correlation value between the pixel (1, j) in the block [1, 1] and the pixel (3, j) in the block [3, 9] is s. _j (J = 1 to 64). As described above, the correlation value between pixels is obtained by calculating the absolute value of the difference between the pixel data of two pixels, the square value of the difference, and the like. Next, the correlation value s between these pixels _j Are added in one block. Correlation value s in one block _j Is the correlation value S between the blocks _j It becomes. An average value may be used instead of the sum.
[0059]
When a pair of blocks having the smallest correlation value (one block in the first frame and one block in the third frame) are selected, the center of the window in the second frame is used by using the image data of these selected blocks. Is generated by the interpolation operation.
[0060]
For example, if the block [1,1] of the first frame and the block [3,9] of the third frame are selected, the block [2,5] of the second frame is used by using the image data of the pixels included in these blocks. ] Is calculated as follows. The image data of the pixel (2, j) in the block [2, 5] is the image data of the pixel (1, j) in the block [1, 1] and the image data of the pixel (3, j) in the block [3, 9]. It is obtained by an interpolation operation using data (for example, an average value operation). This interpolation calculation is performed by the interpolation pixel value calculation circuit 17A. This circuit 17A outputs image data for one block (8 × 8 pixels).
[0061]
The windows in the first frame and the third frame are sequentially scanned in the horizontal and vertical directions in units of the length of one side of the block (the number of pixels), and image data of the second frame is obtained for each block of pixels at each position. Will be done.
[0062]
The circuit shown in FIG. 4 can be realized by hardware, or a part or all of its functions can be achieved by software. In any case, real-time processing (for example, processing time of 1/30 second per frame) is possible. The hardware circuit may be an extension of the circuit described with reference to FIG. The window circuits 13A and 14A will be composed of three blocks (for example, 8 × 3 = 24) of line memories. The correlation operation circuit 15A includes a number of subtraction circuits corresponding to the number of pixels (for example, 8 × 8 = 64) included in one block, and an absolute value circuit or a squaring circuit will be connected to each subsequent stage of these subtraction circuits. Also, an adder circuit for calculating the sum of the outputs of these absolute value circuits or squaring circuits will be provided. The interpolation reference block selection circuit will include a minimum value detection circuit and the same number of comparison circuits as the number of blocks. When calculating the interpolated image data for each pixel, the interpolated pixel value calculation circuit 17A may include one average value circuit. The calculated image data for one pixel is sequentially output.
[0063]
FIG. 6 shows another method of motion detection. In this manner, the circuit shown in FIG. 4 is applied.
[0064]
The correlation value between the image data in the upper left block [1,1] of the window set in the first frame and the image data in the lower right block [3,256] of the window set in the third frame is calculated. Is done. Next, block [1, 2] is set at a position shifted by one pixel to the right from block [1, 1] in the window of the first frame. Similarly, block [3,255] is set at a position shifted by one pixel to the left from block [3,256] in the window of the third frame. The correlation value between the pixel data in these blocks [1, 2] and the image data in block [3, 255] is calculated.
[0065]
As described above, in the first frame, the block is shifted to the right in the horizontal direction by one pixel, and in the third frame, the block is shifted to the left in the horizontal direction by one pixel. A correlation value between the image data in the block of the frame and the image data in the third frame is calculated.
[0066]
When the block reaches the right end in the first frame and reaches the left end in the third frame, the block is set at a position shifted by one pixel from the position of the block [1, 1] in the first frame. In three frames, blocks are set at positions shifted upward by one pixel from the position of block [3, 256], and correlation values of image data in these blocks are calculated. In the first frame, the correlation value is calculated at each position while the block is shifted to the right in the horizontal direction, and in the third frame, the block is shifted to the left in the horizontal direction by one pixel.
[0067]
In the first frame, the blocks are horizontally shifted from left to right and vertically from top to bottom. In the third frame, the blocks are horizontally shifted from right to left and vertically from bottom to top. The correlation value between the image data of the block of the first frame and the image data of the block of the third frame is calculated at all positions while shifting by the pixel. The block of the first frame and the block of the third frame are always in point symmetry with respect to the block [2,128] located at the position corresponding to the center position of the window in the second frame.
[0068]
If the window is square and its side is 24 pixels, and the block is square and its side is 8 pixels, 256 correlation values are obtained. The smallest of these correlation values is selected. The image data of each pixel included in the block [2,128] of the second frame is interpolated using the image data of the selected block, and the window is moved horizontally and vertically by pixels on one side of the block. The above-described processing is performed while shifting by minutes, as in the case of the second embodiment.
[0069]
When calculating the correlation value between blocks, instead of shifting the block by one pixel as described above, the block may be shifted by several pixels. This is also applicable to the first embodiment for calculating the correlation value between pixels.
[0070]
Although the inter-frame interpolation is performed in the above embodiment, the interpolation processing can be performed in units of field images.
[0071]
When the position of the image is shifted in a frame (or field) before and after the missing frame (or field), for example, the position of the image in the first frame and the position of the image in the third frame are shifted. (If the position of the image is displaced in two frames due to movement of the camera, for example, not due to the movement of the image), and if the amount of this displacement is known in advance, this After the correction (shifting of the image data by an amount corresponding to the shift amount) in the frame of (1), the interpolation image may be generated by the above-described processing. This shift correction can also be achieved by shifting the window setting position in any frame. Of course, it is not necessary to perform the displacement correction. After generating the interpolation image by performing the shift correction, the shift may be returned to the original state.
[0072]
If the direction and amount of motion of the image in the time between two frames (or fields) are known in advance, a pixel pair or block pair for which a correlation value is to be calculated is selected according to the direction and amount of motion. Thus, the number of correlation value calculations can be reduced. If the magnitude of the movement is too large to fit within the window, the position of the window in one of the frames (fields) is shifted in advance according to the direction and magnitude of the movement, or at least one of the images May be shifted according to the direction and magnitude of the movement, and then the above-described interpolation image data generation processing may be performed.
[0073]
The above-described interpolation image data generation processing is performed on any image data such as only luminance data, both luminance data and chrominance data, and all or part of R, G, and B color image data (for example, only B color image data). Applicable. In this case, if the degree of motion obtained based on the luminance data is different from the degree of motion obtained based on the chrominance data, either one of the motion detection results may be used, or the weighted average value may be used. May be used.
[0074]
As described in the section of the prior art, in order to efficiently transmit or store a moving image represented by an image of a number of continuous frames, for example, every other frame is dropped from a number of continuous frames. . The image data of the image remaining after the frame is dropped and transmitted or stored according to an appropriate image compression algorithm.
[0075]
The interpolated image data generating apparatus and method according to the present invention are useful when restoring an image of a frame lost due to such a frame loss. The compressed image data is provided to the interpolated image data generation device after decompression processing. In this device, an image of a missing piece due to a missing piece is generated. As a result, the original moving image represented by the continuous frame images is reproduced.
[0076]
The interpolated image data generating device according to the present invention is also suitably used in a digital video tape recorder or a digital image data reproducing device described in detail below.
[0077]
Before describing the configuration and operation of the digital video tape recorder, an existing standard industry standard relating to a recording method on a magnetic tape by the digital video tape recorder will be described.
[0078]
The recording format of the magnetic tape is shown in FIGS. 10 (A) and 10 (B). FIG. 10A shows tracks Tr of the magnetic tape 28, and a large number of tracks Tr are formed at a fixed angle in a diagonal direction with respect to the longitudinal direction of the magnetic tape 28. Digital image data for one frame is recorded using ten consecutive tracks among these many tracks Tr.
[0079]
FIG. 10B shows the track format. One track Tr includes a subcode recording area, a video recording area, an auxiliary recording area, an audio recording area, and a track information recording area. Information such as a time code and an absolute track number for high-speed search is recorded in the subcode recording area. Digital image data representing a subject image is recorded in the video recording area. Data representing a sound is recorded in the audio recording area. In the track information recording area, information serving as a reference of the track Tr for the magnetic head to trace the center of the track Tr is recorded. The auxiliary recording area is provided at intervals, and additional information is recorded in the auxiliary recording area. The illustration of the gaps provided between the respective regions is omitted.
[0080]
A CCD used in a digital video tape recorder generally has 720 pixels in the horizontal direction and 480 pixels in the vertical direction and has about 350,000 pixels. One frame of digital image data obtained using such a CCD is recorded on ten tracks of the magnetic tape 28. This is an existing standard.
[0081]
In order to improve the image quality of the image represented by the image data recorded on the magnetic tape 28, a CCD having a large number of pixels may be used. However, in a digital video tape recorder, the standard is defined so that digital image data obtained by imaging a subject using a CCD having 350,000 pixels is recorded on 10 tracks. Therefore, if digital image data obtained by capturing an image of a subject using a CCD having a larger number of pixels than 350,000 pixels is recorded on the magnetic tape 28, this standard is not met. The digital video tape recorder described here captures an image of a subject using a CCD having a larger number of pixels than 350,000 pixels, which is generally used for a digital video tape recorder, and uses a digital camera. -It is possible to record image data conforming to the conventional recording standard of a video tape recorder.
[0082]
FIG. 7 is a block diagram showing an electrical configuration of a digital video tape recorder (DVTR) capable of recording and reproducing digital image data. The overall operation of the digital video tape recorder is controlled by a system controller 30 (control lines are not shown).
[0083]
As shown in FIG. 8, the CCD 31 used in this digital video tape recorder has about 700,000 pixels of 1440 pixels in the horizontal direction and 480 pixels in the vertical direction. Is obtained twice as much image data. The data amount of one frame of image data obtained from the 700,000 pixel CCD 31 corresponds to the image data of two frames of the 350,000 pixel CCD, that is, the data amount of image data of four fields.
[0084]
In this digital video tape recorder, one frame of image data obtained from the CCD 31 is divided into four fields of image data, and each two fields is recorded on the magnetic tape 28 using ten tracks each. It enables recording of image data that conforms to the conventional recording standards of digital video tape recorders while achieving high image quality shooting.
[0085]
In the recording mode, the subject is continuously photographed at a constant period of 1/15 second by the CCD 31 having about 700,000 pixels. The shutter speed for photographing is determined by so-called electronic shutter control so that the shutter speed is appropriate (for example, 1/60 second, or a shorter time if necessary). A video signal representing a subject image is output from the CCD 31 every 1/15 second and supplied to a CDS (correlate double sample) circuit 32. The video signal is amplified in the CDS circuit 32 and converted to digital image data in the analog / digital conversion circuit 33. The digital image data is applied to a gamma correction circuit 34 and gamma corrected. The digital image data after the gamma correction is supplied to the frame memory 25 via the data compression circuit 35 (in this case, the data is not yet compressed), and is temporarily stored. The frame memory 25 has a capacity to store color image data for 700,000 pixels per frame.
[0086]
One frame of image data stored in the frame memory 25 is divided into four fields of image data in the process of being read from the frame memory 25. The method of this division is as follows.
[0087]
FIG. 9 schematically illustrates a pixel array in the CCD 31 and a state in which image data of these pixels is divided into four fields. In the horizontal direction and the vertical direction, the number of pixels is drawn extremely small for convenience of illustration. The arrangement method (mosaic, stripe, delta, etc.) does not matter here. As shown in the upper part of FIG. 9, one frame of image data output from the CCD 31 and subjected to A / D conversion is applied to all pixels in the horizontal and vertical directions of the CCD 31 (720 pixels in the horizontal direction and 480 pixels in the vertical direction). Image data. The frame memory 25 stores image data corresponding to all the pixels of the CCD 31. The image data is a concept including image data (luminance data) representing a black-and-white image and image data (R, G, B color data or a combination of luminance data and color difference data) representing a color image. In particular, in the case of color image data, the arrangement of color data or color difference data will differ depending on the arrangement of the CCD color filters, the resampling method, and other processing methods. For example, one pixel may be represented by R, G, and B primary color data, or may be represented by one or two primary color data.
[0088]
Reading of these image data from the frame memory 25 is performed over four fields from the first field to the fourth field (see the lower part of FIG. 9).
[0089]
The image data of the first field read out at the first time and the image data of the third field read out at the third time are image data of the pixels of the odd-numbered rows. These odd-numbered rows of image data are represented by black triangles and white triangles. Black triangles indicate image data in odd columns, and white triangles indicate image data in even columns. The image data of the first field is configured by alternately repeating the image data of the odd and even columns in the vertical direction. The image data of the third field is configured by alternately repeating the image data of the even-numbered columns and the image data of the odd-numbered columns in the vertical direction, and these are not included in the first field.
[0090]
The image data of the second field read out in the second time and the image data of the fourth field read out in the fourth time are image data of pixels in even-numbered rows. These even lines of image data are represented by white circles and black circles. White circles indicate image data in odd columns, and black circles indicate image data in even columns. The image data of the second field is constituted by alternately repeating the image data of the odd and even columns in the vertical direction. The image data of the fourth field is configured by alternately repeating even-numbered column image data and odd-numbered column image data in the vertical direction, and these are not included in the second field.
[0091]
In this way, the image data of all the pixels is read out only once in any one of the fields. Moreover, the pixels of the image data constituting each field are discrete in the vertical and horizontal directions, and the entire subject image can be expressed by the image data of any field. One frame is composed of the first field and the second field. One frame is constituted by the third field and the fourth field. Each of these frames is suitable for interlaced scanning. Therefore, in the reproduction of a moving image to be described later, the reproduction and display by the conventional normal interlace scanning can be performed.
[0092]
Reading of image data over these four fields can be easily expressed by controlling the addressing of the frame memory 25. For example, when reading the first field, an odd-numbered row is specified as a vertical address. As the horizontal address, an address signal for designating an odd column in the case of (4m + 1) rows and an even signal in the case of (4m + 3) rows may be generated (m = 0, 1, 2,...). The reading of the image data for one field may be performed in 1/60 second. Such address signal generation may be performed by the system controller 30, or may be constituted by a counter, a frequency divider, a logic circuit, or the like.
[0093]
The image data read from the frame memory 25 is sequentially applied to the data compression circuit 35 from the image data of the first field. Data compression is performed on image data by performing DCT (Discrete Cosine Transform) processing, quantization processing, and the like in the data compression circuit 35. The image data compressed in the data compression processing circuit 35 is applied to (simply passed through) the error correction code adding circuit 36 to the frame memory 26, where it is temporarily stored.
[0094]
The image data for four fields stored in the frame memory 26 is sequentially applied to an error correction code adding circuit 36, where an error correction code is added. The image data to which the error correction code has been added is again provided to the frame memory 26 and stored. The image data is read out again from the frame memory 26 and applied to the error correction code adding circuit 36. The error correction code adding circuit 36 is also provided with recording area data from the system controller 30. This recording area data identifies four field data that constitute one frame, and is added to each of the four field image data (auxiliary area).
[0095]
The image data output from the error correction code adding circuit 36 is applied to a recording / encoding circuit 37, encoded (for example, NRZI encoded), and applied to the recording / reproducing amplifying circuit 27. The image data amplified by the recording / reproducing amplifying circuit 27 is given to a magnetic head 38. As a result, the video recording area A of each track of the magnetic tape 28 is moved by the magnetic head 38. ₂ Image data is recorded in the auxiliary recording area A ₃ The recording area data is recorded in the. Auxiliary recording area A ₃ In a field, when one frame image is divided into four fields, recording area data indicating on which track of the magnetic tape 28 the image data representing these four fields is recorded. Recording of audio data and track information is of course also performed.
[0096]
One frame of image data obtained by using the 700,000-pixel CCD 31, that is, the first two fields of image data of the four-field image data are subject to image data using the generally used 350,000-pixel CCD 31. Corresponds to the data amount of one frame of image data obtained by photographing the video recording area. ₂ Recorded in. Of the four fields of image data, the remaining two fields of image data are the video recording area A of the next ten tracks after the first two fields of image data have already been recorded. ₂ Recorded in. One frame of image data obtained by using the 700,000 pixel CCD 31 has twice the data amount of one frame of image data obtained by using the generally used 350,000 pixel CCD 31, The data is recorded on the magnetic tape 28 using 20 tracks. The recording operation for four fields is performed at a period of 1/15 second similarly to the photographing operation for four fields.
[0097]
The digital video tape recorder shown in FIG. 7 can also reproduce digital image data recorded on the magnetic tape 28. The playback modes preferably include a movie playback mode and a still playback mode.
[0098]
In the digital image data reproduction mode, image data, recording area data, and other data recorded on the magnetic tape 28 are read by the magnetic head 41 and supplied to the recording / reproducing amplifier 27. The data amplified by the recording / reproducing amplifying circuit 27 is supplied to a demodulating circuit 42. Data demodulation is performed in the demodulation circuit 42, and is applied to the frame memory 26 via the error correction circuit 43 and is temporarily stored. The data stored in the frame memory 26 is read and applied to the error correction circuit 43. If there is a data error in the data demodulated by the demodulation circuit 42, the error correction circuit 43 performs an error correction process. Digital image data representing the subject image among the data subjected to the error correction processing is applied to the frame memory 25 via the data expansion circuit 44, and the recording area data is applied to the system controller 30. This reproduction operation is also performed at a period of 1/15 second for the image data for four fields.
[0099]
In the movie playback mode, the image data stored in the frame memory 25 is supplied to the data expansion circuit 44, and the data expansion processing of the compressed image data is performed.
[0100]
By photographing at 1/15 second intervals as described above, image data (first field to fourth field) of a data amount corresponding to two frames is obtained, and these image data are recorded on the magnetic tape 28 for a total of 20 tracks. Recorded in. Of these four fields of image data, two fields (for example, the image data of the first field and the second field are interpolated at an interval of 1/15 second) according to the present invention. 45 and the delay circuit 49. The remaining two fields of image data are not used in the movie reproduction mode.
[0101]
Since the interpolated image data generating device 45 receives image data of odd frames (see FIG. 13) at 1/15 second intervals (two frame periods), the image data of these two frames temporally adjacent to each other is input. , Image data of an even-numbered frame (see FIG. 13) temporally located therebetween is generated.
[0102]
The delay circuit 49 delays image data for one frame by two frame periods. Therefore, of the two frames of image data read from the magnetic tape 28, one frame of image data (odd frame) is supplied from the delay circuit 49 to the one frame of image data (even frame) generated by the interpolation processing. The interpolated image data generation device 45 alternately supplies the data to the monitor display device 50 every frame period.
[0103]
Interlaced scanning is performed based on the image data of each frame provided to the monitor display device 25, and a moving image is reproduced on the monitor display device 25. By the interpolation processing in consideration of the above-described movement in the interpolation image data generation device 45, the movement of the image in the displayed moving image becomes smooth. The monitor display device 50 may be provided on a digital video tape recorder.
[0104]
The digital video tape recorder shown in FIG. 7 is capable of reproducing still images in addition to reproducing moving images. In the still reproduction mode, the image data subjected to data expansion in the data expansion circuit 44 is supplied to the image synthesis circuit 46. In the image synthesizing circuit 46, four field data constituting one frame are identified based on the recording area data given to the system controller 30, and as shown in FIG. , One frame of image data is generated.
[0105]
One frame of image data generated in the image synthesizing circuit 46 is given to the vertical interpolation circuit 47.
[0106]
The CCD 11 used in the digital video tape recorder shown in FIG. 7 has 1440 pixels in the horizontal direction and a large number of pixels, but has 480 pixels in the vertical direction and has a general number of pixels (350,000 pixels). It is not much different from the number of pixels in the vertical direction. In order to increase the resolution in the vertical direction, a process of interpolating pixel data in the vertical direction is performed. The vertical interpolation circuit 47 performs this interpolation processing.
[0107]
The image data subjected to the vertical interpolation processing in the vertical interpolation circuit 47 is supplied to a printer 48, and a high-quality still image is printed. The vertical interpolation processing circuit 47 may be provided in the printer 48.
[0108]
However, the image data may be provided to the printer 48 without performing the vertical interpolation processing. The image data (vertically interpolated or not) may be provided to the monitor display device 50 to display a high-quality still image.
[0109]
The vertical interpolation processing in the vertical interpolation circuit 47 is performed as follows.
[0110]
FIG. 12 shows a part of the pixel array, and the central pixel S _{m, n} Are pixels for which the image data is to be generated by interpolation. The image data S of the pixel to be interpolated _{m, n} (Hereinafter, image data uses the same symbol as the pixel) _{m, n} Pixel D adjacent above and below _{m, n} Image data and D _{m, n + 1} Difference from the image data Δ ₁ = | D _{m, n + 1} -D _{m, n} |, Pixel D adjacent diagonally up and down _{m-1, n} And D _{m + 1, n + 1} And image data difference Δ ₂ = | D _{m + 1, n} -D _{m-1, n + 1} | And D _{m + 1, n} And D _{m-1, n + 1} And the difference Δ ₃ = | D _{m-1, n + 1} -D _{m + 1, n} | Are calculated respectively.
[0111]
Subsequently, the difference Δ between the calculated image data ₁ , Δ ₂ And Δ ₃ Is detected. The arithmetic mean of the two image data used to calculate the smallest difference is calculated. The pixel S to be interpolated is obtained from the data obtained by the arithmetic averaging. _{m, n} Image data. For example, Δ ₁ ~ Δ ₃ Δ ₁ Is the smallest, the image data of the pixel to be interpolated is S _{m, n} = (D _{m, n + 1} + D _{m, n} ) / 2. Δ ₂ Is the smallest when _{m, n} = (D _{m + 1, n} + D _{m-1, n + 1} ) / 2, and Δ ₃ Is the smallest when _{m, n} = (D _{m-1, n + 1} + D _{m + 1, n} ) / 2.
[0112]
As a result, the vertical interpolation is performed so that the correlation becomes strong in the vertical and diagonal directions, so that the edges of the obtained still image become smooth.
[0113]
Such vertical interpolation processing is performed between all columns, and image data having substantially twice 480 pixels in the vertical direction is obtained (see the lowermost part of FIG. 11).
[0114]
A switch (or a key or the like) for switching or selecting between the recording mode and the reproduction mode is provided, and the above-described switch between the recording mode and the reproduction mode is performed. Further, a switch for switching or selecting between the movie playback mode and the still playback mode among the playback modes is provided, and the switch between the movie playback and the still playback is performed by this switch. The mode selection signals from these switches are supplied to the system controller 10, and the system controller 30 controls each circuit so that the processes in the various modes described above are executed.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating an interpolation image data generation device according to a first embodiment.
FIG. 2 is a perspective view showing a correspondence relationship between pixels for interpolation.
FIG. 3 is a plan view showing a correspondence relationship between pixels for interpolation.
FIG. 4 is a block diagram illustrating an interpolated image data generation device according to a second embodiment.
FIG. 5 is a plan view showing a correspondence relationship between pixels for interpolation.
FIG. 6 shows how blocks are shifted.
FIG. 7 is a block diagram showing an electrical configuration of the digital video tape recorder.
FIG. 8 shows a configuration of a CCD.
FIG. 9 shows a procedure for dividing an image for one frame into four field images.
10A shows a recording format of a magnetic tape, and FIG. 10B shows a track format.
FIG. 11 shows a procedure for forming a one-frame image and a vertical interpolation image from four field images.
FIG. 12 is a diagram illustrating an example of a pixel array and explaining vertical interpolation.
FIG. 13 shows a dropped frame and frames before and after the dropped frame.
[Explanation of symbols]
11,12 frame memory
13,14,13A, 14A window circuit
15, 15A correlation operation circuit
16 Interpolation reference pixel selection circuit
16A interpolation reference block selection circuit
17, 17A interpolation pixel value calculation circuit

Claims (7)

The first image, the second image, and the third image, each constituting one frame, are continuous in this order, and the second image is missing, and the first image representing the first image is omitted. When given image data and third image data representing the third image, the second image data representing the second image therebetween is replaced with the first image data and the third image data. It is a device that generates using data.
In the windows set at the corresponding positions on the first image and the third image, the windows are point-symmetric with respect to the center unit area at the position corresponding to the center of the window on the second image. Unit area image data extracting means for extracting image data of the unit area while changing the position of the unit area on the first image and the third image;
A correlation value calculating unit that calculates a correlation value between the unit area image data of the first image and the unit area image data of the third image extracted by the unit area image data extracting unit; and the correlation value calculating unit. An interpolating means for interpolating the image data of the central unit area of the second image based on the pair of unit area image data having generated the correlation value representing the strongest correlation among the calculated correlation values;
An interpolated image data generation device comprising: While sequentially shifting the position of the window in the horizontal and vertical directions by the horizontal and vertical sizes of the unit area on the first image and the third image, respectively, 2. The interpolated image data generating apparatus according to claim 1, further comprising means for controlling to repeat the unit area image data extracting operation by the area image data extracting means, the correlation operation by the correlation operation means, and the interpolation operation by the interpolation operation means. . 2. The interpolation image data generation device according to claim 1, wherein the unit area has a size of one pixel, and the window has a size of an odd number of pixels (excluding one pixel) in the horizontal and vertical directions. 2. The interpolated image data generation device according to claim 1, wherein the unit area is a block including a plurality of pixels, and the window has a size of an odd number of blocks (excluding one block) in the horizontal and vertical directions. The unit area is a block including a plurality of pixels, the window is set to be larger than one block, and the unit area image data extracting means sets the position of the block by one pixel or a plurality of pixels in the horizontal and vertical directions. 2. The interpolation image data generating apparatus according to claim 1, wherein the unit area image data is taken out while changing. The first image, the second image, and the third image each forming one frame are continuous in this order, and the second image is missing, and the first image representing the first image is omitted. When given image data and third image data representing a third image, the second image data representing the second image therebetween is replaced with the first image data and the third image data. It is a method of generating using data.
In the windows set at the corresponding positions on the first image and the third image, the windows are point-symmetric with respect to the center unit area at the position corresponding to the center of the window on the second image. Extracting the image data of the unit area while changing the position of the unit area on the first image and the third image;
Calculating a correlation value between the extracted unit area image data of the first image and the unit area image data of the third image,
Interpolating the image data of the central unit area of the second image based on the pair of unit area image data that generated the correlation value representing the strongest correlation among the calculated correlation values;
Interpolated image data generation method. While sequentially shifting the position of the window in the horizontal and vertical directions by the horizontal and vertical sizes of the unit area on the first image and the third image, respectively, 7. The method according to claim 6, wherein all the second image data representing the second image is obtained by repeating the area image data extracting process, the correlation operation, and the interpolation operation.

Images