JP2006279982A - Motion vector detecting method, frame interpolation image creating method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable accurate motion vector detection suitable for creating an interpolation image. <P>SOLUTION: The method for detecting a motion vector between an m-th frame ((m) is an arbitrary integer) and an (m+n)th frame ((n) is an integer of ≥1) of an image includes: extracting a plurality of first blocks of a predetermined size and a predetermined shape from the m-th frame (S101); extracting a plurality of second blocks of the same size and the same shape as the first blocks from the (m+n)th frame (S102); determining a differential absolute value for each of images corresponding to each other between the first blocks and the second blocks (S103); counting the number of pixels with the differential absolute value equal to or less than a threshold, to determine a count value (S104); extracting a pair of blocks each including a pixel of the maximum count value from the first blocks and the second blocks and selecting a vector between such block pairs as a motion vector between the m-th frame and the (m+n)th frame (S105). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a motion vector detection method, a frame interpolation image creation method by motion compensation using a motion vector, and an image display system.

  In general, as an image display device, an impulse display device (for example, a CRT or a field emission display device (FED) that continues to emit light only after the phosphor afterglow time after image writing, and before the image writing is newly performed. There are two types of hold-type display devices (for example, a liquid crystal display device (LCD), an electroluminescence display (ELD), etc.) that keeps displaying the frame.

One of the problems with the hold-type display device is a blurring phenomenon that occurs in moving image display. As shown in FIG. 28 (a), the occurrence of the blur phenomenon occurs when a moving object (indicated by an ellipse) exists in an image extending over a plurality of frames, and the observer's eyes follow the movement of the moving object. This is because the images of a plurality of frames are superimposed and projected on the retina as shown in FIG. Until the period when the display image switches from the previous frame to the next frame, the eye predicts the display of the image of the next frame even though the image of the same previous frame continues to be displayed. Is observed while moving in the moving direction of the moving object. In other words, the following movement of the eye is continuous, and sampling is performed finer than the frame interval. As a result, it is observed as blurring by visually recognizing an image between two adjacent frames.
Another problem with the hold-type display device is that an unnatural movement occurs when a moving image with a small number of frames is displayed.

  In order to solve these problems, the display frame interval may be shortened. As a specific method, it is conceivable to create an interpolated image using motion compensation used in MPEG (Motion Picture Experts Group phase 2) and perform interpolation between adjacent frames. In motion compensation, a motion vector detected by block matching is used. However, in MPEG2, since an image is created in units of blocks, a correlated portion and a non-correlated portion occur in the block, and block distortion occurs due to the uncorrelated portion.

Japanese Patent Laid-Open No. 2000-224593 (Patent Document 1) discloses an inter-frame interpolation method that solves this problem. In Japanese Patent Laid-Open No. 2000-224593, when determining the pixel value of the interpolation block in the interpolation frame, the motion compensation interframe difference absolute value between the two frames of the small block to be decoded is compared with the threshold value for each corresponding pixel, and below the threshold value Are divided into a first pixel region having a difference absolute value of 2 and a second pixel region having a difference absolute value equal to or greater than a threshold value. For the first pixel area, an average value of the pixel value of the area and the corresponding pixel value in the reference block indicated by the motion vector between the two frames is obtained to create an interpolation frame. For the second pixel region, the hidden surface relationship in the decoding target frame is determined, the motion vector search direction between the two frames is reset based on the determination result, the second motion vector is detected, and the motion vector obtained by scaling this is detected. When it is pointed to, the pixel value on the search reference frame is copied to the interpolation frame.
JP 2000-224593 A.

  In Japanese Patent Laid-Open No. 2000-224593, there is no particular description about a motion vector detection method, but if a motion vector detection method similar to that of normal MPEG2 is adopted, an erroneous motion vector is likely to be used. MPEG2 is basically premised on increasing the compression rate, and motion vector errors are generated by motion compensation so that the motion vector error accurately reproduces the actual motion. This is because it does not matter much.

  In Japanese Patent Laid-Open No. 2000-224593, since the second pixel area is not necessarily in the hidden surface relationship in the interpolation of the second pixel area, there is a possibility that an erroneous area is extracted by the determination. Further, when the interpolation image is created, the first pixel region is interpolated by a method in which the averaged pixel data is stored at the address position on the interpolation frame memory using the scale-converted vector. Overlap can occur. As for the second pixel region, when the same method is used, a gap or an overlap may occur in the interpolation image.

An object of the present invention is to enable accurate motion vector detection suitable for creating an interpolation image.
More specifically, the present invention is suitable for area division within a block based on a threshold value, and an optimal motion vector can be detected for each area for a pixel block after area division.
Furthermore, the present invention provides a gap or overlap for each pixel block after segmentation when creating an interpolated image to improve blur caused by the hold-type display device and unnatural motion in a moving image with a small number of frames. It is an object of the present invention to provide a frame interpolation image creation method and an image display system using the frame interpolation image creation method that can make the display image quality more realistic.

  In order to solve the above problems, in the first aspect of the present invention, a motion vector between an mth frame (m is an arbitrary integer) and an m + n frame (n is an integer of 1 or more) of an image composed of a plurality of pixels. In the motion vector detection method to detect, (a) a plurality of first blocks having a predetermined size and a predetermined shape obtained by dividing the m-th frame are extracted from the m-th frame, and (b) the same size as the first block from the m + n frame. In addition, a plurality of second blocks having the same shape are extracted, and (c) a difference absolute value for each corresponding pixel between the first block and the second block is obtained, and the number of pixels for which the difference absolute value is equal to or less than a threshold is calculated. (D) extracting block pairs each including a pixel having the maximum count value from the first block and the second block, and (e) calculating a vector between the block pairs as the m-th frame. And then selecting a motion vector between m + n frames.

  In the second viewpoint of the present invention, more accurate motion is obtained by bidirectional motion vector detection in which a method for obtaining a motion vector from the m + n frame to the mth frame is added to the motion vector detection method according to the first viewpoint. Find a vector.

  In the third aspect of the present invention, an m + k frame (k is an integer of n−1 or less, k is an arbitrary real number of 1 or more) is assumed between the mth frame and the m + n frame. Using a certain m + k frame as a fulcrum, a block pair having the maximum number of pixels equal to or smaller than the threshold between the mth frame and the m + n frame is extracted, and a vector connecting the block pairs is selected as a motion vector.

  In the fourth aspect of the present invention, after obtaining a plurality of motion vectors for a plurality of block pairs having a number of pixels that is equal to or smaller than a threshold value between the m-th frame and the m + n-th frame, and scaling the motion vectors. , A block pair is extracted by fitting these motion vectors between the m + k (k is an integer of n−1 or less, k is an integer of 1 or more) frame and the m + n frame, which are actual frames of the image, The number of pixels for which the absolute value of the difference is less than or equal to the threshold is counted, and the motion vector that maximizes the number of pixels for which the count value is less than or equal to the threshold is selected.

  In the fifth aspect of the present invention, one block is divided into a plurality of areas, and a motion vector is obtained for each area.

  According to a sixth aspect of the present invention, in order to improve the error that can be caused by a portion that is divided into a small number of regions having about 1 to 2 pixels after the region is divided, a space region is secured in order to secure the size of the region. After passing through the low-pass filter, a motion vector is detected for each region.

  According to a seventh aspect of the present invention, the redundancy of the method for detecting the first motion vector in order to improve an error in which a portion divided into small regions having about one or two pixels after region division may become noise. Threshold processing is performed not only for luminance but also for color differences.

  In an eighth aspect of the present invention, in order to compensate for a lack of data due to a decrease in the number of pixels in each area after area division, adjacent blocks are connected to expand each small area in the spatial direction, and motion vector detection accuracy is improved. Increase.

  According to a ninth aspect of the present invention, motion vector detection in a small region is performed by extending a motion vector search range between adjacent frames in order to compensate for a lack of data due to a decrease in the number of pixels in each region after region division. The accuracy is improved and the detection accuracy between objects having a hidden surface relationship is also increased.

  In the 10th viewpoint of this invention, in order to reproduce the complicated motion in a block, while dividing an area | region, the motion vector detection of each area | region is also repeatedly performed.

  In the eleventh aspect of the present invention, a temporal position without a frame image between the mth frame (m is an arbitrary integer) and the m + n frame (n is an integer of 1 or more) of the original image, that is, the m + k frame (k In an interpolation image creation method for creating an interpolation image to be interpolated at a temporal position of (any real number), (a) the m-th frame obtained by the motion vector detection method of any of the first to fourth viewpoints; The motion vector between the m + n frames and the region-specific motion vector obtained by the motion vector detection method of any of the fifth to tenth viewpoints are scale-converted according to the temporal position of the m + k frame, ( b) According to the motion vector after the scale conversion from the (m + n) th frame, the interpolation block on the (m + k) th frame that is in the same spatial position as the block on the (m + th) frame, and Extract a fifth block that is a destination of the inter-region interpolation block on the (m + k) -th frame at the same spatial position as the pixel block after the region division, and (c) change the block to the (m + k) -th interpolation block and the inter-region interpolation block. An interpolation image is created by assigning 5 blocks.

  According to the present invention, an accurate motion vector can be detected by a motion vector detection method focusing on the shape of an image, and further, a motion vector for each region obtained by dividing a block into regions can also be detected. Therefore, it is possible to create an interpolated image that does not cause block distortion or interpolation error, and it is possible to reproduce and display a more realistic image in a moving image.

Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
As shown in FIG. 1, in order to detect a motion vector between an mth frame (m is an arbitrary integer) 1 and an m + n frame (n is an integer of 1 or more) 2 of the original image, In the embodiment, a procedure as shown in FIG. 2 is used. Hereinafter, the processing procedure of this embodiment will be described with reference to FIGS. 1 and 2.

First, the image data of the m-th frame 1 is divided into a plurality of first blocks 11, and each first block 11 is sequentially extracted (step S101).
Next, the second block 12 having the same size and shape as the first block 11 extracted in step S101 is extracted from the image data of the (m + n) th frame 2 (step S102).
Next, an absolute difference value for each pixel corresponding to the first block 11 extracted in step S101 and the second block 12 extracted in step S102 is obtained (step S103). Therefore, the absolute difference value is obtained by the number of pixels in the block.
Next, each difference absolute value is compared with a predetermined common threshold value, and the number of pixels whose difference absolute value is less than or equal to the threshold value is counted to obtain a count value p (p is an integer greater than or equal to 0) ( Step S104).

  Next, for each first block 11 extracted in step S101, a pair of the first block 11 and the second block 12 having the maximum count value p obtained in step S104 is obtained, and a vector connecting the block pairs is obtained. It selects as a motion vector (step S105).

  FIG. 3 shows how the interpolated image changes depending on the motion vector detection result. When the motion vector detected for the first block 11 is A, an image 3A is created from the second block 12A indicated by the motion vector A. When the motion vector detected for the first block 11 is B, an image 3B is created from the second block 12B indicated by the motion vector B. In this example, the motion vector A is correct and the motion vector B is incorrect.

  In creating an interpolated image, which is the object of the present embodiment, an interpolated image is created where there is no original image, so the image quality of the interpolated image is almost determined by the accuracy of the motion vector. When an erroneous motion vector B is detected as shown in FIG. 3, the image quality of the interpolated image is greatly degraded as in the image 3B.

  FIG. 4 shows an example of image data corresponding to FIG. The motion vector A and the motion vector B shown in FIG. 3 are detected for the first block 11 of the m-th frame 1 shown in FIG. 4A, and are indicated by the motion vector A and the motion vector B from the m + n frame 2. Assume that the second blocks 12A and 12B shown are extracted as shown in FIGS. 4B and 4C, respectively.

  As shown in FIG. 4A, the brightness of the m + n frame 2 image is slightly changed. The second block 12A pointed to by the correct motion vector A is generally brighter than FIG. 4A as shown in FIG. 4B, but the shape is not changed. is important. On the other hand, FIG. 4C, which is the second block 12B pointed to by the wrong motion vector B, is a part of the entire raster image, and the shape itself is different from FIG.

  Next, block pairs of the first block and the second block, that is, pixel values between FIG. 4 (a) and FIG. 4 (b) and between FIG. 4 (a) and FIG. Are obtained, respectively (hereinafter simply referred to as difference sum sum). The difference absolute value sum between FIGS. 4 (a) and 4 (b) is “180” as shown in FIG. 4 (d), whereas the difference absolute value between FIGS. 4 (a) and 4 (c). The sum is “150” as shown in FIG. If the smaller sum of absolute differences is selected, the block pairs in FIGS. 4A and 4C are selected, and the corresponding motion vector B is selected. However, when judged from the shape of an edge or the like, the block pair shown in FIGS. 4A and 4B has a higher correlation.

  FIG. 5 is an example of image data corresponding to FIG. 3 according to the present embodiment, and FIGS. 5A, 5B, and 5C are the same as FIGS. 4A, 4B, and 4C. According to the present embodiment, the threshold value is set to “20”, for example, and the count value p is obtained by counting the number of pixels whose difference absolute value for each pixel is equal to or less than the threshold value between the first and second blocks. In this case, the count value of the number of pixels for which the sum of absolute differences between the first block 11 shown in FIG. 5A and the second block 12A indicated by the motion vector A is equal to or less than the threshold is “ 9 ". The count value of the number of pixels whose sum of absolute differences is equal to or less than the threshold value between FIG. 5A and FIG. 5C which is the second block 12B indicated by the motion vector B is “6”.

  According to the present embodiment, since the motion vector between the block pairs with the larger count value p is selected, the motion vector A between the block pairs in FIGS. 5A and 5B is selected. From experiments conducted by the inventors, it was confirmed that the detection of erroneous motion vectors is reduced by setting the above threshold value so that it is about 3% of the average pixel value (luminance value) of the image. . For example, in an image with 256 gradations, the effect can be confirmed by setting the threshold value to 5 levels.

  Next, FIG. 6 shows a configuration of a motion vector detection apparatus that performs the above-described motion vector detection processing according to the present embodiment. Here, n = 1 is assumed to simplify the description. The input image signal 31 is sequentially input to the (m + 1) th frame memory 32 and the mth frame memory 33, and the (m + 1) th and mth frame image signals 34 and 35 are read from the frame memories 32 and 33, respectively.

  The first block extraction unit 41 performs the process of step S101 in FIG. 2, that is, the first block extraction from the mth frame, and the second block extraction unit 42 performs the process of step S102, that is, the second block from the m + 1th frame. Perform extraction. The image signal of the first block and the image signal of the second block are input to the inter-block difference absolute value calculation unit 51, and the process of step S103, that is, the calculation of the difference absolute value E1 for each corresponding pixel is performed. The difference absolute value E1 is calculated by the following equation.

  Here, D represents a motion vector, X represents a block position vector, and f (X, m) represents pixel data corresponding to the position (X) and frame (m) of each block. This also applies to other embodiments described later.

  The difference absolute value E1 obtained by the difference absolute value calculation unit 51 is input to the count unit 52 of the number of pixels that is equal to or smaller than the threshold value, and the process of step S104, that is, the difference absolute value E1 is compared with the threshold value to be equal to or smaller than the threshold value. A calculation for obtaining the pixel count value p is performed for each block pair of the first and second blocks. The count value p is input to the motion vector selection unit 53, and the process of step S105, that is, the extraction of the block pair having the maximum count value p and the detection / selection of the motion vector are performed.

  As described above, in this embodiment, motion vector detection can be performed with emphasis on the shape between the m-th frame and the m + n frame.

(Second Embodiment)
As shown in FIG. 7, a motion vector from the mth frame (m is an arbitrary integer) of the original image to the m + n frame (n is an integer of 1 or more) is detected, and the m + n frame to the mth frame is detected. Detect motion vectors. In the second embodiment of the present invention, such a bidirectional motion vector is detected according to the procedure shown in FIG. Hereinafter, the processing procedure of this embodiment will be described with reference to FIGS. 7 and 8. In FIG. 8, the processes of steps S201 to S205 are the same as steps S101 to S105 of FIG.

That is, first, the image data of the m-th frame 1 is divided into a plurality of first blocks 11, and each first block 11 is sequentially extracted (step S101).
Next, the second block 12 having the same size and shape as the first block 11 extracted in step S201 is extracted from the image data of the m + n frame 2 (step S202).
Next, a difference absolute value (first difference absolute value sum) for each corresponding pixel between the first block 11 extracted in step S201 and the second block 12 extracted in step S202 is obtained (step S203). .
Next, each first difference absolute value is compared with a predetermined common first threshold value, the number of pixels whose first difference absolute value is equal to or less than the first threshold value is counted, and a first count value p ( p is an integer greater than or equal to 0) (step S204).
Next, for each first block 11 extracted in step S201, a pair of the first block 11 and the second block 12 having the maximum first count value p obtained in step S204 is obtained, and the block pairs are connected. A vector is selected as the first motion vector candidate D (step S205).

Next, the image data of the (m + n) th frame 2 is divided into a plurality of third blocks 13, and each third block 13 is sequentially extracted (step S206).
Next, the fourth block 14 having the same size and shape as the third block 13 extracted in step S206 is extracted from the image data of the mth frame 1 (step S207).
Next, a difference absolute value (second difference absolute value sum) for each corresponding pixel between the third block 13 extracted in step S206 and the fourth block 14 extracted in step S207 is obtained (step S208). .
Next, each second difference absolute value is compared with a predetermined common second threshold value, the number of pixels whose second difference absolute value is equal to or less than the second threshold value is counted, and a second count value q ( q is an integer greater than or equal to 0) (step S209).

  Next, for each third block 13 extracted in step S206, a pair of the third block 13 and the fourth block 14 having the maximum second count value q obtained in step S209 is obtained, and the block pairs are connected. A vector is selected as the second motion vector candidate E (step S210).

Next, the count values p and q are compared (step S211). If p is equal to or greater than q, the first vector candidate D is selected as a motion vector according to the comparison result (step S212). If is smaller than q, the second vector candidate E is selected as a motion vector (step S213).
Note that the first and second threshold values are appropriately selected values, and may be different or the same value.

  Next, FIG. 9 shows the configuration of a motion vector detection apparatus that implements the motion vector detection method according to this embodiment. Here, n = 1 is assumed to simplify the description. In FIG. 9, third and fourth block extraction units 43 and 44 are added to the motion vector detection device according to the first embodiment shown in FIG. Further, in addition to the first block and second block difference absolute value error calculation unit 51A corresponding to the inter-block difference absolute value error calculation unit 51 shown in FIG. 5, the third block and fourth block difference absolute value error. A calculation unit 51B is added, and a count unit 52B for the number of pixels q is added to the count unit 52A for the number of pixels p corresponding to the count unit 52 shown in FIG.

  The process for obtaining the first motion vector candidate is performed in the same manner as the process for obtaining the motion vector D in the first embodiment. The first block extraction unit 41 performs the process of step S201 in FIG. 8, that is, the first block extraction from the mth frame, and the second block extraction unit 42 performs the process of step S202, that is, the second block from the m + 1th frame. Perform extraction.

  The image signal of the first block and the image signal of the second block are input to the inter-block difference absolute value calculation unit 51A, and the processing of step S203, that is, the calculation of the difference absolute value E1 for each corresponding pixel is performed according to equation (1). Do. The difference absolute value E1 obtained by the difference absolute value calculation unit 51A is input to the count unit 52A for the number of pixels P that is equal to or less than the threshold, and the process of step S204, that is, the difference absolute value E1 is compared with the threshold, The calculation for obtaining the count value p of the pixel is performed for each block pair of the first and second blocks. The count value p is input to the motion vector selection unit 53, and the process of step S205, that is, the extraction of the block pair that maximizes the count value p and the detection / selection of the first motion vector candidate D based thereon are performed.

  On the other hand, in the process of detecting the second motion vector candidate E, first, the third block extraction unit 43 performs the process of step S206 in FIG. 8, that is, the third block extraction from the (m + 1) th frame, and the fourth block extraction unit 44. In step S207, the fourth block extraction from the m-th frame is performed. The image signal of the third block and the image signal of the fourth block are input to the inter-block difference absolute value calculation unit 51B, and the process of step S208, that is, the calculation of the difference absolute value E2 for each corresponding pixel is performed. The absolute difference value E2 is calculated by the following equation. Here, E represents the second motion vector candidate.

  The difference absolute value E2 obtained by the difference absolute value calculation unit 51B is input to the count unit 52B of the number of pixels that is equal to or less than the threshold value, and the process of step S209, that is, the difference absolute value E2 is compared with the threshold value to be equal to or less than the threshold value. The calculation for obtaining the pixel count value q is performed for each block pair of the third and fourth blocks. The count value q is input to the motion vector selection unit 53, and the processing of step 210, that is, the extraction of the block pair that maximizes the count value q and the detection / selection of the second motion vector candidate E based on the extraction.

  Next, in step 211, that is, when the first motion vector is detected, the magnitude relationship between the count value p and the count value q is compared. If p is greater than or equal to q, the first motion vector candidate D is selected. When p is smaller than q, the second motion vector E is finally selected as a motion vector.

  As described above, in this embodiment, both the motion vector detection from the m-th frame to the m + n frame and the motion vector detection from the m + n frame to the m-th frame are performed, and the more reliable one is selected. Thus, motion vector detection with higher accuracy can be performed.

(Third embodiment)
As shown in FIG. 10, in order to detect the motion vector F between the mth frame (m is an arbitrary integer) 1 and the m + n frame (n is an integer equal to or greater than k + 1, k is an arbitrary real number) 2 of the original image. Assuming an m + k (k is an arbitrary real number) frame 4 between the mth frame 1 and the m + n frame 2, the directions with respect to the mth frame 1 and the m + n frame 2 are centered on the m + k frame 4. Assume reverse vectors (third motion vector and fourth motion vector).

  Here, the m + k frame 4 is a virtual frame in which the original image does not exist, which is assumed between the mth frame 1 and the m + n frame 2 as described above, and image data that can be newly constructed based on the original image. It is a gathering. In the third embodiment of the present invention, a procedure as shown in FIG. 11 is used. Hereinafter, the processing procedure of this embodiment will be described with reference to FIGS. 10 and 11.

First, the image data of the m + k frame 4 is divided into a plurality of fifth blocks 15, and each fifth block 15 is extracted (step S301).
Next, the sixth block 16 having the same size and shape as the fifth block 15 is extracted from the image data of the mth frame 1 (step S302).
Next, a vector connecting the fifth block 15 and the sixth block 16 is obtained as a third motion vector (step S303). If the motion vector between the mth frame 1 and the m + n frame 2 is F, the corresponding third motion vector is a motion vector obtained by scaling F to a vector between the m + k frame 4 and the mth frame 1. Therefore, as shown in FIG. 10, −F · k / n.
Next, a fourth motion vector that is − (n−k) / k times the third motion vector is calculated (step S304). As described above, if the third motion vector corresponding to the motion vector F between the mth frame 1 and the m + n frame 2 is −F · k / n, the fourth motion vector is F (n−k) / n. This is a motion vector obtained by scaling F to a vector between the m + n frame 1 and the m + k frame 4.
Next, in accordance with the fourth motion vector, the seventh block 17 that is the movement destination of the fifth block 15 is extracted from the m + n frame 2 (step S305).
Next, an absolute difference value E3 for each pixel corresponding to the sixth block 16 and the seventh block 17 is obtained (step S306).
Next, the difference absolute value E3 obtained in step S306 is compared with a predetermined threshold value, the number of pixels that are less than or equal to the threshold value is counted, and a count value r (r is an integer greater than or equal to 0) is obtained (step step). S307).
Next, a block pair having the maximum count value r is obtained from the sixth block 16 and the seventh block 17, and a vector connecting the block pairs is selected as the motion vector F (step S308).

  The configuration of the motion vector detection apparatus that performs the motion vector detection processing according to the present embodiment is basically the same as that in FIG. 6, and the temporally preceding and following images of the m + k frame 4 set at an arbitrary position in time. Data is stored in the frame memories 32 and 33. Here, the absolute difference value E3 is calculated by the following equation.

(Fourth embodiment)
Next, as a fourth embodiment of the present invention, an example in which a vector correctness determination function is added will be described. In this embodiment, unlike the third embodiment, the m + k frame 4 is a frame in which an original image exists. As shown in FIG. 12, the mth frame (m is an arbitrary integer) 1 and the m + n frame (n is an integer equal to or greater than k + 1 and k is an integer equal to or greater than 1) 2 so as to sandwich the m + k frame 4 of the original image. The motion vector between is detected.

  Here, the m + k frame 4 in the present embodiment is an actual frame in which the original image exists as described above, and is different from the m + k frame as the virtual frame described in the third embodiment, but on the time axis. The same position may be used. In the present embodiment, motion vectors are detected using a procedure as shown in FIG. Hereinafter, the processing procedure of this embodiment will be described with reference to FIGS. 12 and 13.

First, the image data of the m-th frame 1 is divided into a plurality of fifth blocks 15, and each fifth block 15 is extracted (step S401).
Next, the sixth block 16 having the same size and shape as the fifth block 15 is extracted from the image data of the m + n frame 2 (step S402).
Next, a vector connecting the fifth block 15 and the sixth block 16 is obtained as the first motion vector F (step S403).
Next, a difference absolute value E4 for each pixel corresponding to the fifth block and the sixth block is obtained (step S404).
Next, the difference absolute value E4 obtained in step S404 is compared with a predetermined threshold value, and the number of pixels equal to or smaller than the threshold value is counted to obtain a count value p (p is an integer of 0 or more) (step S405). ).
Next, the second motion vector F · k / n, which is k / n times the first motion vector F, is calculated corresponding to each of the first motion vectors F (step S406).
Next, in accordance with the second motion vector F · k / n, the seventh block 17 that is the movement destination of the fifth block 15 is extracted from the m + k frame 2 (step S407).
Next, a difference absolute value E5 for each pixel corresponding to the fifth block 15 and the seventh block 17 is obtained (step S408).
Next, the difference absolute value E5 obtained in step S407 is compared with a predetermined threshold, and the number of pixels equal to or smaller than the threshold is counted to obtain a count value q (q is an integer of 0 or more) (step S409). ).
Next, a weighted addition value r (r = x × p + (1−x) r; x is a real number between 0 and 1) is obtained from the count value p and the count value q (step S410).
Next, a vector pair maximizing the weighted addition value r is obtained from the fifth block 15 and the sixth block 16, and the first vector F connecting the block pairs is determined as the motion vector between the mth frame 1 and the m + n frame 2. (Step S411).

  Next, FIG. 14 shows a configuration of a motion vector detection apparatus that performs motion vector detection processing according to the present embodiment. Here, in order to simplify the description, n = 2 and k = 1. The input image signal 30 is sequentially input to the m-th frame memory 31, the (m + 1) th frame memory 32, and the (m + 2) th frame memory 33, and the mth, m + 1th, and m + 2th frame image signals are input from these memories 31, 32, and 33. Is read out.

  The fifth block extraction unit 45 extracts the fifth block from the m-th frame, and the sixth block extraction unit 46 extracts the sixth block from the m + 1-th frame. The first motion vector calculation unit 61 calculates a first motion vector F connecting the fifth block and the sixth block, and the image signal of the fifth block and the sixth block difference absolute value calculation unit 51C correspond to each other. A difference absolute value E4 between the image signal of the fifth block and the image signal of the sixth block is obtained for each pixel. The difference absolute value E4 is input to the counting unit 52A for the number of pixels that are equal to or smaller than the threshold value, and is compared with the threshold value to obtain the count value p of the pixels that are equal to or smaller than the threshold value. The count value p is obtained for each block pair and input to the motion vector selection unit 53.

  Next, the first motion vector F is input to the second motion vector generation unit 62, and a second motion vector F · k / n that is k / n times the first motion vector F is calculated. The seventh block extraction unit 47 extracts the seventh block that is the movement destination of the fifth block from the (m + 1) th frame according to the second motion vector F · k / n. The fifth block image signal and seventh inter-block difference absolute value calculation unit 51D obtains the absolute difference value E5 between the fifth block image signal and the seventh block image signal for each corresponding pixel. The absolute difference value E5 is input to the count unit 52B for the number of pixels that are equal to or less than the threshold value, and is compared with the threshold value to obtain the count value q of the pixels that are equal to or less than the threshold value. The count value q is obtained for each block pair and input to the motion vector selection unit 53.

  Finally, the motion vector selection unit 53 obtains a weighted addition value r (r = x × p + (1−x) r; x is a real number between 0 and 1) from the count value p and the count value q, and the weighted addition value r. The first motion vector F connecting the fifth block and the sixth block with the largest value is selected as the motion vector selected as the motion vector between the mth frame and the m + n frame.

  Here, the absolute difference E4 between the fifth block and the sixth block is calculated by the following equation.

  The difference absolute value E5 between the fifth block and the seventh block is calculated by the following equation.

(Fifth embodiment)
In the fifth embodiment, as shown in FIG. 15, for example, when detecting a motion vector between the mth frame (m is an arbitrary integer) 1 and the m + n frame (n is an integer of 1 or more) 2 of the original image, Each block in each frame is further divided into a plurality of regions, and a function of detecting a motion vector for each region is added. In this embodiment, a procedure as shown in FIG. 16 is used. Hereinafter, the processing procedure of this embodiment will be described with reference to FIGS. 15 and 16. Here, for simplification of explanation, n = 1 and the block is divided into two areas.

First, the image data of the mth frame 1 is divided into a plurality of first blocks 11, and each of the first blocks 11 is sequentially extracted (step S501).
Next, the second block 12 having the same size and shape as the first block 11 is extracted from the image data of the (m + 1) th frame 2 (step S502).
Next, a difference absolute value E1 for each pixel corresponding to the first block 11 and the second block 12 is obtained (step S503).
Next, the difference absolute value E1 is compared with a predetermined first threshold, and the number of pixels for which the difference absolute value E1 is less than or equal to the first threshold is counted to obtain a count value p (p is an integer greater than or equal to 0). (Step S504).
Next, a vector connecting the first block 11 and the second block 12 having the maximum count value p is set as a motion vector (first region-specific motion vector) E between the first region and the (m + 1) th frame (step S505). ).
Next, the difference absolute value E1 is compared with the determined second threshold value, and the pixels in the first block 11 in which the difference absolute value E1 is equal to or less than the second threshold value are extracted as the pixel block 21 in the first region (step). S506), the pixels in the first block 11 where the difference absolute value E1 is larger than the second threshold value are extracted as the pixel blocks 22 in the second region (step S507).
Next, the third block 13 having the same size and shape as the pixel block 22 in the second area is extracted from the image data of the (m + 1) th frame 2 for the pixel block 22 in the second area (step S508).

  Next, a difference absolute value E6 for each pixel corresponding to the pixel block 22 and the third block 13 in the second region is obtained (step S509).

  Next, the difference absolute value E6 is compared with a predetermined third threshold, and the number of pixels that are equal to or smaller than the third threshold is counted to obtain a count value s (s is an integer equal to or greater than 0) (step S510).

Finally, a vector connecting the pixel block 22 and the third block 13 in the second area where the count value s is maximum is selected as a motion vector (second area-specific motion vector) between the second area and the (m + 1) th frame. (Step S511).
The first to third thresholds are values that are appropriately selected, and may be all different or two or more of them may be the same value.

  FIG. 17 shows the original image of the mth frame 1 and the m + n frame 2, and the image of the frame 5 in which the mth frame 1 is reproduced from the motion vector E of the first region when the region is not divided. Yes. Here, the object O1 moves rightward, the object O2 moves downward, and the object O3 is stationary. The motion vector E of the first area is obtained from the component of the object O1, but the object O3 enters the block in the m + n frame 2. If the block is captured in the image of the reproduction frame 5 of the m-th frame 1, an error part is reproduced in the block of the reproduction frame 5, which is a so-called block distortion. This is particularly problematic when generating an interpolated image that does not use a difference signal.

  On the other hand, according to the present embodiment, such block distortion can be prevented by further dividing the block into regions and detecting motion vectors for each region.

  Next, FIG. 18 shows a configuration of a motion vector detection apparatus that performs the above-described motion vector detection processing according to the present embodiment. The input image signal 30 is sequentially input to the (m + 1) th frame memory 32 and the mth frame memory 33, and the (m + 1) th frame image signal and the mth frame image signal are read from these frame memories 32 and 33, respectively.

  The first block extraction unit 41 extracts the first block from the mth frame, and the second block extraction unit 42 extracts the second block from the m + 1th frame. The image signal of the first block and the image signal of the second block are input to the inter-block difference absolute value calculation unit 51A, and the difference absolute value E1 for each corresponding pixel is obtained. Here, the difference absolute value E1 is obtained by the equation (1). The difference absolute value E1 is input to the count unit 52A for the number of pixels that are equal to or smaller than the threshold value, and is compared with the threshold value to obtain the count value p of the pixels that are equal to or smaller than the threshold value. The count value p is obtained for each block pair, and input to the motion vector selection unit 53A in the first area, so that the extraction of the block pair having the maximum count value p and the detection / selection of the motion vector in the first area are performed. Done.

  Next, the pixel block extraction unit 55A in the first region extracts pixels in the first block that are equal to or lower than the threshold value, and the pixel block extraction unit 55B in the second region further extracts pixels in the first block that are larger than the threshold value. To extract. From the pixel block extractor 55B in the second area, the address signal of the pixel in the second area is input to the (m + 1) th frame memory 32, and the third block is extracted in the third block extractor 43 from the frame memory 32.

  For the extracted pixel block and third block in the second area, the difference absolute value E6 for each corresponding pixel is obtained in the difference absolute value calculation unit 51E between the pixel block in the second area and the third block. Here, the difference absolute value E6 is obtained by a calculation similar to the calculation for obtaining the difference absolute value E1 of the equation (1). The difference absolute value E3 is input to the count unit 52B for the number of pixels that are equal to or less than the threshold value and is compared with the threshold value, and a count value q for the number of pixels that is equal to or less than the threshold value is obtained. The count value q is obtained for each block pair, and input to the motion vector selection unit 53B in the second area, so that the block pair with the maximum count value q is extracted and the motion vector in the second area is detected and selected. Done.

(Sixth embodiment)
Next, as a sixth embodiment of the present invention, an embodiment in which the extraction method of the pixel block in the first area and the pixel block in the second area in the fifth embodiment is improved will be described with reference to FIG.

  From the experiments by the inventors, it has been found that when a pixel block in the second region is extracted, a portion divided into a small number of pixel regions having the number of pixels of about 1 to 2 can be noise. In the present embodiment, in order to suppress the occurrence of such noise, as a step before extracting the pixel blocks in the second region, low-pass filtering in the spatial direction is performed to restrict the size of the pixel blocks in the first region to some extent. Through a low-pass filter (spatial domain low-pass filter). This is shown in FIG.

  That is, the low-pass filter 6 in the spatial region is provided in the pixel block extraction unit 55A in the first region in FIG. 18 described in the fifth embodiment, or the low-pass filter processing in the spatial region is added to step S506 in FIG. To do. By passing the pixel block 11 in the first area through the low-pass filter 6 in the spatial area, the pixel block in the small area is removed, and the removed block among the pixel blocks in the first block is the final pixel block in the first area. Set as.

  Various types of low-pass filter 6 in the spatial domain are conceivable. For example, a median filter can be used. In the median filter, whether or not adjacent pixels of the target pixel belong to the first area is determined for the adjacent eight pixels including the target pixel, for example, and when five or more pixels belong to the first area, the target pixel is set as the first area, In the case of 4 pixels or less, the target pixel is determined as the second region. By repeating such processing, it is possible to perform low-pass filtering of the spatial domain for all pixels. In addition to this, after creating a monochrome image (hereinafter referred to as a region map) in which the first region and the second region are color-coded, a Fourier transform is performed on the image to extract low-frequency components, and a region is obtained by inverse Fourier transform. A method of reproducing a map may be used.

(Seventh embodiment)
Next, as a seventh embodiment of the present invention, similarly to the sixth embodiment, an example in which the extraction method of the pixel block of the first region and the pixel block of the second region in the fifth embodiment is improved is illustrated. 20 will be described.

  In the present embodiment, in order to suppress the occurrence of noise due to a portion divided into a small number of pixel regions having about 1 to 2 pixels, the first step is to extract the pixel blocks in the second region. In addition to the threshold determination process for the luminance component used in the normal block matching method in the pixel block extraction process for the region, a threshold determination process for the color difference component is added. Specifically, as shown in the flowchart of FIG. 20, for example, the following processing is performed instead of steps S503 and S504 of FIG.

That is, the difference absolute value (first difference absolute value) E11 for the luminance of each corresponding pixel between the first luminance block having the luminance information of the first block and the second luminance block having the luminance information of the second block is set to Obtained (step S512).
Next, the first difference absolute value E11 is compared with a first threshold value for a predetermined luminance, and a pixel having the first difference absolute value E11 equal to or less than the first threshold value is obtained (step S513).
Next, for the pixels having the first difference absolute value E11 of the first block and the second block equal to or lower than the first threshold, the second color difference information of the first color difference block having the color difference information of the first block and the second color difference information of the second block. Each color difference block is extracted (step S514).
Next, a step of obtaining a difference absolute value (second difference absolute value) E12 for the color difference between the first color difference block and the second color difference block for each pixel for which the first difference value E11 is equal to or less than the first threshold (step S515). .
Next, the second difference absolute value E12 is compared with a second threshold value for a predetermined color difference, and the number of images for which the second difference absolute value E12 is equal to or less than the second threshold value is counted to obtain a count value p (p Is an integer greater than or equal to 0) (step S516).
The processing after Step S516 may be the same processing as Steps S505 to S511 in FIG. 16 except that the pixel block in the first region also has a color difference equal to or less than the threshold value. That is, in step S505, a vector connecting the first block 11 and the second block 12 having the maximum count value p is set as a motion vector (first region-specific motion vector) between the first region and the (m + 1) th frame.
Next, in step S506, the second difference absolute value E12 is compared with the determined second threshold value, and the pixels in the first block 11 in which the second difference absolute value E12 is equal to or less than the second threshold value are determined in the first area. Let it be a pixel block 21.
Next, in step S507, the second difference absolute value E12 is compared with a predetermined second threshold value, and a pixel in the first block 11 that is larger than the second threshold value is set as a pixel block 22 in the second region.
In step S508, the third block 13 having the same size and shape as the pixel block 22 in the second area is extracted from the image data of the (m + 1) th frame 2.
Next, in step S509, a third difference absolute value E13 is obtained for the luminance of each pixel corresponding to the pixel block 22 and the third block 13 in the second region.
Next, in step S510, the third difference absolute value E13 is compared with a predetermined third threshold, the number of pixels that are equal to or smaller than the third threshold is counted, and a count value s (s is an integer equal to or greater than 0) is obtained. Ask.
Finally, in step S511, a vector connecting the pixel block 22 and the third block 13 in the second area where the count value s is maximum is a motion vector between the second area and the (m + 1) th frame (second area-specific motion vector). Choose as.
The first to third thresholds are values that are appropriately selected, and may be all different or two or more of them may be the same value.

(Eighth embodiment)
Next, as an eighth embodiment of the present invention, an example in which the pixel block extraction method for the second region in the fifth embodiment is improved will be described. The difference from the fifth embodiment is that when extracting the pixel blocks in the second area, the pixel blocks in the adjacent second area are connected to perform a motion vector search.

For example, as shown in FIG. 21, the second area of the first block adjacent to the second area of the first block in the m-th frame 1 is spatially connected to form a pixel block of the second spatial expansion area 23. Next, with respect to the pixel block 23 in the second spatial expansion region, a pixel block 24 in the third spatial expansion region having the same size and shape as the pixel block 23 in the second spatial expansion region is obtained from the image data of the m + n frame 2. Extract.
Next, an absolute difference value for each corresponding pixel between the pixel block 23 in the second space expansion region and the pixel block 24 in the third space expansion region is obtained.
Next, the difference absolute value is compared with a predetermined threshold, and the number of pixels that are equal to or smaller than the threshold is counted to obtain a count value s (s is an integer equal to or greater than 0).
Next, a vector connecting the pixel block 23 in the second spatial extension region and the pixel block 24 in the third spatial extension region that maximizes the count value s is selected as the motion vector of the second spatial extension region.

(Ninth embodiment)
Next, as a ninth embodiment of the present invention, another example in which the pixel block extraction method for the second region in the fifth embodiment is improved will be described. The difference from the fifth embodiment is that when a pixel block in the second area is extracted, the search area is expanded in the time direction.

  For example, FIG. 22 shows an example in which there are still objects O1 and O2 and an object O3 moving from the lower right to the upper left of the screen on five frames from the (m-2) th frame 7 to the (m + 2) th frame 9. The object O3 appears on the back of the object O1 and appears at the m-th frame 1, and since the m + 1-th frame 2 and thereafter are on the front of the object O2, it moves so as to cover the object O2. That is, the object O2 disappears in the (m + 1) th frame 2 and the (m + 2) frame 9 due to the object O1.

  As can be seen from FIG. 22, a motion search can be performed between the mth frame 1 and the m−1th frame 8 for the pixel block 21 in the first region of the mth frame 1, but Since the pixel block 21 in the first area disappears between the m + 1 frame 2 and the object O3 on the m + 1 frame 2, the motion search cannot be performed. Similarly, for the pixel block 22 in the second area of the m-th frame 1, a motion search is performed because the pixel block 22 in the second area does not appear between the m-th frame 1 and the m-1 frame 8. I can't.

  As described above, there may be a case where the motion search of the pixel block 21 in the first area and the pixel block 22 in the second area cannot be performed due to the hidden surface relationship of the object. As a method for solving such a hidden surface relationship, it is effective to extend the search area in the time direction.

  Therefore, in this embodiment, when a motion vector between the m-th frame (m is an arbitrary integer) 1 and the m + n (n is an integer of 1 or more) frame 2 is obtained, the block pair is searched by expanding in the time axis direction. Adopt the method to do. In particular, by expanding the search for the pixel block 22 in the second region in the time direction, the motion search in the second region with a reduced number of pixels can be performed with high accuracy. Here, for simplicity of explanation, it is assumed that n = 1, k = −2, −1, 2.

First, the pixel block 21 in the first area and the pixel block 22 in the second area are extracted from the m-th frame 1 as in the sixth embodiment, and then the pixel block 22 in the second area is extracted from the image data in the (m + 1) -th frame 2. Extract a third block of the same size and shape.
Next, a vector connecting the pixel block 22 and the third block in the second region is obtained as a third motion vector. Assuming that the third motion vector is F, the motion vector obtained by scaling F to the vector between the (m + 2) th frame 4 and the mth frame 1 is 2F.
Next, the fourth block 17 that is the movement destination of the pixel block 22 is extracted from the (m + 2) th frame according to the motion vector 2F.
Next, the fifth block 18 having the same size and shape as the pixel block 22 in the second area is extracted from the image data of the (m−1) th frame 2.
Next, a vector connecting the pixel block 22 and the fifth block in the second area is obtained as a fourth motion vector. When the fourth motion vector is G, a motion vector obtained by scaling G to a vector between the m-2th frame and the mth frame 1 is 2G.
Next, the sixth block 19 that is the movement destination of the pixel block 22 is extracted from the (m-2) th frame according to the motion vector 2G.
Next, an absolute difference value for each pixel corresponding to the pixel block 22 in the second region and the third, fourth, fifth, and sixth blocks is obtained.
Next, each absolute difference value is compared with a predetermined threshold value, and the number of pixels that are equal to or smaller than the threshold value is counted to obtain a count value s (s is an integer equal to or larger than 0). Here, the count value of the pixel block 22 and the third is S3, the count value of the pixel block 22 and the fourth is S4, the count value of the pixel block 22 and the fifth is S5, and the count value of the pixel block 22 and the sixth is The count value is S6.
Next, the sum of the count value S3 and the count value S4 and the sum of the count value S5 and the count value S6 are compared, and the larger combination is selected. That is, when the sum of S3 and S4 is larger than the sum of S5 and S6, the third motion vector connecting the pixel block 22 and the third block is selected as the motion vector of the pixel block 22 in the second region.

(Tenth embodiment)
Next, as a tenth embodiment of the present invention, an embodiment in which the number of divisions is increased by repeatedly dividing a region in a block to obtain a plurality of motion vectors for pixel blocks in each region will be described with reference to FIG. . In this embodiment, the motion vector detection and the motion vector detection after area division can use any of the methods of the first embodiment to the ninth embodiment, and different points are repeated to increase the number of area divisions. It is in the point to do.

That is, after the pixel block in the second area, first, the i + 2 block having the same size and shape as the pixel block in the i + 1 area is extracted from the image data in the m + 1 frame for the pixel block in the i + 1 area (step S517). .
Next, the absolute difference value for each pixel corresponding to the pixel block in the i + 1 region and the i + 2 block is obtained (step S518).
Next, the absolute difference value obtained in step S518 is compared with a predetermined threshold value, and the number of pixels that are less than or equal to the threshold value is counted, and the count value s _{i + 1} (s _{i + 1} is an integer greater than or equal to 0). The subscript i + 1 is an area number) (step S519).
Next, a vector connecting the pixel block in the (i + 1) th region and the (i + 1) th block having the maximum count value s is selected as a motion vector in the (i + 1) th region (step S520).
Finally, an iterative continuation determination process (step S521) is performed. If it is determined to continue, the processes from step S517 to S520 are repeated again. If it is determined that the continuation is completed, the process proceeds to an end process.

  The processing in the repeated continuation determination processing step S521 may be any method. For example, when the number of pixels larger than the threshold value decreases (for example, 5% or less of the number of pixels in the first block). A method of terminating the continuation can be considered. Moreover, you may perform the process by iterative continuation determination process step S521 from an area map. For example, when the number of pixels larger than the threshold is small (for example, 10% or less of the number of pixels in the first block) and the spatial frequency of the region map is high, about one or two pixels are present on the region map. Since it is considered scattered, if more motion vector searches are performed, the number of errors increases. Therefore, the iteration is terminated.

(Eleventh embodiment)
Next, as an eleventh embodiment of the present invention, an m + k frame (k is an arbitrary real number) between an mth frame (m is an arbitrary integer) and an m + n frame (n is an integer of 1 or more) of the original image. An embodiment of an interpolation image creation method for creating an interpolation image to be interpolated at a temporal position will be described with reference to FIG. In this embodiment, a procedure as shown in FIG. 25 is used. Hereinafter, the processing procedure of this embodiment will be described with reference to FIGS. 24 and 25. Here, for simplification of explanation, n = 1 and k = 0.5, and an interpolated image is created in the middle of the mth frame 1 and the m + 1th frame 2, that is, the position of the m + 0.5th frame 10.

  The method for detecting the pixel block in the first region may be the same as that in the fifth embodiment, and any one of the methods in the first to fourth embodiments may be used as the motion vector detection method. That is, the extraction of the pixel block 21 of the first area in the first block 11 and the detection of the motion vector E of the first area are performed using steps S501 to S506 in FIG.

Next, the motion vector E of the first region is scale-converted according to the temporal position of the m + 0.5 frame 10 (step S522). In this case, the motion vector after the scale conversion is E / 2.
Next, from the (m + 1) th frame 2, the first on the (m + 0.5) th frame 10 that is in the same spatial position as the first block 11 on the mth frame 1 according to the motion vector E / 2 after the scale conversion in step S522. The fourth block 14 that is the movement destination of the interpolation block 25 in the region is extracted (step S523).
Next, the fourth block 14 is assigned to the interpolation block 25 in the first region on the m + 0.5 frame 10 (step S524).
Next, extraction of the pixel block 22 in the second area from the first block 11 and detection of the motion vector F in the second area using the same method as in the fifth embodiment, that is, steps S507 to S511 in FIG. I do.
Next, the motion vector F in the second region is scale-converted according to the temporal position of the m + 0.5 frame 10 (step S525). In this case, the motion vector after the scale conversion is F / 2.
Next, from the (m + 1) th frame 2, on the (m + 0.5) th frame 10 at the same spatial position as the pixel block 21 in the second area on the mth frame 1 according to the motion vector F / 2 after the scale conversion in step S525. The fifth block 15 that is the movement destination of the interpolation block 26 in the second area is extracted (step S526).
Finally, the fifth block 15 is assigned to the interpolation block 25 in the second area of the (m + 0.5) th frame 10 (step S527).

  Next, FIG. 26 shows a configuration of an interpolated image creating apparatus that performs the above-described interpolated image creating process according to the present embodiment. However, only the part added after the motion vector of the 1st area | region and 2nd area | region according to 5th Embodiment is detected is described here. The motion vector signal 54A of the first area is input to the motion vector scale converter 61A of the first area to scale the motion vector, and the scaled motion vector is used to convert the motion vector from the m + 1 frame memory 32 to the first area. The interpolation block 62A is extracted and output to the interpolation frame generation unit 63.

  Similarly, the motion vector signal 54B of the second area is input to the motion vector scale converter 61B of the second area to scale the motion vector, and the scaled motion vector is used to convert the motion vector from the m + 1 frame to the second area. Are extracted and output to the interpolation frame generation unit 63. The interpolation frame generation unit 63 assigns the interpolation block to the interpolation frame and creates image data of the interpolation frame.

(Twelfth embodiment)
Finally, as a twelfth embodiment of the present invention, an image display system using the motion vector detection method described in each of the previous embodiments and the method of creating an interpolated image according to the eleventh embodiment based thereon will be described. To do.

  FIG. 27 shows a schematic configuration of the image display system, and an input image signal 101 is input to the interpolated frame image creation unit 102 and the image switching unit 104. In the interpolated frame image creating unit 102, the interpolated image signal 103 described in the eleventh embodiment is created by the procedure described so far, and the interpolated image signal 103 is output to the image switching unit 104. The image switching unit 104 controls whether the input image signal 101 is output as it is or whether the interpolated image signal 104 is output. An output image signal 105 from the image switching unit 104 is output to a high-speed refresh display device 106 that is a hold-type display device. The display device 106 displays an image at a different refresh rate in response to the synchronization signal included in the output image signal 105.

The figure explaining the motion vector detection method which concerns on the 1st Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment The figure which shows the creation image output result and the effect of the same embodiment The figure which shows the example of the image from which an incorrect motion vector is detected The figure explaining the determination method of the block detected by the method of the embodiment The block diagram which shows the structure of the motion vector detection apparatus which concerns on the same embodiment The figure explaining the motion vector detection method which concerns on the 2nd Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. The block diagram which shows the structure of the motion vector detection apparatus which concerns on the same embodiment The figure explaining the motion vector detection method which concerns on the 3rd Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. The figure explaining the motion vector detection method which concerns on the 4th Embodiment of this invention. A flowchart showing a motion vector detection procedure according to the embodiment. The block diagram which shows the structure of the motion vector detection apparatus which concerns on the same embodiment The figure explaining the area division | segmentation and motion vector detection method which concern on the 5th Embodiment of this invention. Flowchart showing a region division and motion vector detection procedure according to the embodiment Diagram explaining the cause of block distortion FIG. 3 is a block diagram showing a configuration of a region division and motion vector detection device according to the embodiment. The figure explaining the area division | segmentation and motion vector detection method which concern on the 6th Embodiment of this invention. The flowchart which shows the area division | segmentation and motion vector detection procedure based on the 7th Embodiment of this invention. The figure explaining the area division | segmentation and motion vector detection method which concern on the 8th Embodiment of this invention. The figure explaining the area division | segmentation and motion vector detection method which concern on the 9th Embodiment of this invention. The flowchart which shows the area | region division and motion vector detection procedure which concern on the 10th Embodiment of this invention. The figure explaining the area division | segmentation and motion vector detection method which concern on the 11th Embodiment of this invention. Flowchart showing a region division and motion vector detection procedure according to the embodiment The block diagram which shows the structure of the interpolation image production apparatus which concerns on the same embodiment The block diagram which shows the structure of the image display system which concerns on the 12th Embodiment of this invention. The figure explaining the blurring phenomenon in the conventional hold type display device

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... mth frame 2 ... m + n frame 4 ... m + k frame 11-17 ... 1st-7th block 21 ... 1st area pixel block 22 ... 2nd area pixel block 23 ... 2nd space expansion area pixel Block 24 ... Pixel block in the third space expansion area 25 ... Interpolation block in the first area 26 ... Interpolation block in the second area

Claims (10)

Extracting a plurality of first blocks having a predetermined size and a predetermined shape obtained by dividing the m-th frame from an m-th frame (m is an arbitrary integer) of an image composed of a plurality of pixels;
Extracting a plurality of second blocks having the same size and shape as the first block from m + n frames (n is an integer of 1 or more) of the image;
Obtaining an absolute difference value for each corresponding pixel between the first block and the second block;
Counting the number of pixels where the absolute difference value is less than or equal to a threshold value to obtain a count value;
Extracting block pairs each including a pixel having the maximum count value from the first block and the second block, and selecting a vector between the block pairs as a motion vector between the m-th frame and the m + n frame; A motion vector detection method comprising: Extracting a plurality of first blocks having a predetermined size and a predetermined shape obtained by dividing the m-th frame from an m-th frame (m is an arbitrary integer) of an image composed of a plurality of pixels;
Extracting a plurality of second blocks having the same size and shape as the first block from m + n frames (n is an integer of 1 or more) of the image;
Obtaining a first absolute difference value for each corresponding pixel between the first block and the second block;
Counting the number of pixels in which the first absolute difference value is equal to or less than a first threshold value to obtain a first count value;
Extracting a block pair having the maximum first count value from the first block and the second block, and obtaining a vector between the block pairs as a first motion vector candidate;
Extracting a plurality of third blocks having a predetermined size and a predetermined shape obtained by dividing the m + n frame from the m + n frame;
Extracting a fourth block having the same size and shape as the third block from the m-th frame;
Obtaining a second absolute difference value for each corresponding pixel between the third block and the fourth block;
Counting the number of pixels for which the second absolute difference value is equal to or less than a second threshold value to obtain a second count value;
Extracting block pairs each including a pixel having the maximum second count value from the third block and the fourth block, and obtaining a vector between the block pairs as a second motion vector candidate;
The first count value and the second count value are compared. When the first count value is greater than or equal to the second count value, the first motion vector candidate, and when the first count value is smaller than the second count value, Selecting a second motion vector candidate as a motion vector between the m-th frame and the (m + n) -th frame, respectively. The m + k frame is divided from the m + k frame assumed between the m-th frame (m is an arbitrary integer) and the m + n frame (n is an integer not less than k + 1 and k is an arbitrary real number) of an image composed of a plurality of pixels. Extracting a plurality of fifth blocks having a predetermined size and a predetermined shape;
Extracting a plurality of sixth blocks having the same size and shape as the fifth block from the m-th frame;
Obtaining a vector between the fifth block and the sixth block as a third motion vector;
Calculating a fourth motion vector that is − (n−k) / k of the third motion vector;
Extracting a seventh block as a movement destination of the fifth block from the m + n frame according to the fourth motion vector;
Obtaining an absolute difference value for each corresponding pixel between the sixth block and the seventh block;
Counting the number of pixels where the absolute difference value is less than or equal to a threshold value to obtain a count value;
Extracting block pairs each including a pixel having the maximum count value from the sixth block and the seventh block, and selecting a vector between the block pairs as a motion vector between the m-th frame and the m + n frame; A motion vector detection method comprising: Extracting a plurality of fifth blocks having a predetermined size and a predetermined shape obtained by dividing the m-th frame from an m-th frame (m is an arbitrary integer) of an image composed of a plurality of pixels;
Extracting a plurality of sixth blocks having the same size and shape as the fifth block from the m + n frame of the image (n is an integer equal to or greater than k + 1, k is an integer );
Obtaining a vector between the fifth block and the sixth block as a first motion vector;
Obtaining a first difference absolute value for each corresponding pixel between the fifth block and the sixth block;
Counting the number of pixels in which the first absolute difference value is equal to or less than a first threshold value to obtain a first count value;
Extracting a seventh block having the same size and the same spatial position as the fifth block from the m + k frame of the image;
Calculating a second motion vector that is (n−k) / n times the first motion vector;
Extracting an eighth block to be a destination of the seventh block from the m + n frame according to the second motion vector;
Obtaining a second difference absolute value for each corresponding pixel between the seventh block and the eighth block;
Counting the number of pixels for which the second absolute difference value is equal to or less than a second threshold value to obtain a second count value;
Obtaining a weighted addition value represented by r (r = x × p + (1−x) q; x is a real number from 0 to 1), where the first and second count values are p and q, respectively;
Extracting block pairs each including a pixel having the maximum weighted addition value from the fifth block and the sixth block, and selecting a vector between the block pairs as a motion vector between the m-th frame and the m + n frame. A motion vector detection method comprising: Extracting a plurality of first blocks having a predetermined size and a predetermined shape obtained by dividing the m-th frame from an m-th frame (m is an arbitrary integer) of an image composed of a plurality of pixels;
Extracting a second block having the same size and shape as the first block from m + n frames (n is an integer of 1 or more) of the image;
Obtaining a first absolute difference value for each corresponding pixel between the first block and the second block;
Counting the number of pixels in which the first absolute difference value is equal to or less than a first threshold value to obtain a first count value;
A block pair including each pixel having the maximum first count value is extracted from the first block and the second block, and a vector between the block pair is determined between the first area in the first block and the m + n frame. Selecting as a first region-specific motion vector,
Extracting the pixels in the first block whose first difference absolute value is equal to or less than a second threshold as pixel blocks in the first region;
Extracting a pixel in the first block in which the first absolute difference value is larger than the second threshold as a pixel block of a second region;
Extracting a third block having the same size and shape as the pixel block of the second region from the m + n frame;
Obtaining a second absolute difference value for each corresponding pixel between the pixel block of the second region and the third block;
Counting the number of pixels for which the second absolute difference value is equal to or less than a third threshold to obtain a second count value;
A block pair including the pixel having the maximum second count value is extracted from the pixel block and the third block in the second area, and a vector between the block pair is defined between the second area and the m + n frame. Selecting a motion vector as a second region-specific motion vector. Extracting a plurality of first blocks having luminance information and color difference information obtained by dividing the m-th frame from an m-th frame (m is an arbitrary integer) of an image including a plurality of pixels;
Extracting a second block having luminance information and color difference information having the same size and shape as the first block from the m + n frame (n is an integer of 1 or more) of the image;
Obtaining a first absolute difference value for each corresponding pixel between a first luminance block having luminance information of the first block and a second luminance block having luminance information of the second block;
Second pixels having color difference information of the first color difference block and the second block having color difference information of the first block for pixels whose first difference absolute value of the first block and second block is equal to or less than a first threshold value. Obtaining a second absolute difference value for each corresponding pixel between color difference blocks;
Counting the number of pixels for which the second absolute difference value is equal to or less than a second threshold value to obtain a first count value;
A block pair including each pixel having the maximum first count value is extracted from the first block and the second block, and a vector between the block pair is determined between the first area in the first block and the m + n frame. Selecting as a first region-specific motion vector,
Extracting a pixel in the first block in which the second difference absolute value is equal to or less than the second threshold as a pixel block of the first region;
Extracting the pixels in the first block where the second absolute difference value is greater than a second threshold as pixel blocks in a second region;
Extracting a third block having the same size and shape as the pixel block of the second region from the m + n frame;
Obtaining a third difference absolute value for the brightness of each corresponding pixel between the pixel block of the second region and the third block;
Counting the number of pixels for which the third absolute difference value is equal to or less than a third threshold value to obtain a second count value;
A block pair including the pixel having the maximum second count value is extracted from the pixel block and the third block in the second area, and a vector between the block pair is defined between the second area and the m + n frame. Selecting a motion vector as a second region-specific motion vector. Extracting a plurality of first blocks having a predetermined size and a predetermined shape obtained by dividing the m-th frame from an m-th frame (m is an arbitrary integer) of an image composed of a plurality of pixels;
Extracting a second block having the same size and shape as the first block from m + n frames (n is an integer of 1 or more) of the image;
Obtaining a first absolute difference value for each corresponding pixel between the first block and the second block;
Counting the number of pixels in which the first absolute difference value is equal to or less than a first threshold value to obtain a first count value;
A block pair including each pixel having the maximum first count value is extracted from the first block and the second block, and a vector between the block pair is determined between the first area in the first block and the m + n frame. Selecting as a first region-specific motion vector,
Extracting the pixels in the first block whose first difference absolute value is equal to or less than a second threshold as pixel blocks in the first region;
Extracting a pixel in the first block in which the first absolute difference value is larger than the second threshold as a pixel block of a second region;
Extracting, from the m + n frame, a pixel block of a second spatial expansion region having the same size and shape as a first spatial expansion region obtained by spatially connecting the adjacent second regions in the first block;
Obtaining a second difference absolute value for each corresponding pixel between the pixel block of the first spatial extension region and the pixel block of the second spatial extension region;
Counting the number of pixels for which the second absolute difference value is equal to or less than a third threshold to obtain a second count value;
Pixel block pairs each including a pixel having the maximum second count value are extracted from the first space extension region and the second space extension region, and a vector between the pixel block pairs is extracted from the first space extension region and the second space extension region. selecting as a second region-specific motion vector between m + n frames. Extracting a plurality of first blocks having a predetermined size and a predetermined shape obtained by dividing the m-th frame from an m-th frame (m is an arbitrary integer) of an image composed of a plurality of pixels;
Extracting a second block having the same size and shape as the first block from the m + n frames of the image (n is an integer of 1 or more or -1 or less);
Obtaining a first absolute difference value for each corresponding pixel between the first block and the second block;
Counting the number of pixels in which the first absolute difference value is equal to or less than a first threshold value to obtain a first count value;
A block pair including each pixel having the maximum first count value is extracted from the first block and the second block, and a vector between the block pair is determined between the first area in the first block and the m + n frame. Selecting as a first region-specific motion vector,
Extracting the pixels in the first block whose first difference absolute value is equal to or less than a second threshold as pixel blocks in the first region;
Extracting a pixel in the first block in which the first absolute difference value is larger than the second threshold as a pixel block of a second region;
Extracting a third block having the same size and shape as the second region from the m + n frame;
Obtaining a second absolute difference value for each corresponding pixel between the pixel block of the second region and the third block;
Counting the number of pixels for which the second absolute difference value is less than or equal to a second threshold value to obtain a second count value;
Obtaining a vector between the pixel block of the second region and the third block as a third motion vector;
Calculating a fourth motion vector that is k / n of the third motion vector (k is an integer other than n and 0);
Extracting a fourth block, which is a movement destination of the pixel block of the second area, from the m + k frame according to the fourth motion vector;
Obtaining a third absolute difference value for each corresponding pixel between the pixel block of the second region and the fourth block;
Counting the number of pixels for which the third difference absolute value is less than or equal to a second threshold value to obtain a third count value;
A block pair including pixels each having the maximum sum of the second counter value and the third count value is extracted, and a vector between the block pairs is a motion vector for each second area between the second area and the m + n frame. A motion vector detection method comprising the steps of: An interpolation image to be interpolated is created at a temporal position of the m + k frame (k is an arbitrary real number) between the mth frame (m is an arbitrary integer) and the m + n frame (n is an integer of 1 or more) of the original image. In the frame interpolation image creation method,
The motion vector detection method according to any one of claims 5 to 9, and the motion vector between the m-th frame and the m + n frame obtained by the motion vector detection method according to any one of claims 1 to 4. Scaling the region-specific motion vector determined in accordance with the temporal position of the m + k frame;
According to the motion vector after the scale conversion from the m + n frame, the interpolation block on the m + k frame and the pixel block after the area division on the m frame, which are in the same spatial position as the block on the m frame. Extracting a fifth block that is a movement destination of the inter-region interpolation block on the m + k frame at the same spatial position;
A frame interpolation image creating method comprising: creating the interpolation image by assigning the fifth block to the interpolation block and the inter-region interpolation block of the m + k frame.   An image display system for displaying the interpolated image and the original image created by the frame interpolated image creating method according to claim 9.

Images