CN113726980B - Image processing methods - Google Patents

Abstract

An image processing method, comprising: receiving first and second pictures; generating a plurality of motion vectors according to the first picture and the second picture, wherein one motion vector corresponds to a first block of the first picture and a second block of the second picture; calculating a compensation position of the compensation block, a first motion vector between the compensation block and the first block, and a second motion vector between the compensation block and the second block; determining whether the sum of the compensation position plus the compensation value plus the first or second motion vector exceeds the target range to set the mixing coefficient; adjusting the first data of the first block or the second data of the second block according to the mixing coefficient; and generating the complementary frame data according to the adjusted first data or the second data.

Description

Image processing methods

Technical Field

The present invention relates to an image processing method, and more particularly to an image processing method for improving image discontinuity.

Background technique

In the field of image processing, when using motion estimation motion compensation (MEMC) technology to process low frame rate images, some situations may cause the image to be discontinuous in space or time. For example, when the image moves quickly, the motion vector points to the edge (out of boundary) or the black edge (out of rim). Or, when the image has a covered (cover) or uncovered (uncover) area. Or when using the bilateral interpolation method to calculate, one side cannot obtain data (invalid by search range) or is invalid (invalid by boundary or rim).

Therefore, how to solve the phenomenon of picture discontinuity is an important issue in this field.

Summary of the invention

The present invention application relates to an image processing method, which includes: receiving a first picture and a second picture; generating multiple motion vectors according to the first picture and the second picture, wherein one motion vector corresponds to a first block of the first picture and a second block of the second picture; calculating a compensation position of a compensation block, a first motion vector between the compensation block and the first block, and a second motion vector between the compensation block and the second block; determining whether the sum of the compensation position plus the compensation value plus the first motion vector or the second motion vector exceeds a target range to set a mixing coefficient; adjusting the first data of the first block or the second data of the second block according to the mixing coefficient; and generating supplementary frame data according to the adjusted first data or second data.

In summary, the motion frame rate converter of the image processing device calculates, determines and adjusts the mixing ratio of two blocks for interpolation according to the image processing method, so as to reduce the phenomenon of picture discontinuity.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, operations and effects of the present invention are described in detail below with reference to the accompanying drawings for preferred embodiments.

FIG. 1 is a schematic diagram of an image processing device according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of continuous frames contained in an image according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a frame interpolation image according to an embodiment of the present invention.

FIG. 4 is a flow chart of an image processing method according to an embodiment of the present invention.

FIG. 5 is a functional block diagram of a motion compensator according to an embodiment of the present invention.

FIG. 6 and FIG. 7 are schematic diagrams of another type of interpolated frame images according to an embodiment of the present invention.

Symbol Description:

100: Image processing device

110: Buffer

120: Motion Evaluation Processor

130: Motion Frame Rate Converter

140: Edge Detector

150: Cover/reveal detector

160: Sign detector

Vin: video input signal

MV: Motion Vector

RIM: Edge

CR: Coverage/Revealing Area

Vout: video output signal

400: Image Processing Methods

S410, S420, S430, S440, S450, S460: Operation

Fk-1, Fk, Fc, Fc1~Fc4: Frame

MV1, MV2, MVk, MV1[x1,y1], MV1[x2,y2], MV2[x1,y1], MV2[x2,y2], MV1[i,j], MV2[i,j], MV1a, MV1b, MV2a, MV2b: motion vector

BL1a, BL1b, BL1c, BL2a, BL2b, BL2c, BL3a, BL3b, BL3c, BL4a, BL4b, BL4c, B11, B12, B21, B22, Bc1, Bc2: Block

Pc[x1,y1], Pc[x2,y2], P1[x1,y1], P1[x2,y2], P2[x1,y1], P2[x2,y2], P[i,j]: position

C[i,j]: compensation value

IND[i,j]: mixing coefficient

Detailed ways

The following is a detailed description of the embodiments with the help of the accompanying drawings, but the specific embodiments described are only used to explain the present invention and are not used to limit the present invention. The description of the structural operation is not used to limit the execution order. Any structure recombined by the components to produce a device with equal functions is within the scope of the present invention application.

Please refer to Figure 1. Figure 1 is a schematic diagram of an image processing device 100 according to an embodiment of the present invention. As shown in Figure 1, the image processing device 100 includes a buffer 110, a motion estimation processor 120, a motion compensation frame rate converter 130, an edge detector 140 (boundary detector), a coverage/display detector 150 (de-halo detector or cover/uncover detector) and a logo detector 160 (logo detector). Structurally, the buffer 110 is connected to the motion estimation processor 120 and the motion frame rate converter 130. The edge detector 140 is connected to the motion estimation processor 120 and the motion frame rate converter 130. The coverage/display detector 150 and the logo detector 160 are respectively connected to the motion frame rate converter 130.

In operation, the buffer 110 is used to receive the image input signal Vin, perform buffering, and then transmit it to the motion estimation processor 120 and/or the motion frame rate converter 130. The edge detector 140 is used to receive and obtain the four edges RIM of the input image at the top, bottom, left, and right according to the image input signal Vin, and use the area surrounded by the four edges RIM as the screen area. Specifically, the edge detector 140 can obtain the four edges according to the image size of the image input signal Vin. In addition, in some embodiments, the edge detector 140 can detect the black rim portion in the image input signal Vin, and use the area after deducting the black rim portion as the screen area.

The cover/uncover detector 150 is used to receive and obtain a cover/uncover region CR between two frames of an input image according to the image input signal Vin. The logo detector 160 is used to obtain a logo, such as a channel logo, in the input image according to the image input signal Vin.

The motion estimation processor 120 is used for receiving and performing motion estimation (ME) according to the image input signal Vin and the edge RIM, and transmitting the generated motion vector to the motion frame rate converter 130. The motion frame rate converter 130 is used for receiving and performing frame interpolation compensation according to the image input signal Vin, the edge RIM and the cover/visible area, and outputting the image output signal Vout after frame rate conversion.

For example, as shown in FIG. 2 , the frame Fk is the current frame in the image input signal Vin received by the image processing device 100. The frame Fk-1 is a reference frame corresponding to the current frame in the image input signal Vin. In some embodiments, the frame Fk and the frame Fk-1 are two adjacent frames. The motion estimation processor 120 cuts the frame Fk-1 and the frame Fk into a plurality of blocks in units of i times j, and uses various motion vector search methods to find the best motion vector of each block and transmits it to the motion frame rate converter 130.

Afterwards, the motion frame rate converter 130 generates a compensation picture between the frame Fk-1 and the frame Fk according to the frame Fk-1, the frame Fk and the best motion vector of each block. For example, if the best motion vector is approximately a vector from the lower left to the upper right, then according to the circle located at the lower left corner of the frame Fk-1 and the circle located at the upper right corner of the frame Fk, a compensation picture as shown in the frames Fc1 to Fc4 can be generated. In other words, the image output signal Vout output by the image processing device 100 using the image processing method will include the frame Fk-1, the frames Fc1 to Fc4 and the frame Fk.

However, when the motion vector used to generate the compensation picture points to the outside of the boundary or the outside of the black edge, the reference data required for interpolation compensation cannot be obtained or invalid data is obtained. This situation may cause the content of the compensation picture to be discontinuous, or the compensation picture and the adjacent picture to be discontinuous in time. For example, as shown in FIG3, the frame Fc is a compensation picture generated according to the current frame Fk and the reference frame Fk-1. In theory, the block BL1c of the frame Fc is obtained by weighted averaging the blocks BL1a and BL1b corresponding to the motion vector MV1 with a weight of 0.5 each. However, if the block BL2c is taken as an example, the block BL2a and the block BL2b must be referenced when calculating the motion vector MV2, but the block BL2a has exceeded the edge of the image, so the block BL2a cannot be referenced. If the block BL2a that cannot be referenced and the block BL2b pointed to by the motion vector MV2b are weighted averaged, the obtained compensation block BL2c is likely to be discontinuous. In view of the above problem, the present invention proposes the following image processing method to solve this problem.

Please refer to FIG. 4. FIG. 4 is a flow chart of an image processing method 400 according to an embodiment of the present invention. For the sake of convenience and clarity of description, the following image processing method 400 will be described in conjunction with the embodiments shown in FIG. 1 to FIG. 7, but it is not limited thereto. Any person skilled in the art can make various changes and modifications to it without departing from the spirit and scope of the present invention. As shown in FIG. 4, the image processing method 400 includes operations S410, S420, S430, S440, S450 and S460.

First, in operation S410, a first picture and a second picture are received. Specifically, as shown in FIG1 , the motion estimation processor 120 in the image processing device 100 receives two different frames of the image input signal Vin. In some embodiments, the first picture and the second picture may be two adjacent frames, such as the current frame Fk and the reference frame Fk-1 in FIG2 .

Next, in operation S420, a plurality of motion vectors are generated according to the first picture and the second picture, wherein one motion vector corresponds to the first block of the first picture and the second block of the second picture. Specifically, the motion evaluation processor 120 divides the current frame Fk into a plurality of current blocks of fixed size. For the sake of convenience of explanation, the following paragraphs will represent these current blocks as a matrix of i times j. Thereafter, the motion evaluation processor 120 searches for the corresponding reference block with the highest matching degree in the reference frame Fk-1 for each current block, and uses the vector between the current block and the corresponding reference block as the motion vector of the current block. In some embodiments, the calculation of the matching degree can be implemented using the sum of absolute differences (SAD), but the present invention is not limited thereto. In other embodiments, mean square error (MSE) or mean absolute deviation (MAD) can also be used.

In this way, the motion estimation processor 120 can calculate i times j motion vectors corresponding to i times j current blocks according to the frame Fk and the frame Fk-1.

Next, in operation S430, the compensation position of the compensation block, the first motion vector between the compensation block and the first block, and the second motion vector between the compensation block and the second block are calculated according to the motion vector, the first block, and the second block. Specifically, taking the motion vector MVk, the block B11, and the block B21 in FIG. 5 as an example, the motion frame rate converter 130 can calculate the compensation position of the compensation block Bc1 according to the motion vector MVk, the block BL11, and the block BL21. As shown in FIG. 5, the compensation position Pc[x1, y1] of the compensation block Bc1 is the midpoint of the position P1[x1, y1] of the block B11 and the position P2[x1, y1] of the block B21. In addition, the motion frame rate converter 130 can use the vector MV1[x1, y1] between the compensation position Pc[x1, y1] of the compensation block Bc1 and the position P1[x1, y1] of the block B11 as the first vector, and use the vector MV2[x1, y1] between the position Pc[x1, y1] of the compensation block Bc1 and the position P2[x1, y1] of the block B21 as the second vector.

In this way, the motion estimation processor 120 can calculate the compensation positions of i times j compensation blocks (such as position P[i,j] in Figure 6), i times j first motion vectors (such as MV1[i,j] in Figure 6), and i times j second motion vectors (such as MV2[i,j] in Figure 6) based on the positions of i times j current blocks and the positions of i times j reference blocks corresponding to the current blocks.

Next, in operation S440, it is determined whether the sum of the compensation position of the compensation block plus the compensation value plus the first motion vector or the second motion vector exceeds the target range to set the blending coefficient. Specifically, as shown in FIG6 , the motion frame rate converter 130 calculates the positions of the first block and the second block corresponding to the compensation block according to the compensation position P[i,j], the first motion vector MV1[i,j], the second motion vector MV2[i,j] and the compensation value C[i,j] of the compensation block, and determines whether the first block or the second block (i.e., the current block or the reference block) exceeds the target range (i.e., the area surrounded by four edge RIMs) according to the edge RIM. When the first block or the second block exceeds the target range, the blending coefficient IND[i,j] is set according to the size of the exceeding portion. When the first block or the second block does not exceed the target range, the blending coefficient IND[i,j] of the first block or the second block is marked as zero or discarded (fail), indicating that the block does not need to be processed additionally.

For example, if the compensation position of the compensation block plus the first motion vector and the compensation value do not exceed the target range, the mixing coefficient of the first block is set to zero. When the compensation position of the compensation block plus the second motion vector and the compensation value do not exceed the target range, the mixing coefficient of the second block is set to zero. On the other hand, if the compensation position of the compensation block plus the first motion vector and the compensation value exceed the target range, the mixing coefficient of the first block is set to an interpolated value of the two. When the compensation position of the compensation block plus the second motion vector and the compensation value exceed the target range, the mixing coefficient of the second block is set to an interpolated value of the two.

For further details, as shown in the following formula (1), Px is the horizontal coordinate value of the compensation position of the compensation block, and Py is the vertical coordinate value of the compensation position of the compensation block. MVx is the horizontal coordinate value of the first motion vector or the second motion vector, and MVy is the vertical coordinate value of the first motion vector or the second motion vector. OFFSET is the compensation value. RIMtop, RIMbottom, RIMleft, and RIMright are the top, bottom, left, and right borders of the input image, respectively, which are defined by the multiple coordinate values on the four borders. IND[x,y] is the mixing coefficient of the first block or the second block.

IND[x,y]＝OFFSET

Else IND[x,y]＝0 Formula (1)

As shown in FIG5 , since the coordinate value corresponding to the position Pc[x1,y1] of the compensation block Bc1 plus the first motion vector MV1[x1,y1] and the compensation value will fall outside the top frame RIMtop, the blending coefficient of the first block B11 is marked as the compensation value OFFSET according to formula (1). Since the coordinate value corresponding to the position Pc[x1,y1] of the compensation block Bc1 plus the second motion vector MV2[x1,y1] and the compensation value will fall within the input image defined by the four frames, the blending coefficient of the second block B21 is set to zero according to formula (1).

For another example, as shown in FIG5 , since the coordinate value corresponding to the position Pc[x2,y2] of the compensation block Bc2 plus the first motion vector MV1[x2,y2] and the compensation value will fall within the input image defined by the four borders, the blending coefficient of the first block B12 is marked as zero according to formula (1). Since the coordinate value corresponding to the position Pc[x2,y2] of the compensation block Bc2 plus the second motion vector MV2[x2,y2] and the compensation value will fall outside the bottom border RIMbottom, the blending coefficient of the second block B22 is marked as the compensation value OFFSET according to formula (1).

It is worth noting that the offset value OFFSET is the distance range to be processed, which can be set according to the needs of the user. In addition, in some embodiments, the offset value OFFSET can be multiplied by a gain. The gain can be controlled by the firmware. In other words, the offset value OFFSET is adjustable.

Next, in operation S450, the first data of the first block or the second data of the second block is adjusted according to the mixing coefficient. Specifically, as shown in FIG6, the motion frame rate converter 130 adjusts the data of each block included in the frame Fk and the frame Fk-1 according to the mixing coefficient IND[i,j].

Further, when one of the two mixing coefficients of the first block and the second block is zero and the other has a non-zero value, the block whose value is non-zero is adjusted to the block whose value is zero. For example, as shown in the following formula (2), INDa[x,y] is the mixing coefficient of the first block, INDb[x,y] is the mixing coefficient of the second block. DATAa[x,y] is the first data of the first block, and DATAb[x,y] is the second data of the second block. R is a rounding bit.

If (INDa[x,y]==0AND INDb[x,y]!=0)

DATAb[x,y]＝(INDb[x,y]*DATAb[x,y]+(MODE-INDb[x,y])*INDa[x,y]+R)/MODE

Else If (INDb[x,y]==0AND INDa[x,y]!=0)

DATAa[x,y]＝(INDa[x,y]*DATAa[x,y]+(MODE-INDa[x,y])*INDb[x,y]+R)/MODE

Formula (2)

As shown in FIG5 , since the mixing coefficient INDa[x,y] of block B11 is not zero and the mixing coefficient INDb[x,y] of block B21 is zero, the data of block B21 and the data of block B11 are weighted averaged in the ratio of INDb[x,y] and (MODE-INDb[x,y]) to obtain the new data of block B11. On the other hand, since the mixing coefficient INDb[x,y] of block B22 is not zero and the mixing coefficient INDa[x,y] of block B12 is zero, the data of block B12 and the data of block B22 are weighted averaged in the ratio of INDa[x,y] and (MODE-INDa[x,y]) to obtain the new data of block B22.

In some embodiments, MODE is a value matched with the compensation value OFFSET, which may be 32 or 64, but the present invention is not limited thereto, and a person skilled in the art may design it according to actual needs.

Since in operation S440, it is determined whether the calculation result exceeds on any side according to each border RIM of the upper, lower, left and right sides, and the exceeding value is recorded as the ratio for adjusting the mixing coefficient in operation S450. In other words, the first mixing coefficient is positively correlated with the value of the first block exceeding the border RIM. The second mixing coefficient is positively correlated with the value of the second block exceeding the border RIM. Therefore, the block that originally performed single interpolation can maintain single interpolation. The block that originally performed bilateral interpolation (bi-interpolation) and still did not exceed the target range after adding this calculation can continue to use bilateral interpolation. There is another transition calculation between single interpolation and bilateral interpolation to reduce the phenomenon of picture discontinuity.

Next, in operation S460, the interpolation frame data is generated according to the adjusted first data or the second data. Specifically, as shown in FIG6 , the motion frame rate converter 130 generates the interpolation frame data by weighted averaging the adjusted first data of the first block and the second data of the second block with a weight of 0.5, but the present invention is not limited thereto, and a person skilled in the art can design it according to actual needs.

In this way, by calculating whether the position of the block used for interpolation exceeds the edge, it is determined whether this block may cause picture discontinuity. When bilateral interpolation is performed, if the data of one of the two picture blocks used for interpolation is invalid or cannot be obtained (that is, it may cause picture discontinuity), the data content of this block is adjusted by using the other block in a weighted manner. In other words, the blending ratio of the two blocks for interpolation is adjusted by the blending coefficient, so that the generated compensation block is more similar to the block with valid data, and the picture discontinuity caused by mixing inappropriate data is avoided.

In some other embodiments, in operation S440 of the image processing method 100, the target range may also be a coverage/visibility region CR. In the input image, the area where the pattern of the previous frame is covered in the subsequent frame is called the coverage area. On the other hand, the area where the pattern that did not appear in the previous frame appears in the subsequent frame is called the visibility region. Specifically, the motion frame rate converter 130 determines whether the first block or the second block (i.e., the current block or the reference block) exceeds the coverage/visibility region CR obtained by the coverage/visibility detector 150. When the first block or the second block exceeds the coverage/visibility region CR, the mixing coefficient IND[i,j] is set according to the size of the exceeding portion. When the first block or the second block does not exceed the coverage/visibility region CR, the mixing coefficient IND[i,j] of the first block or the second block is marked as zero or discarded, indicating that no additional processing is required for this block.

For example, as shown in FIG7 , the area covered by the balloon in the current frame Fk is the coverage area relative to the reference frame Fk-1. The block covered by the balloon in the current frame Fk relative to the reference frame Fk-1 is the display area. Taking the compensation block BL3c as an example, since the second block BL3b falls into the coverage/display area CR, the mixing coefficient of the second block BL3b is not zero, and the block data will be adjusted according to the mixing coefficient of the first block BL3a. In addition, taking the compensation block BL4c as an example, since the first block BL4a falls into the coverage/display area CR, the mixing coefficient of the second block BL3b is not zero, and the block data will be adjusted according to the mixing coefficient of the first block BL3a.

In this way, by calculating whether the position of the block used for interpolation falls within the coverage/display region CR, it is determined whether this block may cause picture discontinuity. When bilateral interpolation is performed, if both of the two picture blocks used for interpolation fall within the coverage/display region CR, the data content of the block located in the coverage region is adjusted in a weighted manner using the block located in the display region. In other words, the mixing ratio of the two blocks for interpolation is adjusted by the mixing coefficient, so that the generated compensation block is more similar to the block with valid data, thereby avoiding picture discontinuity caused by mixing inappropriate data.

In summary, the image discontinuity phenomenon can be reduced by the motion frame rate converter 130 of the image processing device 100 calculating, determining and adjusting the mixing ratio of two blocks for interpolation according to the image processing method 300. In addition, the image processing method 300 of the present invention does not need to use a hardware line buffer, so a large amount of additional cost will not be generated.

Although the embodiments of the present invention have been described above, these embodiments are not intended to limit the present invention. A person skilled in the art may make changes and adjustments to the technical features of the present invention based on the explicit or implicit contents of the present invention. However, all kinds of changes may fall within the scope of protection of the present invention. In other words, the scope of protection of the present invention shall be subject to the scope defined in the claims of the present invention application.

Claims (9)

1. An image processing method, characterized in that the image processing method comprises:

Receiving a first picture and a second picture;

Generating a plurality of motion vectors according to the first picture and the second picture, wherein one of the plurality of motion vectors corresponds to a first block of the first picture and a second block of the second picture;

calculating a compensation position of a compensation block, a first motion vector between the compensation block and the first block, and a second motion vector between the compensation block and the second block;

determining whether the sum of the compensation position plus a compensation value and the first motion vector or the second motion vector exceeds a target range to set a mixing coefficient;

adjusting the first data of the first block or the second data of the second block according to the mixing coefficient; and

Generating complementary frame data according to the adjusted first data or second data;

wherein setting the mixing coefficient comprises:

when the compensation position is added with the first motion vector and the compensation value does not exceed the target range, setting a first mixing coefficient of the first block as the compensation value;

when the compensation position is added with the second motion vector and the compensation value does not exceed the target range, setting a second mixing coefficient of the second block as the compensation value;

Marking the first mixing coefficient as discarded when the compensation position plus the first motion vector and the compensation value exceed the target range; and

And marking the second mixing coefficient as discarded when the compensation position plus the second motion vector and the compensation value exceed the target range.

2. The image processing method according to claim 1, wherein adjusting the first data or the second data according to the mixing coefficient comprises:

When a first mixing coefficient is marked as discarded and the second mixing coefficient is not zero, adjusting the first data with the second mixing coefficient according to the second data; and

When a first blending coefficient is not zero and the second blending coefficient is marked as discarded, the second data is adjusted with the first blending coefficient according to the first data.

3. The image processing method according to claim 1, wherein the first mixing coefficient is positively correlated with a value of the sum exceeding the target range, and the second mixing coefficient is positively correlated with a value of the sum exceeding the target range.

4. The image processing method according to claim 1, wherein the target range includes a picture area or an overlay presentation area.

5. The image processing method according to claim 4, wherein the image processing method further comprises:

Acquiring four edges of the input image by an edge detector; and

And taking the area surrounded by the four edges as the picture area.

6. The image processing method according to claim 4, wherein the image processing method further comprises:

The overlay appearance areas in the first and second frames are detected by an overlay appearance detector.

7. The image processing method of claim 1, wherein calculating the compensation position comprises:

and calculating the midpoint positions of the first block and the second block as the compensation positions.

8. The image processing method of claim 1, wherein the compensation value is adjustable.

9. The image processing method of claim 1, wherein generating the supplemental frame data comprises:

an arithmetic average of the first data and the second data is calculated as the frame complement data.