KR20200002036A - Optical Flow Estimation for Motion Compensated Prediction in Video Coding - Google Patents

Abstract

A portion of the light flow reference frame (eg, block or entire frame) is generated that can be used for inter prediction of blocks of the current frame in the video sequence. The forward reference frame and the reverse reference frame are used for the light flow estimation to generate respective motion fields for the pixels of the current frame. Motion fields are used to warn some or all of the pixels of the reference frames to the pixels of the current frame. The warped reference frame pixels are blended to form a lightflow reference frame portion. Inter prediction may be performed as part of the encoding or decoding portions of the current frame.

Description

Optical Flow Estimation for Motion Compensated Prediction in Video Coding

[0001] Digital video streams can represent video using a series of frames or still images. Digital video can be used in a variety of applications, including, for example, video conferencing, high resolution video entertainment, video advertisements, or sharing of user generated videos. Digital video streams may contain large amounts of data and may consume a significant amount of computing or communication resources of a computing device for processing, transmitting, or storing video data. Various approaches have been proposed, including compression and other encoding techniques, to reduce the amount of data in video streams.

[0002] One technique for compression uses a reference frame to generate a predictive block corresponding to the current block to be encoded. The differences between the prediction block and the current block may be encoded instead of the values of the current block itself to reduce the amount of encoded data.

[0003] FIELD OF THE INVENTION The present invention generally relates to encoding and decoding video data, and more particularly to the use of block-based optical flow estimation for motion compensation prediction in video compression. Frame level based light flow estimation is also described, which can interpolate collocated reference frames for motion compensated prediction in video compression.

[0004] The present invention describes encoding and decoding methods and apparatus. The method according to an embodiment of the present invention comprises the steps of determining a first frame portion of a first frame to be predicted, the first frame being in a video sequence, video for forward inter prediction of the first frame. Determining a first reference frame from the sequence, determining a second reference frame from the video sequence for reverse inter prediction of the first frame, light flow using the first reference frame and the second reference frame By performing optical flow estimation, generating a light flow reference frame portion for inter prediction of the first frame portion, and performing a prediction process for the first frame portion using the light flow reference frame portion. It includes. The first frame portion and the light flow reference frame portion may be, for example, blocks or entire frames.

[0005] An apparatus according to an embodiment of the present invention includes a non-transitory storage medium or memory and a processor. The medium includes instructions executable by a processor to perform a method, the method comprising determining a first frame to be predicted in a video sequence, and availability of the first reference frame for forward inter prediction of the first frame; Determining availability of a second reference frame for reverse inter prediction of the first frame. The method also uses, in response to determining the availability of both the first reference frame and the second reference frame, using the first reference frame and the second reference frame as inputs to the light flow estimation process, the first frame portion. Generating respective motion fields for the pixels of, warping the first reference frame portion to the first frame portion using the motion fields to form a first warped reference frame portion, the first reference frame The portion includes pixels of a first reference frame in parallel with the pixels of the first frame portion—using the motion fields to form a second reference frame portion to form a second warped reference frame portion. Warping to the second reference frame portion wherein the second reference frame portion is in parallel with the pixels of the first frame portion. Comprising pixels of the frame, and blending the first warped reference frame portion and the second warped reference frame portion to form a lightflow reference frame portion for inter prediction of at least one block of the first frame. It includes a step.

[0006] Another apparatus according to one embodiment of the invention also includes a non-transitory storage medium or memory and a processor. The medium includes instructions executable by a processor to perform a method, the method comprising: initializing motion fields for pixels of a first frame portion at a first processing level for light flow estimation, the first processing level being Indicating a downscaled motion within the first frame portion and including one of a plurality of levels—a first reference frame portion from the video sequence and a first reference frame portion of the video sequence Generating a lightflow reference frame portion for inter prediction of a block of frames, and for each of the plurality of levels, a first reference frame using motion fields to form a first warped reference frame portion; Warping the portion to the first frame portion, forming a second warped reference frame portion Warping the second reference frame portion to the first frame portion using motion fields to estimate motion fields between the first warped reference frame portion and the second warped reference frame portion using light flow estimation. And updating the motion fields for the pixels of the first frame portion using the motion fields between the first warped reference frame portion and the second warped reference frame portion. The method also includes: for the last of the plurality of levels: warping the first reference frame portion to the first frame portion using the updated motion fields to form the final first warped reference frame portion, the final Warping the second reference frame portion to the first frame portion using the updated motion fields to form a second warped reference frame portion of the second warped reference frame portion, and a final first warped to form the lightflow reference frame portion. Blending the reference frame portion and the second warped reference frame portion.

[0007] Another apparatus according to one embodiment of the invention also includes a non-transitory storage medium or memory and a processor. The medium includes instructions executable by a processor to perform a method, the method comprising: determining a first frame portion of a first frame to be predicted, wherein the first frame is in a video sequence; Determining a first reference frame from the video sequence for inter prediction, determining a second reference frame from the video sequence for reverse inter prediction of the first frame, using the first reference frame and the second reference frame Generating a light flow reference frame portion for inter prediction of the first frame portion by performing flow estimation, and performing a prediction process for the first frame portion using the light flow reference frame portion.

[0008] These and other aspects of the invention are disclosed in the following detailed description of the embodiments, the appended claims, and the accompanying drawings.

The description herein refers to the accompanying drawings described below, wherein like reference numerals refer to like parts throughout the various figures unless otherwise noted.
1 is a schematic diagram of a video encoding and decoding system.
2 is a block diagram of an example of a computing device that may implement a transmitting station or a receiving station.
3 is a diagram of a typical video stream to be encoded and subsequently decoded.
4 is a block diagram of an encoder in accordance with implementations of the invention.
5 is a block diagram of a decoder in accordance with implementations of the invention.
6 is a block diagram of an example of a reference frame buffer.
FIG. 7 is a diagram of a group of frames of a display order of a video sequence. FIG.
FIG. 8 is a diagram of an example of coding order for a group of frames of FIG. 7.
FIG. 9 is a diagram used to describe the linear projection of a motion field in accordance with the teachings herein.
10 is a flowchart of a process for motion compensated prediction of a video frame using at least a portion of a reference frame generated using light flow estimation.
11 is a flowchart of a process for generating a lightflow reference frame portion.
12 is a flowchart of another process for generating a lightflow reference frame portion.
FIG. 13 is a diagram illustrating the processes of FIGS. 11 and 12.
FIG. 14 is a diagram illustrating object occlusion. FIG.
FIG. 15 is a diagram illustrating a technique for optimizing a decoder. FIG.

[0025] The video stream may be compressed by various techniques to reduce the bandwidth required to transmit or store the video stream. The video stream can be encoded into a bitstream that involves compression, which is then sent to a decoder that can decode or decompress the video stream and prepare it for viewing or further processing. Compression of video streams often uses spatial and temporal correlation of video signals through spatial and / or motion compensated prediction. Inter-prediction uses one or more motion vectors to generate, for example, a block (also called a prediction block) similar to the current block to be encoded using previously encoded and decoded pixels. By encoding the motion vector (s) and the difference between the two blocks, the decoder receiving the encoded signal can regenerate the current block. Inter prediction may also be referred to as motion compensated prediction.

[0026] Each motion vector used to generate the predictive block in the inter prediction process may refer to a frame other than the current frame, that is, a reference frame. The reference frames may be located before or after the current frame in the sequence of the video stream, and may be frames reconstructed before being used as the reference frame. In some cases, there may be three reference frames used to encode or decode blocks of the current frame of the video sequence. One is a frame, which may be referred to as a golden frame. The other is the most recently encoded or decoded frame. The last is an alternate reference frame that is encoded or decoded before one or more frames in the sequence but displayed after these frames in the output display order. In this way, the replacement reference frame is a reference frame usable for backward prediction. One or more forward and / or backward reference frames may be used to encode or decode the block. When used to encode or decode a block within the current frame, the validity of the reference frame may be measured based on the resulting signal-to-noise ratio or other measurements of rate-distortion.

[0027] In this technique, the pixels that form the predictive blocks are obtained directly from one or more of the available reference frames. Reference pixel blocks or their linear combinations are used for prediction of a given coding block in the current frame. This direct block based prediction does not capture the true motion activity available from the reference frames. For this reason, motion compensation prediction accuracy may be degraded.

[0028] To more fully utilize the motion information from the available bidirectional reference frames (eg, one or more forward and one or more reverse reference frames), implementations of the teachings herein estimates true motion activities in a video signal. To illustrate the reference frame portions collocated with current coding frame portions using a per-pixel motion field calculated by the light flow. Reference frame portions are interpolated that allow tracking of complex non-translational motion activity beyond the ability of conventional block-based motion compensated prediction determined directly from reference frames. Use of such reference frame portions can improve prediction quality. As used herein, a frame portion refers to some or all of a frame, such as a block, slice, or entire frame. The frame portion of one frame is juxtaposed with the frame portion of another frame if this frame portion and the frame portion of another frame have the same dimensions and are at the same pixel positions within the dimensions of each frame.

[0029] Further details on using light flow estimation to interpolate reference frame portions for use in video compression and reconstruction are described herein with an initial reference to a system in which the teachings herein can be implemented.

[0030] 1 is a schematic diagram of a video encoding and decoding system 100. The transmitting station 102 may be, for example, a computer having an internal configuration of hardware as described in FIG. 2. However, other suitable implementations of the transmitting station 102 are possible. For example, the processing of the transmitting station 102 may be distributed among multiple devices.

[0031] The network 104 may connect the transmitting station 102 and the receiving station 106 for encoding and decoding of the video stream. In particular, the video stream may be encoded at the transmitting station 102 and the encoded video stream may be decoded at the receiving station 106. Network 104 may be, for example, the Internet. The network 104 may also be from a local area network (LAN), wide area network (WAN), virtual private network (VPN), cellular telephone network, or in this example from the transmitting station 102. It may be any other means for transmitting the video stream to the receiving station 106.

[0032] Receiving station 106 may be, in one example, a computer having an internal configuration of hardware as described in FIG. 2. However, other suitable implementations of the receiving station 106 are possible. For example, the processing of receiving station 106 may be distributed among multiple devices.

[0033] Other implementations of the video encoding and decoding system 100 are possible. For example, one implementation may omit the network 104. In another implementation, the video stream may be encoded and stored for later transmission to the receiving station 106 or any other device having a non-transitory storage medium or memory. In one implementation, the receiving station 106 receives the encoded video stream (eg, via the network 104, the computer bus, and / or a predetermined communication path) and stores the video stream for later decoding. . In an example implementation, Real Time Transmission Protocol (RTP) is used for the transmission of encoded video over network 104. In other implementations, a transport protocol other than RTP may be used, for example, a video streaming protocol based on Hypertext Transfer Protocol (HTTP).

[0034] When used in a video conferencing system, for example, the transmitting station 102 and / or the receiving station 106 may include the ability to encode and decode the video stream as described below. For example, the receiving station 106 receives, decodes and views the encoded video bitstream from the video conferencing server (eg, the transmitting station 102) and also encodes its own video bitstream to other participants. The video conferencing participant may be sent to a video conferencing server for decoding and viewing.

[0035] 2 is a block diagram of an example of a computing device 200 that may implement a transmitting station or a receiving station. For example, computing device 200 may implement either or both of transmit station 102 and receive station 106 of FIG. 1. Computing device 200 may be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, tablet computer, laptop computer, notebook computer, desktop computer, or the like.

[0036] CPU 202 in computing device 200 may be a central processing unit. Alternatively, the CPU 202 may be any other type of device or multiple devices capable of manipulating or processing information that is present or developed in the future. The disclosed embodiments may be implemented with one processor, for example, the CPU 202, as shown, but using two or more processors may benefit from speed and efficiency.

[0037] The memory 204 of the computing device 200 may be a read-only memory (ROM) device or a random access memory (RAM) device in an implementation. Any other suitable type of storage device or non-transitory storage medium may be used as the memory 204. Memory 204 may include code and data 206 that are accessed by CPU 202 using bus 212. The memory 204 may further include an operating system (OS) 208 and application programs 210, which may allow the CPU 202 to perform the methods described herein. It includes at least one program. For example, application programs 210 may include applications 1 through N, which further include a video coding application that performs the methods described herein. Computing device 200 may also include secondary storage 214, which may be, for example, a memory card used with a mobile computing device. Because video communication sessions can contain a significant amount of information, they can be stored in whole or in part in secondary storage 214 and loaded into memory 204 as needed for processing.

[0038] Computing device 200 may also include one or more output devices, such as display 218. Display 218 may be, in one example, a touch sensitive display that combines the display with a touch sensitive element operable to sense touch inputs. Display 218 may be coupled to CPU 202 via bus 212. Other output devices may be provided in addition to or as an alternative to the display 218 to allow a user to program or otherwise use the computing device 200. If the output device is a display or includes a display, the display may be in various ways, including by a light emitting diode (LED) display such as a liquid crystal display (LCD), a cathode ray tube (CRT) display, or an organic LED (OLED) display. It can be implemented as.

[0039] Computing device 200 is an image sensing device 220, such as a camera, or any other image sensing currently or future developed that can sense an image, such as an image of a user who is operating computing device 200. Device 220 may also include or be in communication with it. The image sensing device 220 may be positioned to be directed toward the user who operates the computing device 200. In one example, the position and optical axis of the image sensing device 220 can be configured to include an area where the field of view is immediately adjacent to the display 218 and the display 218 is visible.

[0040] Computing device 200 also includes or is in communication with a sound sensing device 222, such as a microphone, or any other sound sensing device presently or developed in the future capable of sensing sound near computing device 200. can do. The sound sensing device 222 may be positioned to be directed toward the user operating the computing device 200, and the sound produced by the user, such as voice or other utterances, while the user is operating the computing device 200. utterances).

[0041] 2 illustrates the CPU 202 and memory 204 of the computing device 200 integrated into one unit, however other configurations may be used. The operations of the CPU 202 may be distributed over multiple machines (individual machines may have one or more processors) that may be coupled directly or across a local area or other network. Memory 204 may be distributed across multiple machines, such as network-based memory or memory of multiple machines performing the operations of computing device 200. Although shown here as one bus, the bus 212 of the computing device 200 may consist of multiple buses. In addition, secondary storage 214 can be directly coupled to other components of computing device 200 or can be accessed via a network and includes an integrated unit such as a memory card or multiple units such as multiple memory cards. can do. Computing device 200 may thus be implemented in a variety of configurations.

[0042] 3 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. Video stream 300 includes video sequence 302. At the next level, video sequence 302 includes a number of adjacent frames 304. Although three frames are shown as adjacent frames 304, video sequence 302 may include any number of adjacent frames 304. Adjacent frames 304 may then be further subdivided into individual frames, eg, frame 306. In the next level, frame 306 may be divided into a series of planes or segments 308. Segments 308 may be, for example, a subset (subset) of frames that enable parallel processing. Segments 308 may also be a subset of frames that may separate video data into separate colors. For example, frame 306 of color video data may include a luminance plane and two chrominance planes. Segments 308 may be sampled at different resolutions.

[0043] Regardless of whether frame 306 is divided into segments 308, frame 306 may include, for example, blocks 310 that may include data corresponding to 16 × 16 pixels of frame 306. It can be further subdivided. Blocks 310 may also be arranged to include data from one or more segments 308 of pixel data. Blocks 310 may also be any other suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or more. Unless stated otherwise, the terms block and macroblock are used interchangeably herein.

[0044] 4 is a block diagram of an encoder 400 in accordance with implementations of the invention. Encoder 400 may be implemented at transmission station 102, as described above, for example, by providing a computer software program stored in a memory, for example memory 204. The computer software program may include machine instructions that, when executed by a processor such as the CPU 202, cause the transmitting station 102 to encode the video data in the manner described in FIG. 4. Encoder 400 may also be implemented as special hardware included in transmission station 102, for example. In one particularly preferred embodiment, the encoder 400 is a hardware encoder.

[0045] Encoder 400 performs the following steps to perform various functions in the forward path (shown as solid connection lines) to generate encoded or compressed bitstream 420 using video stream 300 as input. Have: intra / inter prediction step 402, transform step 404, quantization step 406, and entropy encoding step 408. The encoder 400 may also include a reconstruction path (shown with dashed connection lines) to reconstruct the frame for encoding of future blocks. In FIG. 4, encoder 400 has the following steps to perform various functions in the reconstruction path: inverse quantization step 410, inverse transform step 412, reconstruction step 414, and loop filtering step 416. Other structural variations of encoder 400 may be used to encode video stream 300.

[0046] When video stream 300 is presented for encoding, each frame 304, such as frame 306, may be processed in units of blocks. In intra / inter prediction step 402, each block may be encoded using intra-frame prediction (also called intra prediction) or inter-frame prediction (also called inter prediction). In any case, prediction blocks may be formed. In the case of intra prediction, a prediction block may be formed from samples of a current frame previously encoded and reconstructed. In the case of inter prediction, a prediction block may be formed from samples of one or more previously configured reference frames. The designation of reference frames for groups of blocks is discussed in more detail below.

[0047] Next, still referring to FIG. 4, the prediction block may be subtracted from the current block in an intra / inter prediction step 402 to produce a residual block (also referred to as a residual). Transform step 404 transforms the residuals into transform coefficients in the frequency domain using, for example, block-based transforms. Quantization step 406 converts the transform coefficients into discrete quantum values, referred to as quantization transform coefficients, using a quantizer value or quantization level. For example, the transform coefficients can be divided or truncated into quantizer values. The quantized transform coefficients are then entropy encoded by entropy encoding step 408. The entropy encoded coefficients are then combined with the compressed bitstream, along with other information used to decode the block, which may include, for example, the prediction type used, transform type, motion vectors, and quantizer value. 420). The compressed bitstream 420 may be formatted using various techniques such as variable length coding (VLC) or arithmetic coding. Compressed bitstream 420 may also be referred to as an encoded video stream or an encoded video bitstream, so these terms will be used interchangeably herein.

[0048] The reconstruction path of FIG. 4 (shown with dashed connection lines) to ensure that the encoder 400 and decoder 500 (described below) use the same reference frames to decode the compressed bitstream 420. Can be used. The reconstruction path inversely quantizes the quantized transform coefficients in inverse quantization step 410 and inversely transforms the inverse quantized transform coefficients in inverse transform step 412 to produce a differential residual block (also called a derivative residual). Performing functions similar to those occurring during the decoding process discussed in more detail below. In reconstruction step 414, the prediction block predicted in intra / inter prediction step 402 may be added to the differential residual to generate a reconstructed block. Loop filtering step 416 may be applied to the reconstructed block to reduce distortion, such as blocking artifacts.

[0049] Other variations of encoder 400 may be used to encode compressed bitstream 420. For example, a non-transformation based encoder can quantize the residual signal directly without the transform step 404 for certain blocks or frames. In another implementation, the encoder may combine the quantization step 406 and the dequantization step 410 into a common step.

[0050] 5 is a block diagram of a decoder 500 in accordance with implementations of the invention. Decoder 500 may be implemented at receiving station 106, for example, by providing a computer software program stored in memory 204. The computer software program may include machine instructions that, when executed by a processor such as the CPU 202, cause the receiving station 106 to decode the video data in the manner described in FIG. 5. Decoder 500 may also be implemented, for example, in hardware included in transmitting station 102 or receiving station 106.

[0051] Decoder 500 includes the following steps to perform various functions for generating output video stream 516 from compressed bitstream 420 in one example, similar to the reconstruction path of encoder 400 discussed above. Entropy decoding step 502, inverse quantization step 504, inverse transform step 506, intra / inter prediction step 508, reconstruction step 510, loop filtering step 512, and deblocking filtering step ( 514). Other structural variations of the decoder 500 may be used to decode the compressed bitstream 420.

[0052] When the compressed bitstream 420 is presented for decoding, data elements within the compressed bitstream 420 may be decoded by entropy decoding step 502 to produce a set of quantized transform coefficients. Inverse quantization step 504 inverse quantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by the quantizer value), and inverse transform step 506 inverse transforms the inverse quantized transform coefficients to obtain Generate a differential residual, which may be the same as produced by inverse transform step 412 at 400. Using the header information decoded from the compressed bitstream 420, the decoder 500 uses the intra / inter prediction step 508 to perform an encoder 400, for example at an intra / inter prediction step 402. The same prediction block as that generated can be generated. At reconstruction step 510, the predictive block may be added to the differential residual to generate a reconstructed block. Loop filtering step 512 can be applied to the reconstructed block to reduce blocking artifacts.

[0053] Other filtering may be applied to the reconstructed block. In this example, deblocking filtering step 514 is applied to the reconstructed block to reduce the blocking distortion, and the result is output to the output video stream 516. The output video stream 516 may also be referred to as a decoded video stream, so these terms will be used interchangeably herein. Other variations of the decoder 500 may be used to decode the compressed bitstream 420. For example, decoder 500 may generate output video stream 516 without deblocking filtering step 514.

[0054] 6 is a block diagram of an example of a reference frame buffer 600. Reference frame buffer 600 stores reference frames used to encode or decode blocks of frames of a video sequence. In this example, the reference frame buffer 600 includes reference frames identified by the last frame LAST_FRAME 602, the golden frame GOLDEN_FRAME 604, and the replacement reference frame ALTREF_FRAME 606. The frame header of the reference frame may include a virtual index for a position in the reference frame buffer in which the reference frame is stored. Reference frame mapping may map a virtual index of a reference frame to a physical index of a memory in which the reference frame is stored. If two reference frames are the same frame, these reference frames will have the same physical index even if they have different virtual indices. The number, types, and names of reference positions in the reference frame buffer 600 used are merely examples.

[0055] Reference frames stored in reference frame buffer 600 may be used to identify motion vectors for predicting blocks of frames to be encoded or decoded. Different reference frames may be used depending on the type of prediction used to predict the current block of the current frame. For example, in bi-prediction, blocks of the current frame may be forward predicted using the frame stored as LAST_FRAME 602 or GOLDEN_FRAME 604, and reversed using the frame stored as ALTREF_FRAME 606. Can be predicted.

[0056] There may be a finite number of reference frames that can be stored in the reference frame buffer 600. As shown in FIG. 6, the reference frame buffer 600 may store up to eight reference frames, and each stored reference frame may be associated with a different virtual index of the reference frame buffer. Three of the eight spaces in the reference frame buffer 600 are used by frames designated as LAST_FRAME 602, GOLDEN_FRAME 604, and ALTREF_FRAME 606, but five spaces are used to store other reference frames. It stays as possible. For example, one or more available spaces in the reference frame buffer 600 may be used to store additional reference frames, particularly some or all of the interpolated reference frames described herein. Although the reference frame buffer 600 is shown capable of storing up to eight reference frames, other implementations of the reference frame buffer 600 may store additional or fewer reference frames.

[0057] In some implementations, the replacement reference frame designated as ALTREF_FRAME 606 can be a frame of a video sequence that is far from the current frame in display order but encoded or decoded earlier than that displayed. For example, the replacement reference frame may be ten, twelve, or more (or less) frames after the current frame in display order. Additional replacement reference frames may be frames located closer to the current frame in display order.

[0058] The replacement reference frame may not correspond directly to a frame in the sequence. Instead, alternative reference frames may be applied using filtering, combining together, or combining together, as well as generated using one or more of the filtered frames. The replacement reference frame may not be displayed. Instead, the replacement reference frame may be part of a frame or frame that is generated and transmitted for use only in the prediction process (ie, it is omitted when the decoded sequence is displayed).

[0059] 7 is a diagram of a group of frames in the display order of a video sequence. In this example, the group of frames is preceded by a frame 700, which may be referred to as a key frame or in some cases an overlay frame, and includes eight frames 702-716. No block in frame 700 is inter predicted using the reference frames of the group of frames. Frame 700 is a key (also referred to as an intra predicted frame) in this example, which refers to the state in which predicted blocks within the frame are predicted only using intra prediction. However, frame 700 may be an overlay frame, which is an inter predicted frame, which may be a reconstructed frame of a previous group of frames. In an inter predicted frame, at least some of the predicted blocks are predicted using inter prediction. The number of frames forming each group of frames may vary depending on the spatial / temporal nature of the video and other encoded configurations such as, for example, a key frame interval selected for random access or error resilience. have.

[0060] The coding order of a group of frames may be different from the display order. This allows a frame located after the current frame in the video sequence to be used as a reference frame for encoding the current frame. A decoder such as decoder 500 may share a common group coding structure with an encoder such as encoder 400. The group coding structure assigns different roles that each frame in the group can play in the reference buff (eg, the last frame, the replacement reference frame, etc.) and defines or represents the coding order for the frames in the group.

[0061] 8 is a diagram of an example coding order for the group of frames of FIG. 7. The coding order of FIG. 8 relates to a first group coding structure in which a single backward reference frame is available for each frame of the group. Since the encoding and decoding order is the same, the order shown in FIG. 8 is generally referred to herein as the coding order. The key or overlay frame 700 is designated as a golden frame in the reference frame buffer, such as GOLDEN_FRAME 604 in the reference frame buffer 600. Frame 700 is intra predicted in this example, so it does not require a reference frame, but overlay frame as frame 700, which is a frame reconstructed from the previous group, also does not use the reference frame of the group of current frames. The last frame 716 in the group is designated an alternate reference frame in the reference frame buffer, such as ALTREF_FRAME 606 in the reference frame buffer 600. In this coding order, frame 716 is coded out of display order after frame 700 to provide a reverse reference frame for each of the remaining frames 702-714. In coding the blocks of frame 716, frame 700 serves as an available reference frame for the blocks of frame 716. 8 is just one example of a coding order for a group of frames. Other group coding structures may specify one or more different or additional frames for forward and / or backward prediction.

[0062] As briefly mentioned above, the available reference frame portion may be a reference frame portion that is interpolated using light flow estimation. The reference frame portion may be, for example, a block, a slice, or an entire frame. When frame level light flow estimation is performed as described herein, the resulting reference frame is referred to herein as a col-located reference frame because its dimensions are the same as the current frame. This interpolated reference frame may also be referred to herein as a lightflow reference frame.

[0063] 9 is a diagram used to illustrate the linear projection of a motion field in accordance with the teachings herein. Within the hierarchical coding framework, the light flow of the current frame (also called the motion field) can be estimated using the closest available reconstructed (eg, reference) frame before and after the current frame. In FIG. 9, reference frame 1 is a reference frame that can be used for forward prediction of the current frame 900, while reference frame 2 is a reference frame that can be used for backward prediction of the current frame 900. 6-8 for illustrative purposes, if the current frame 900 is a frame 706, the previous or last frame 704 (e.g., LAST_FRAME 602) to the reference frame buffer 600 The stored reconstructed frame) may be used as reference frame 1, while frame 716 (eg, a reconstructed frame stored in reference frame buffer 600 with ALTREF_FRAME 606) may be used as reference frame 2.

[0064] Knowing the display indices of the current and reference frames, assuming that the motion field is linear in time, motion vectors can be projected to the pixels of the current frame 900 between the pixels of the reference frame 1 and the reference frame 2. In the simple example described in connection with FIGS. 6-8, the index for the current frame 900 is 3, the index for reference frame 1 is 0, and the index for reference frame 2 is 716. In FIG. 9, the projected motion vector 904 for pixel 902 of the current frame 900 is shown. Using the previous example in the description, the display indices of the group of frames of FIG. 7 show that frame 704 is closer in time to frame 706 than to frame 716. Thus, the single motion vector 904 shown in FIG. 9 represents the amount of motion between reference frame 1 and current frame 900 that is different from reference frame 2 and current frame 900. Nevertheless, the projected motion field 906 is linear between reference frame 1, current frame 900, and reference frame 2.

[0065] Selecting the closest available reconstructed forward and reverse reference frames and assuming a motion field for each pixel of the current frame that is linear in time, is an interpolated reference frame using light flow estimation without transmitting additional information. The generation of may be performed at both the encoder and the decoder (eg, in the intra / inter prediction step 402 and the intra / inter prediction step 508). Instead of the closest available reconstructed reference frames, it is also possible that different frames can be used as specified a priori between the encoder and the decoder. In some implementations, an identification of the frames used for lightflow estimation can be sent. The generation of interpolated frames is discussed in more detail below.

[0066] 10 is a flowchart of a method or process 1000 for motion compensated prediction of a frame of a video sequence using at least a portion of a reference frame generated using light flow estimation. The reference frame portion may be, for example, a block, slice, or the entire reference frame. The lightflow reference frame portion may also be referred to herein as a juxtaposed reference frame portion. Process 1000 may be implemented as a software program that may be executed by computing devices such as, for example, transmitting station 102 or receiving station 106. For example, a software program may be stored in a memory such as memory 204 or secondary storage 214 and may cause the computing device to perform process 1000 when executed by a processor such as CPU 202. Machine-readable instructions. Process 1000 may be implemented using special hardware or firmware. Some computing devices may have multiple memories or processors, and the operations described in process 1000 may be distributed using multiple processors, memories, or both.

[0067] At 1002, the current frame to be predicted is determined. The frames can be coded in any order, such as the coding order shown in FIG. 8, and so can be predicted. Frames to be predicted may also be referred to as frames of the first, second, third, and the like. The first, second, etc. labels do not necessarily indicate the order of the frames. Instead, unless otherwise stated, labels are used herein to distinguish one current frame from another. In an encoder, a frame may be processed in units of blocks in a block coding order, such as a raster scan order. At the decoder, the frame may also be processed in units of blocks upon receipt of its encoded residuals in the encoded bitstream.

[0068] At 1004, forward and reverse reference frames are determined. In the examples described herein, the forward and backward reference frames are the closest reconstructed frames before and after the current frame (eg, in display order), such as current frame 900. Although not explicitly shown in FIG. 10, if there is no forward or backward reference frame, process 1000 ends. The current frame is then processed without considering the light flow.

[0069] If there are forward and reverse reference frames at 1004, the lightflow reference frame portion may be generated using the reference frames at 1006. Generation of the light flow reference frame portion is described in more detail with reference to FIGS. The lightflow reference frame portion may be stored at a defined location in the reference frame buffer 600 in some implementations. Initially, light flow estimation in accordance with the teachings herein is described.

[0070] Light flow estimation can be performed for each pixel of the current frame portion by minimizing the following Lagrange function 1:

(1)[0071]

(One)

는 명도 항등성 가정(즉, 이미지의 작은 부분의 강도 값은 위치 변화에도 불구하고 시간 경과에 따라 변화하지 않고 유지된다는 가정)에 기초한 데이터 페널티이다.

은 모션 필드의 평활도(즉, 인접한 픽셀들은 이미지에서 동일한 객체 항목에 속할 가능성이 높고 그래서 실질적으로 동일한 이미지 모션을 초래한다는 특성)에 기초한 공간 페널티이다. 라그랑지 파라미터 λ는 모션 필드의 평활도의 중요성을 제어한다. 파라미터 λ의 큰 값은 모션 필드를 보다 평활하게 하고, 보다 큰 스케일의 모션을 더 잘 감안할 수 있다. 대조적으로, 파라미터 λ의 더 작은 값은 객체의 에지들 및 작은 객체들의 이동에 보다 효과적으로 적응할 수 있다.[0072] In function 1,

Is a data penalty based on the lightness identity assumption (i.e. the intensity value of a small portion of the image remains unchanged over time despite position changes).

Is a spatial penalty based on the smoothness of the motion field (ie, the property that adjacent pixels are likely to belong to the same object item in the image and thus result in substantially the same image motion). The Lagrange parameter λ controls the importance of the smoothness of the motion field. Larger values of the parameter λ can make the motion field smoother and better account for motion at larger scales. In contrast, smaller values of the parameter λ can more effectively adapt to the movement of the edges and small objects of the object.

[0073] According to an implementation of the teachings herein, the data penalty can be expressed as a data penalty function:

(2)[0074]

(2)

[0075] The horizontal component of the motion field for the current pixel is represented by u, while the vertical component of the motion field is represented by v. Roughly speaking, E _x , E _y , and E _t are derivatives of pixel values of reference frame portions for horizontal axis x, vertical axis y, and time t (eg, as indicated by the frame indices). admit. The horizontal axis and the vertical axis are defined for an array of pixels forming a current frame, such as current frame 900, and reference frames, such as reference frames 1 and 2.

In the data penalty function, the derivatives E _x , E _y , and E _t can be calculated according to the following functions (3), (4), and (5):

[0077]

(3)[0078]

(3)

[0079]

(4)[0080]

(4)

(5)[0081]

(5)

은 인코딩되는 현재 프레임 내의 현재 픽셀 위치의 모션 필드에 기초한 레퍼런스 프레임 1 내의 투영된 위치에서의 픽셀 값이다. 유사하게, 변수

는 인코딩되는 현재 프레임 내의 현재 픽셀 위치의 모션 필드에 기초한 레퍼런스 프레임 2 내의 투영된 위치에서의 픽셀 값이다.[0082] variable

Is the pixel value at the projected position in reference frame 1 based on the motion field of the current pixel position in the current frame being encoded. Similarly, variable

Is the pixel value at the projected position in reference frame 2 based on the motion field of the current pixel position in the current frame being encoded.

은 레퍼런스 프레임 1의 디스플레이 인덱스이며, 여기서 프레임의 디스플레이 인덱스는 비디오 시퀀스의 디스플레이 순서에 있어서의 그 인덱스이다. 유사하게, 변수

는 레퍼런스 프레임 2의 디스플레이 인덱스이고, 변수

은 현재 프레임(900)의 디스플레이 인덱스이다.[0083] variable

Is the display index of reference frame 1, where the display index of the frame is that index in the display order of the video sequence. Similarly, variable

Is the display index of reference frame 2,

Is the display index of the current frame 900.

은 선형 필터를 사용하여 레퍼런스 프레임 1에서 계산된 수평 도함수(horizontal derivative)이다. 변수

는 선형 필터를 사용하여 레퍼런스 프레임 2에서 계산된 수평 도함수이다. 변수

은 선형 필터를 사용하여 레퍼런스 프레임 1에서 계산된 수직 도함수이다. 변수

는 선형 필터를 사용하여 레퍼런스 프레임 2에서 계산된 수직 도함수이다.[0084] variable

Is a horizontal derivative calculated in reference frame 1 using a linear filter. variable

Is the horizontal derivative calculated at reference frame 2 using a linear filter. variable

Is the vertical derivative calculated at reference frame 1 using the linear filter. variable

Is the vertical derivative calculated at reference frame 2 using a linear filter.

[0085] In an embodiment of the teachings herein, the linear filter used to calculate the horizontal derivative is a filter coefficient [-1/60, 9/60, -45/60, 0, 45/60, -9/60, 1/60. 7-tap filter with]. The filter may have a different frequency profile, different number of taps, or both. The linear filter used to calculate the vertical derivatives may be the same or different than the linear filter used to calculate the horizontal derivatives.

[0086] Spatial penalty can be expressed as a spatial penalty function:

(6)[0087]

(6)

[0088] In the spatial penalty function 6, Δu is Laplacian of the horizontal component u of the motion field, and Δv is Laplacian of the vertical component v of the motion field.

[0089] 11 is a flowchart of a method or process 1100 for generating a lightflow reference frame portion. In this example, the lightflow reference frame portion is the entire reference frame. Process 1100 may implement step 1006 of process 1000. Process 1100 may be implemented as a software program that may be executed by computing devices such as, for example, transmitting station 102 or receiving station 106. For example, a software program may be stored in a memory such as memory 204 or secondary storage 214 and may cause the computing device to perform process 1100 when executed by a processor such as CPU 202. Machine-readable instructions. Process 1100 may be implemented using special hardware or firmware. As noted above, multiple processors, memories, or both may be used.

[0090] Because forward and reverse reference frames can be relatively far apart from each other, there can be dramatic motion between them, thus degrading the accuracy of the brightness identity assumption. In order to reduce potential errors in motion of the pixel resulting from this problem, estimated motion vectors from the current frame to the reference frames can be used to initialize the light flow estimate for the current frame. At 1102, all pixels in the current frame can be assigned an initialized motion vector. These define initial motion fields that can be used to warp reference frames to the current frame for the first processing level to shorten the motion lengths between the reference frames.

은 다음의 함수에 따라, 현재 픽셀로부터 역방향 레퍼런스 프레임, 이 예에서는 레퍼런스 프레임 2를 가리키는 추정된 모션 벡터

와 현재 픽셀로부터 순방향 레퍼런스 프레임, 이 예에서는 레퍼런스 프레임 1을 가리키는 추정된 모션 벡터

사이의 차이를 나타내는 모션 벡터를 사용하여 초기화될 수 있다:[0091] Motion field of the current pixel

Is an estimated motion vector pointing from the current pixel to the reverse reference frame, in this example, reference frame 2.

And an estimated motion vector pointing from the current pixel to the forward reference frame, in this example, reference frame 1.

It can be initialized using a motion vector representing the difference between:

[0092]

[0093] If either of the motion vectors is not available, it is possible to extrapolate the initial motion using the available motion vector according to any of the following functions:

, 또는[0094]

, or

.[0095]

.

[0096] If the current pixel does not have an available motion vector reference, one or more spatial neighbors with an initialized motion vector can be used. For example, an average of available adjacent initialized motion vectors can be used.

In an example of initializing a motion field for a first processing level at 1102, reference frame 2 may be used to predict a pixel of reference frame 1, where reference frame 1 is the last frame before the current frame is coded. . The motion vector projected on the current frame using linear projection in a manner similar to that shown in FIG. 9 generates a motion field mv _cur at the intersecting pixel location, such as the motion field 906 at pixel location 902. Let's do it.

[0098] 11 refers to initializing a motion field for a first processing level because process 1100 preferably has multiple processing levels. This can be seen with reference to FIG. 13, which is a diagram illustrating the process 1100 of FIG. 11 (and the process 1200 of FIG. 12 discussed below). The following description uses the phrase motion field. This phrase is intended to collectively refer to the motion fields for each pixel, unless the context clearly dictates otherwise. Thus, the phrases "motion fields" or "motion fields" may be used interchangeably when referring to two or more motion fields. Also, when referring to the movement of pixels, the phrase light flow may be used interchangeably with the phrase motion field.

[0099] In order to estimate the motion field / light flow for the pixels of the frame, a pyramid or multilayer structure can be used. For example, in one pyramid structure, the reference frames are scaled down to one or more different scales. Then, the light flow is first estimated to obtain a motion field at the highest level (first processing level) of the pyramid, ie using the most scaled reference frames. Thereafter, the motion field is upscaled and used to initialize the light flow estimate at the next level. This process of upscaling the motion field, using it to initialize the next level of light flow estimation, and obtaining the motion field is performed until the lowest level of the pyramid is reached (i.e., for reference frame portions at full scale). Until flow estimation is completed).

[00100] The logical basis of this process is that it is easier to capture large motion when the image is scaled down. However, using simple rescale filters to scale the reference frames themselves may degrade the reference frame quality. A pyramid structure that scales derivatives instead of pixels of reference frames to estimate light flow to avoid loss of detail due to rescaling. This pyramid scheme represents a regression analysis for light flow estimation. This scheme is shown in FIG. 13 and is implemented by process 1100 of FIG. 11 and process 1200 of FIG. 12.

[00101] After initialization, the Lagrange parameter λ is set at 1104 to solve the Lagrange function 1. Preferably, process 1100 uses multiple values for the Lagrange parameter λ. The first value at which the Lagrange parameter λ is set at 1104 may be a relatively large value such as 100. Although the process 1100 preferably uses multiple values for the Lagrange parameter [lambda] within the Lagrange function 1, it is also possible that only one value is used as described below in the process 1200 described below. .

는 워핑을 수행하기 전에 그 풀 해상도 값으로부터 레벨의 해상도로 다운스케일링된다는 점에 유의할 필요가 있다. 모션 필드의 다운스케일링은 아래에서 보다 상세히 논의된다.At 1106, the reference frames are warped to the current frame according to the motion field for the current processing level. Warping the reference frames to the current frame may be performed using subpixel location rounding. Motion Fields Used at First Processing Level

It should be noted that is downscaled from its full resolution value to the resolution of the level before performing warping. Downscaling of the motion field is discussed in more detail below.

을 알면, 레퍼런스 프레임 1을 워핑하기 위한 모션 필드는 다음과 같이 (예를 들면, 모션은 시간 경과에 따라 선형으로 투영된다는) 선형 투영 가정에 의해 추론된다:[00103] light flow

Knowing that, the motion field for warping reference frame 1 is inferred by the linear projection assumption (e.g., the motion is projected linearly over time) as follows:

[00104]

의 수평 성분

및 수직 성분

은 Y 성분에 대해서는 1/8 픽셀 정밀도로 및 U 및 V 성분에 대해서는 1/16 픽셀 정밀도로 라운딩될 수 있다. 서브픽셀 위치 라운딩에 대한 다른 값들도 사용될 수 있다. 라운딩 후, 워핑된 이미지의 각각의 픽셀

은 모션 벡터

에 의해 주어진 참조된 픽셀로 계산된다. 종래의 서브픽셀 보간 필터를 사용하여 서브픽셀 보간이 수행될 수 있다.[00105] To perform warping, a motion field

Horizontal component of

And vertical components

Can be rounded to 1/8 pixel precision for the Y component and 1/16 pixel precision for the U and V components. Other values for subpixel position rounding can also be used. After rounding, each pixel of the warped image

Silver motion vector

Computed with the referenced pixel given by. Subpixel interpolation may be performed using conventional subpixel interpolation filters.

가 얻어지는데, 여기서 모션 필드는 다음에 의해 계산된다:[00106] The same warping approach is performed for reference frame 2 to which the warped image

Is obtained, where the motion field is calculated by:

[00107]

[00108] At the end of the calculation at 1106, there are two warped reference frames. Two warped reference frames are used to estimate the motion field between them at 1108. Estimating the motion field at 1108 may include a number of steps.

및/또는

)이 얻어질 수 있다. 그 다음에, 다수의 층들이 있으면, 도함수들은 현재 레벨로 다운스케일링된다. 도 13에 도시된 바와 같이, 레퍼런스 프레임들은 세부 사항들을 캡쳐하기 위해 원래 스케일에서 도함수를 계산하는 데 사용된다. 각각의 레벨 l에서 도함수들을 다운스케일링하는 것은 2¹ x 2¹ 블록 내에서 평균화함으로써 계산될 수 있다. 도함수들을 계산하는 것뿐만 아니라 도함수들을 평균화하여 다운스케일링하는 것은 모두 선형 연산들이기 때문에, 2 개의 연산들은 각각의 레벨 l에서 도함수들 계산하기 위해 단일의 선형 필터로 결합될 수 있다는 것이 주목된다. 이는 계산들의 복잡도를 낮출 수 있다.First, the derivatives E _x , E _y , and E _t are calculated using the functions (3), (4), and (5). When calculating the derivatives, the frame boundaries of the warped reference frame can be extended by copying the nearest available pixel. In this way, pixel values (i.e., when projected positions are outside of the warped reference frame)

And / or

) Can be obtained. Then, if there are multiple layers, the derivatives are downscaled to the current level. As shown in FIG. 13, reference frames are used to calculate the derivative at the original scale to capture details. Downscaling the derivatives at each level l can be calculated by averaging within 2 ¹ x 2 ¹ blocks. It is noted that the two operations can be combined into a single linear filter to calculate the derivatives at each level l, as well as calculating the derivatives as well as averaging downscaling the derivatives. This can lower the complexity of the calculations.

및

),

개의 선형 방정식들로 프레임의 모든 N 픽셀들에 대해 성분 u 및 v를 풀 수 있다. 이것은 라플라시안(Laplacians)이 2 차원(2D) 필터에 의해 근사된다는 사실에 기인한다. 정확하지만 매우 복잡한 선형 방정식들을 직접 푸는 대신에, 라그랑지 함수(1)를 최소화하기 위해 반복 접근법들이 사용되어 더 빠르지만 덜 정확한 결과를 얻을 수 있다.If the derivatives are downscaled to the current processing level, light flow estimation may be performed according to the Lagrange function 1, as appropriate. More specifically, by setting the derivatives of the Lagrange function 1 for the horizontal component u of the motion field and the vertical component v of the motion field to 0 (ie

And

),

The linear equations can solve components u and v for all N pixels of the frame. This is due to the fact that Laplacians are approximated by a two-dimensional (2D) filter. Instead of solving the exact but very complex linear equations directly, iterative approaches can be used to minimize the Lagrangian function (1) to achieve faster but less accurate results.

[00111] At 1108, the motion field for the pixels of the current frame is updated or improved using the estimated motion field between the warped reference frames. For example, the current motion field for a pixel can be updated by adding an estimated motion field for each pixel on a pixel-by-pixel basis.

[00112] Once the motion field is estimated at 1108, a query is made at 1110 to determine whether there are additional values available for the Lagrange parameter λ. Smaller values of the Lagrange parameter λ can cope with motion of a smaller scale. If there are additional values, process 1100 can return to 1104 to set the next value for the Lagrange parameter [lambda]. For example, process 1100 may repeat while cutting the Lagrange parameter λ in half at each iteration. The motion field updated at 1108 is the current motion field to warp reference frames at 1106 in this next iteration. The motion field is then estimated again at 1108. Processing at 1104, 1106, and 1108 continues until all possible Lagrange parameters at 1110 have been processed. In one example, there are three levels in the pyramid as shown in FIG. 13, so in one example the minimum value of the Lagrange parameter λ is 25. This iterative process performed while changing the Lagrange parameter may be referred to as annealing the Lagrange parameter.

[00113] Once there is no remaining value for the Lagrange parameter [lambda] at 1110, process 1100 proceeds to 1112 to determine whether there are more processing levels to process. If there are additional processing levels at 1112, process 1100 proceeds to 1114 where the motion field is upscaled before processing the next layer using each of the available values for the Lagrange parameter λ starting at 1104. Upscaling of the motion field may be performed using any known technique, including but not limited to the inverse of the downscaling calculations described above.

[00114] In general, the light flow is first estimated to obtain a motion field at the highest level of the pyramid. Thereafter, the motion field is upscaled and used to initialize the light flow estimate at the next level. This process of upscaling the motion field, using it to initialize the next level of light flow estimation, and obtaining the motion field is performed at 1112 until the lowest level of the pyramid is reached (i.e., full-scaled derivatives). Until the light flow estimation is completed.

를 형성한다. 1116에서 블렌딩된 워핑된 레퍼런스 프레임들은 1108에서 추정된 모션 필드를 사용하여 1106에 기술된 프로세스에 따라 다시 워핑되는 풀 스케일 레퍼런스 프레임들일 수 있음에 유의하자. 다시 말하면, 풀 스케일의 레퍼런스 프레임들은 2 회 ― 이전 처리 층으로부터의 최초의 업스케일링된 모션 필드를 사용하여 한 번 및 모션 필드가 풀 스케일 레벨로 개선된 후에 다시 ― 워핑될 수 있다. 블렌딩은 다음과 같이 (예를 들면, 프레임들이 동일한 시간 간격들로 이격되어 있다는) 시간 선형성 가정을 이용하여 수행될 수 있다:[00115] Once the level reaches a level where the reference frames are not downscaled (ie, the reference frames are at their original resolution), process 1100 proceeds to 1116. For example, the number of levels can be three, such as in the example of FIG. 13. At 1116, the warped reference frames are blended to form a lightflow reference frame

To form. Note that the warped reference frames blended at 1116 may be full scale reference frames warped again in accordance with the process described at 1106 using the motion field estimated at 1108. In other words, the full scale reference frames may be warped twice-once using the first upscaled motion field from the previous processing layer and again after the motion field has been improved to the full scale level. Blending can be performed using the assumption of linearity of time (eg, frames are spaced at equal time intervals) as follows:

[00116]

로 표시된) 레퍼런스 프레임 1의 레퍼런스 픽셀은 경계들을 벗어하는(예를 들면, 프레임의 치수들의 외부에 있음) 반면 레퍼런스 프레임 2의 레퍼런스 픽셀은 경계들을 벗어나지 않으면, 레퍼런스 프레임 2에서 발생하는 워핑된 이미지의 픽셀만 다음에 따라 사용된다:In some implementations, it is desirable to prefer a pixel of only one of the warped reference frames over the blended value. For example, (

The reference pixel of reference frame 1, which is indicated by, is outside the boundaries (eg, outside of the dimensions of the frame), while the reference pixel of reference frame 2 does not leave the boundaries, so that the reference pixel of the warped image Only pixels are used according to:

[00118]

[00119] Selective occlusion can be performed as part of the blending. Occlusion of objects and backgrounds is common in video sequences, where parts of an object appear in one reference frame but are hidden in another. In general, the light flow estimation method described above cannot estimate the motion of an object in such a situation because the brightness identity assumption is violated. If the size of the occlusion is relatively small, the smoothness penalty function can estimate the motion quite accurately. In other words, if an undefined motion field in the hidden part is smoothed by adjacent motion vectors, the motion of the entire object can be accurate.

[00120] However, even in this case, the simple blending method described above may not provide satisfactory interpolation results. This can be explained with reference to FIG. 14, which is a diagram illustrating object occlusion. In this example, the occluded portion of object A is visible in reference frame 1 and hidden by object B in reference frame 2. Since the hidden portion of object A is not displayed in reference frame 2, the referenced pixel from reference frame 2 is that of object B. In this case, it is preferable to use only warped pixels from reference frame one. Thus, using techniques to detect occlusions instead of or in addition to the above blending may provide better blending results and thus better reference frames.

및

는 2 개의 상이한 객체들에서 온 것이기 때문에 매우 다르다는 것이다. 두 번째 상황은 객체 B의 픽셀들은 현재 프레임 내의 객체 B 및 현재 프레임 내의 객체 A의 폐색 부분에 대한 다수의 모션 벡터들에 의해 참조된다는 것이다.Regarding the detection of occlusion, if occlusion occurs from FIG. 14 and the motion field is quite accurate, it is observed that the motion vector of the occluded portion of object A points to object B in reference frame 2. This can lead to the following situations. The first situation is warped pixel values

And

Is very different because it comes from two different objects. The second situation is that the pixels of object B are referenced by a number of motion vectors for object B in the current frame and occlusion portion of object A in the current frame.

에 대한

만의 폐색 및 사용을 결정하기 위해 다음의 조건들이 확립될 수 있으며,

에 대해

만 사용하는 경우에도 유사한 조건들이 적용된다:[00122] By these observations,

For

To determine the occlusion and use of the bay, the following conditions can be established,

About

Similar conditions apply for use only:

이 문턱값

보다 크고; 그리고[00123]

This threshold

Greater than; And

가 문턱값

보다 크다.[00124]

Fall threshold

Greater than

는 레퍼런스 프레임 1의 참조된 픽셀이 현재의 병치된 프레임 내의 임의의 픽셀에 의해 참조되는 총 횟수이다. 전술한 서브픽셀 보간이 존재하는 경우, 레퍼런스 서브픽셀 위치가 관심있는 픽셀 위치의 1 픽셀 길이 내에 있을 때

가 카운트된다. 또한,

가 서브픽셀 위치를 가리키는 경우, 4 개의 인접한 픽셀들의 가중 평균

는 현재의 서브픽셀 위치에 대한 레퍼런스들의 총 수이다.

도 유사하게 정의할 수 있다.[00125]

Is the total number of times the referenced pixel of reference frame 1 is referenced by any pixel in the current collocated frame. When the aforementioned subpixel interpolation exists, when the reference subpixel position is within 1 pixel length of the pixel position of interest.

Is counted. Also,

Is a weighted average of four adjacent pixels,

Is the total number of references to the current subpixel location.

Similarly, it can be defined.

[00126] Thus, occlusion may be detected in the first reference frame using the first warped reference frame and the second warped reference frame. The blending of the warped reference frames may then include populating pixel positions of the lightflow reference frame corresponding to the occlusion with pixel values from the second warped reference frame. Similarly, occlusion may be detected in the second reference frame using the first warped reference frame and the second warped reference frame. The blending of the warped reference frames may then include filling pixel locations of the lightflow reference frame corresponding to the occlusion with pixel values from the first warped reference frame.

선형 방정식들을 이용한다. 다시 말해서, 광흐름 추정의 계산 복잡도는 프레임 크기의 다항식 함수이며, 이는 디코더의 복잡도에 부담을 준다. 따라서, 서브프레임 기반(예를 들면, 블록 기반)의 광흐름 추정이 다음에 설명되는데, 이는 도 11과 관련하여 설명된 프레임 기반 광흐름 추정보다 디코더의 복잡도를 저감시킬 수 있다.It is experimentally suggested that process 1100 provides substantial compression performance gains. These performance gains include gains of 2.5% in PSNR and 3.3% in SSIM for a set of low resolution frames, and 3.1% in PSNR and 4.0% in SSIM for a set of medium resolution frames. However, and as described above, the light flow estimation performed according to the Lagrange function 1 is used to solve the horizontal component u and vertical component v of the motion field for all N pixels of the frame.

Use linear equations. In other words, the computational complexity of the light flow estimation is a polynomial function of the frame size, which burdens the complexity of the decoder. Accordingly, subframe based (eg, block based) light flow estimation is described next, which may reduce the complexity of the decoder than the frame based light flow estimation described in connection with FIG. 11.

[00128] 12 is a flowchart of a method or process 1200 for generating a lightflow reference frame portion. In this example, the light flow reference frame portion is smaller than the entire reference frame. In this example the juxtaposed frame portions are described with reference to the block, but other frame portions can be processed according to FIG. 12. Process 1200 may implement step 1006 of process 1000. Process 1200 may be implemented as a software program that may be executed by computing devices such as, for example, transmitting station 102 or receiving station 106. For example, a software program may be stored in a memory such as memory 204 or secondary storage 214 and may cause the computing device to perform process 1200 when executed by a processor such as CPU 202. Machine-readable instructions. Process 1200 may be implemented using special hardware or firmware. As noted above, multiple processors, memories, or both may be used.

[00129] At 1202, all pixels in the current frame are assigned an initialized motion vector. These define initial motion fields that can be used to warp reference frames to the current frame for the first processing level to shorten the motion lengths between the reference frames. Initialization at 1202 may be performed using the same processing as described with respect to initialization at 1102, so the description is not repeated here.

At 1204, reference frames—eg, reference frame 1 and reference frame 2 — are warped to the current frame according to the motion field initialized at 1202. The warping at 1204 is preferably the same process as described with respect to warping at 1106, except that the motion field mv _cur initialized at 1202 is not downscaled from its full resolution value before warping the reference frames. It can be performed using.

[00131] At the end of the calculation at 1204, there are two warped reference frames in full resolution. Like process 1100, process 1200 may estimate a motion field between two reference frames using a multi-level process similar to that described with respect to FIG. 13. Roughly speaking, process 1200 calculates the derivatives for the level until all levels are taken into account, performs the light flow estimation using the derivatives, and upscales the resulting motion field for the next level.

이 1206에서 초기화된다. 블록은 현재 프레임의 스캔 순서(예를 들면, 래스터 스캔 순서)에서 선택된 현재 프레임의 블록일 수 있다. 블록에 대한 모션 필드

은 블록의 각각의 픽셀들에 대한 모션 필드를 포함한다. 즉, 1206에서, 현재의 블록 내의 모든 픽셀들에는 초기화된 모션 벡터가 할당된다. 초기화된 모션 벡터들은 레퍼런스 프레임들의 레퍼런스 블록들 사이의 길이들을 단축시키기 위해 레퍼런스 블록들을 현재 블록으로 워핑하는데 사용된다.More specifically, the motion field for the block at the current (or first) processing level

This is initialized at 1206. The block may be a block of the current frame selected in the scan order (eg, raster scan order) of the current frame. Motion Fields for Blocks

Contains a motion field for each pixel of the block. That is, at 1206, all pixels in the current block are assigned an initialized motion vector. The initialized motion vectors are used to warp the reference blocks to the current block to shorten the lengths between the reference blocks of the reference frames.

은 그 풀 해상도 값으로부터 레벨의 해상도로 다운스케일링된다. 다시 말하면, 1206에서의 초기화는 1202에서 초기화된 풀 해상도 값으로부터 블록의 각각의 픽셀들에 대한 모션 필드를 다운스케일링하는 것을 포함할 수 있다. 다운스케일링은 전술한 다운스케일링과 같은, 임의의 기술을 사용하여 수행될 수 있다.[00133] At 1206, a motion field

Is downscaled from its full resolution value to the resolution of the level. In other words, the initialization at 1206 can include downscaling the motion field for each pixel of the block from the full resolution value initialized at 1202. Downscaling may be performed using any technique, such as the downscaling described above.

를 알면, 워핑을 위한 모션 필드는 다음과 같이 (예를 들면, 모션은 시간 경과에 따라 선형으로 투영된다는) 선형 투영 가정에 의해 추론된다:At 1208, the collocated reference blocks corresponding to the motion field of each of the warped reference frames are warped into the current block. Warping of the reference blocks is performed similarly to process 1100 at 1106. That is, the light flow of pixels of the reference block of reference frame 1

Knowing that, the motion field for warping is inferred by the linear projection assumption (e.g., the motion is projected linearly over time) as follows:

[00135]

의 수평 성분

및 수직 성분

은 Y 성분에 대해서는 1/8 픽셀 정밀도로 및 U 및 V 성분에 대해서는 1/16 픽셀 정밀도로 라운딩될 수 있다. 다른 값들도 사용될 수 있다. 라운딩 후, 워핑된 블록의 각각의 픽셀, 예를 들면

은 모션 벡터 mv_r1에 의해 주어진 참조된 픽셀로 계산된다. 종래의 서브픽셀 보간 필터를 사용하여 서브픽셀 보간이 수행될 수 있다.[00136] To perform warping, a motion field

Horizontal component of

And vertical components

Can be rounded to 1/8 pixel precision for the Y component and 1/16 pixel precision for the U and V components. Other values may be used. After rounding, each pixel of the warped block, for example

Is computed with the referenced pixel given by the motion vector mv _r1 . Subpixel interpolation may be performed using conventional subpixel interpolation filters.

가 얻어지는데, 여기서 모션 필드는 다음에 의해 계산된다:[00137] The same warping approach is also performed on the reference block of reference frame 2 to warped blocks, for example

Is obtained, where the motion field is calculated by:

[00138]

[00139] At the end of the calculation at 1208, there are two warped reference blocks. Two warped reference blocks are used at 1210 to estimate the motion field between them. The processing at 1210 may be similar to that described with respect to the processing at 1108 of FIG. 11.

및/또는

로 사용될 수 있다. 투영된 픽셀들이 프레임 경계부들의 외부에 있는 경우, 가장 가까운 이용 가능한(즉, 경계들 내의) 픽셀을 복사하는 것이 여전히 사용될 수 있다. 도함수들이 계산된 후, 이들은 현재 레벨로 다운스케일링될 수 있다. 각각의 레벨 l에서 다운스케일링된 도함수들은 앞서 논의된 바와 같이, 2¹ x 2¹ 블록 내에서 평균화함으로써 계산될 수 있다. 도함수들을 계산하고 평균화하는 두 가지 선형 연산들을 단일의 선형 필터로 결합함으로써 계산들의 복잡도를 낮출 수 있지만, 필수는 아니다.More specifically, the two warped reference blocks may be full resolution. According to the pyramid structure of FIG. 13, the derivatives E _x , E _y , and E _t are calculated using the functions (3), (4), and (5). When calculating derivatives for frame level estimation, frame boundaries can be extended by copying the nearest available pixel to obtain out of bounds pixel values as described in connection with process 1100. However, for other frame portions, adjacent pixels in the warped reference frames at 1204 are often available. For example, for block-based estimation, pixels of adjacent blocks are available in warped reference frames unless the block itself is at the frame boundary. Thus, in the case of pixels that are out of bounds for the warped reference frame portion, the pixels of adjacent portions of the warped reference frame are pixel values, if applicable.

And / or

Can be used as If the projected pixels are outside of the frame boundaries, copying the nearest available (ie within the boundaries) pixel can still be used. After the derivatives are calculated, they can be downscaled to the current level. The downscaled derivatives at each level l can be calculated by averaging within 2 ¹ x 2 ¹ blocks, as discussed above. The complexity of the calculations can be reduced, but not required, by combining two linear operations that calculate and average the derivatives into a single linear filter.

및

),

개의 선형 방정식들을 푸는 것에 의해, 부분, 여기서는 블록의 모든 N 픽셀들에 대해 모션 필드의 수평 성분 u 및 수직 성분 v가 결정될 수 있다. 이를 위해, 경계를 벗어난 모션 벡터들을 다루는 두 가지 선택적 방법이 있다. 하나의 방법은 인접한 블록들과 제로 상관(zero correlation)을 가정하고, 경계를 벗어난 모션 벡터가 경계를 벗어난 픽셀 위치에 가장 가까운 경계부 위치에 있는 모션 벡터와 동일하다고 가정하는 것이다. 다른 방법은 현재 픽셀에 대해 초기화된 모션 벡터(즉, 1206에서 초기화된 모션 필드)를 현재 픽셀에 대응하는 경계를 벗어난 픽셀 위치에 대한 모션 벡터로 사용하는 것이다.Continuing with the processing at 1210, the downscaled derivatives can be used as inputs of the Lagrange function 1 to perform light flow estimation to estimate the motion field between the warped reference portions. Set the derivatives of the Lagrange function 1 for the horizontal component u and the vertical component v to 0 (i.e.

And

),

By solving the linear equations, the horizontal component u and the vertical component v of the motion field can be determined for a portion, here all N pixels. To do this, there are two alternative ways of dealing with out of bounds motion vectors. One method is to assume zero correlation with adjacent blocks and to assume that the out of bounds motion vector is the same as the motion vector at the border position closest to the out of bound pixel position. Another method is to use the motion vector initialized for the current pixel (ie, the motion field initialized at 1206) as the motion vector for the off-boundary pixel position corresponding to the current pixel.

[00142] After the motion field is estimated, the current motion field for the level is updated or improved using the estimated motion field between the warped reference blocks to complete the processing at 1210. For example, the current motion field for a pixel can be updated by adding an estimated motion field for each pixel on a pixel-by-pixel basis.

[00143] In process 1100, an additional loop to set decreasing values for Lagrange parameter λ such that at each level the motion field is estimated and improved using smaller and smaller values for Lagrange parameter λ. Included. In process 1200, this loop is omitted. That is, in the process 1200 as shown, only one value for the Lagrange parameter [lambda] is used to estimate the motion field at the current processing level. This may be a relatively small value, such as 25. For example, other values are possible for the Lagrange parameter λ, depending on the smoothness of the motion, image resolution, or other variables.

[00144] In other implementations, process 1200 can include an additional loop to change the Lagrange parameter λ. In implementations in which such a loop is included, warping the reference blocks at 1208 and estimating and updating the motion field at 1210, as described with respect to the processing at 1104 and 1110 in process 1100, is applied to the Lagrange parameter λ. The Lagrange parameter λ may be set before estimating the motion field at 1210 so that all values for are repeated until used.

[00145] Process 1200 proceeds to query 1212 after estimating and updating the motion field at 1210. This is done after the first and only motion field estimation and update at the level of 1210 when a single value is used for the Lagrange parameter λ. When multiple values for Lagrange parameter λ are modified at the processing level, process 1200 proceeds to query 1212 after estimating and updating the motion field at 1210 using the final value of Lagrange parameter λ.

[00146] If there are additional processing levels in response to the query at 1212, process 1200 proceeds to 1214, where the motion field starts up at 1206 and is upscaled before processing the next layer. Upscaling can be performed according to any known technique.

[00147] In general, the light flow is first estimated to obtain a motion field at the highest level of the pyramid. The motion field is then upscaled and used to initialize the light flow estimate at the next level. This process of upscaling the motion field, using it to initialize the next level of light flow estimation, and obtaining the motion field is performed at 1212 until the lowest level of the pyramid is reached (i.e., full-scaled derivatives). Until the light flow estimation is completed.

)을 형성한다. 1216에서 블렌딩된 워핑된 레퍼런스 블록들은 1210에서 추정된 모션 필드를 사용하여 1208에 기술된 프로세스에 따라 다시 워핑되는 풀 스케일 레퍼런스 블록들일 수 있음에 유의하자. 다시 말하면, 풀 스케일의 레퍼런스 블록들은 2 회 ― 이전 처리 층으로부터의 최초의 업스케일링된 모션 필드를 사용하여 한 번 및 모션 필드가 풀 스케일 레벨로 개선된 후에 다시 ― 워핑될 수 있다. 블렌딩은 1116에서 설명된 처리와 유사하게 시간 선형성 가정을 이용하여 수행될 수 있다. 1116에서 설명되고 도 14에 예로서 도시된 바와 같은 선택적인 폐색 검출은 1216에 블렌딩의 일부로서 통합될 수도 있다.[00148] Once the level reaches a level where the reference frames are not downscaled (ie, the reference frames are at their original resolution), the process 1200 proceeds to 1216. For example, the number of levels can be three, such as in the example of FIG. 13. At 1216, the warped reference blocks are blended to form a lightflow reference block (eg, as described above).

). Note that the warped reference blocks blended at 1216 may be full scale reference blocks warped again according to the process described at 1208 using the motion field estimated at 1210. In other words, the full scale reference blocks can be warped twice-once using the first upscaled motion field from the previous processing layer and again after the motion field has been improved to the full scale level. Blending can be performed using the assumption of linearity of time similar to the process described at 1116. Selective occlusion detection, as described at 1116 and shown as an example in FIG. 14, may be incorporated as part of the blending at 1216.

[00149] After the collocated reference block is generated at 1216, process 1200 proceeds to 1218 to determine whether there are additional frame portions (here, blocks) for prediction. If so, process 1200 repeats, beginning at 1206 for the next block. Blocks may be processed in scan order. If there are no more blocks to consider in response to the query at 1218, process 1200 ends.

[00150] Referring back to FIG. 10, process 1200 may implement 1006 in process 1000. At the end of the processing at 1006, there are one or more warped reference frame portions, whether performed in accordance with process 1100, process 1200, or variations thereof as described herein.

[00151] At 1008, the prediction process is performed using the portion of the lightflow reference frame generated at 1006. Performing the prediction process at the encoder may include generating the predictive block from the lightflow reference frame for the current block of the frame. The light flow reference frame may be a light flow reference frame output by the process 1100 and stored in a reference frame buffer such as the reference frame buffer 600. The light flow reference frame may be a light flow reference frame generated by combining the light flow reference portions output by the process 1200. Combining the light flow reference portions may be to arrange the light flow reference portions (eg, collocated reference blocks) according to the pixel positions of each current frame portions used to generate each of the light flow reference portions. It may include. The resulting lightflow reference frame may be stored for use in the encoder's reference frame buffer, such as the reference frame buffer 600 of the encoder 400.

[00152] Generating the predictive block at the encoder may include selecting the collocated block in the lightflow reference frame as the predictive block. Generating the predictive block at the encoder may instead include performing a motion search within the lightflow reference frame to select the best matching predictive block for the current block. However, the predictive block is generated at the encoder, and the resulting residual can be further processed using, for example, the lossy encoding process described with respect to encoder 400 of FIG.

[00153] At the encoder, process 1000 includes a rate for the current block using various prediction modes, including both single and complex inter prediction modes using one or more intra prediction modes and prediction frames available for the current frame. It can form part of a rate distortion loop. The single inter prediction mode uses only a single forward or backward reference frame for inter prediction. Complex inter prediction mode uses both forward and backward reference frames for inter prediction. In the rate distortion loop, the rate (eg, number of bits) used to encode the current block using the respective prediction modes is compared with the distortion due to the encoding. The distortion can be calculated with the differences between the pixel values of the block before encoding and after decoding. The differences can be another measure that captures the sum of absolute differences or cumulative error for blocks of frames.

[00154] In some implementations, it may be desirable to limit the use of lightflow reference frames to a single inter prediction mode. That is, the light flow reference frame may be excluded as a reference frame in all composite reference modes. This can simplify the rate distortion loop, and since the optical flow reference frame already considers both the forward and reverse reference frames, it is expected that there will be little additional impact on the encoding of the block. According to the implementations described herein, a flag may be encoded into the bitstream to indicate whether a lightflow reference frame is available for use in encoding the current frame. The flag may in one example be encoded when any single block in the current frame is encoded using the lightflow reference frame block. If the lightflow reference frame is available for the current frame, it may include an additional flag or other indicator (eg, at the block level) indicating whether the current block has been encoded by inter prediction using the lightflow reference frame.

[00155] The prediction process at 1008 may be repeated for all blocks of the current frame until the current frame is encoded.

[00156] At the decoder, performing the prediction process using the lightflow reference frame portion at 1008 may be due to the determination that the lightflow reference frame is available for decoding the current frame. In some implementations, the decision is made by examining a flag indicating that at least one block of the current frame has been encoded using the lightflow reference frame portion. Performing the prediction process at 1008 at the decoder may include generating a prediction block. Generating the predictive block may include using an inter prediction mode decoded from the encoded bitstream, as in the block header. A flag or indicator can be decoded to determine the inter prediction mode. If the inter prediction mode is the lightflow reference frame mode (ie, the block is inter predicted using the lightflow reference frame portion), the prediction block for the current block to be decoded is the pixels and motion vectors of the lightflow reference frame portion. Generated using mode and / or motion vectors.

[00157] The same processing for generating a lightflow reference frame for use in the prediction process as part of the decoding may be performed at a decoder such as decoder 500, as performed at the encoder. For example, when the flag indicates that at least one block of the current frame has been encoded using the lightflow reference frame portion, the entire lightflow reference frame may be generated and stored for use in the prediction process. However, by modifying the process 1200 to limit the performance of the process 1200 in which the coding blocks are identified as using a juxtaposition / lightflow reference frame as the inter prediction reference frame, the computation power at the decoder is further saved. This may be described with reference to FIG. 15, which is a diagram illustrating one technique for optimizing a decoder.

[00158] In FIG. 15, pixels are shown along grid 1500, where w represents the location of the pixel along the first axis of grid 1500 and y represents the location of the pixel along the second axis of grid 1500. Grid 1500 represents pixel locations of a portion of the current frame. The processing at 1006 and 1008 may be combined to perform the prediction process at the decoder at 1008. For example, prior to performing the process at 1006, the prediction process at 1008 may include finding a reference block used to encode the current block (eg, from header information such as a motion vector). In FIG. 15, the motion vector for the current coding block 1502 points to the reference block indicated by the inner dashed line 1504. Current coding block 1502 includes 4x4 pixels. The location of the reference block is shown by dashed line 1504 because the reference block is located in the reference frame rather than the current frame.

[00159] Once the reference block is located, all of the reference blocks that span the reference block (ie, overlap) are identified. This may include extending the size of the reference block by half the filter length at each boundary to account for subpixel interpolation filters. In FIG. 15, the length L of the subpixel interpolation filter is used to extend the reference block to the boundaries indicated by the outer dashed line 1506. As is relatively common, a motion vector generates a reference block that is not perfectly aligned with full-pel positions. The dark areas in FIG. 15 represent the full pel locations. All reference blocks that overlap with the full pel locations are identified. Assuming that the block sizes are the same as the current coding block 1502, the first reference block collocated with the current block, the second reference block on top of the first reference block, and two reference blocks extending from the left side of the first reference block. And two reference blocks extending from the left side of the second reference block are identified.

[00160] Once the reference blocks are identified, process 1200 may be performed at 1006 only for blocks in the current frame that are collocated with the identified reference blocks to generate collocated / lightflow estimated reference blocks. In the example of FIG. 15, this results in six light flow reference frame portions.

[00161] This modified process ensures that the encoder and decoder have the same predictor while the decoder does not need to calculate the entire collocated reference frame. It is noted that the reference block (s) for subsequent blocks that include any extended boundaries may overlap with one or more reference blocks identified during the decoding process of the current block. In this case, light flow estimation only needs to be performed once for any of the identified blocks to further reduce computing requirements at the decoder. In other words, the reference block generated at 1216 may be stored for use in decoding other blocks of the current frame.

[00162] However, the prediction block is generated at the decoder, and the decoded residual for the current block from the encoded bitstream is combined with the prediction block to form a reconstruction block as described by way of example with respect to decoder 500 of FIG. 5. Can be.

[00163] Whether performed after or in conjunction with process 1200, the prediction process at 1008 may be repeated for all blocks of the current frame encoded using the lightflow reference frame portion until the current frame is decoded. Can be. When processing blocks in decoding order, a block that is not encoded using the lightflow reference frame portion may be conventionally decoded in a conventional manner according to the decoded prediction mode for the block from the encoded bitstream.

[00164] For N pixels in a frame or block, the complexity for solving the light flow formula can be represented by O (N * M), where M is the number of iterations to solve the linear equations. M is not related to the number of levels or the number of values of the Lagrange parameter λ. Instead, M relates to computational precision in solving linear equations. The larger the value of M, the better the accuracy. Given this complexity, proceeding from the frame level to the estimation of the subframe level (eg, block-based) offers several options for reducing decoder complexity. First, and because the constraints of motion field smoothness are relaxed at block boundaries, it is easier to converge on the solution when solving the linear equations of the block, so M is smaller for similar precision. Second, solving the motion vector includes its adjacent motion vectors due to the smoothness penalty factor. Motion vectors at block boundaries have fewer contiguous motion vectors, so faster calculations are made. Third, and as discussed above, the light flow need not be calculated for the entire frame, but only for a portion of the blocks of the collocated reference frame identified by those coding blocks using the collocated reference frame for inter prediction. have.

[00165] For simplicity of explanation, each of the processes 1000, 1100, and 1200 are shown and described as a series of steps or operations. However, steps or actions in accordance with the present invention may occur in various orders and / or concurrently. In addition, other steps or actions not shown and described herein may be used. Moreover, not all illustrated steps or actions may be required to implement a methodology in accordance with the disclosed subject matter.

[00166] Aspects of the encoding and decoding described above illustrate some examples of encoding and decoding techniques. However, it should be understood that encoding and decoding may mean compression, decompression, transformation, or any other processing or modification of data, as these terms are used in the claims.

[00167] The word "yes" is used herein to mean functioning as an example, example, or illustration. Any aspect or design described herein as "example" need not necessarily be construed as preferred or advantageous over other aspects or designs. Rather, the use of the word "yes" is intended to present the concept in a concrete manner. The term "or" as used herein is intended to mean a generic "or" rather than an exclusive "or". In other words, unless otherwise specified or apparent from the context, "X includes A or B" is intended to mean any of the natural, comprehensive permutations. That is, X comprises A; X comprises B; Or if X comprises both A and B, then "X comprises A or B" is met in any of the foregoing cases. Also, the articles “a” and “an” used in this application and the appended claims should be generally interpreted to mean “one or more” unless specifically specified or in the context of a singular form. Moreover, the use of the term "one embodiment" or "an embodiment" throughout is not intended to mean this unless it is indicated to mean the same embodiment or embodiment.

[00168] Transmitting station 102 and / or receiving station 106 (and stored therein and / or algorithms, methods, instructions, etc., executed by it, including by encoder 400 and decoder 500) May be implemented in hardware, software, or any combination thereof. The hardware may include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, Servers, microprocessors, digital signal processors, or any other suitable circuit. In the claims, the term "processor" should be understood to include any or all of the foregoing hardware, alone or in combination. The terms "signal" and "data" are used interchangeably. In addition, the parts of the transmitting station 102 and the receiving station 106 need not necessarily be implemented in the same manner.

[00169] Further, in one aspect, for example, the transmitting station 102 or the receiving station 106 has a computer program that, when executed, performs any of the respective methods, algorithms, and / or instructions described herein. It may be implemented using a general purpose computer or a general purpose processor. Additionally or alternatively, a dedicated computer / processor may be used that includes, for example, other hardware for performing any of the methods, algorithms, or instructions described herein.

[00170] The transmitting station 102 and the receiving station 106 may be implemented on computers of a video conferencing system, for example. Alternatively, the transmitting station 102 may be implemented on a server and the receiving station 106 may be implemented on a device separate from the server, such as a hand-held communication device. In this case, the transmitting station 102 may use the encoder 400 to encode the content into an encoded video signal and transmit the encoded video signal to the communication device. In turn, the communication device can then decode the encoded video signal using decoder 500. Alternatively, the communication device may decode content stored locally on the communication device, eg, content that was not transmitted by the transmitting station 102. Other suitable transmit and receive implementation schemes are also available. For example, the receiving station 106 may be a generally stationary personal computer rather than a portable communication device and / or a device that includes the encoder 400 may also include the decoder 500.

[00171] In addition, all or some of the embodiments of the invention may take the form of a computer program product accessible from a computer usable or computer readable medium, for example. The computer usable or computer readable medium can be any device capable of tangibly containing, storing, communicating, or carrying a program for use by or in conjunction with any processor, for example. . The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media are also available.

[00172] Further implementations are summarized in the examples below.

[00173] Example 1: determining a first frame to be predicted in a video sequence; Determining a first reference frame from the video sequence for forward inter prediction of the first frame; Determining a second reference frame from the video sequence for reverse inter prediction of the first frame; Generating an optical flow reference frame for inter prediction of the first frame by performing optical flow estimation using the first reference frame and the second reference frame; And performing a prediction process for the first frame using the lightflow reference frame.

[00174] Example 2: In the method of Example 1, generating the lightflow reference frame includes: performing lightflow estimation by minimizing a Lagrangian function for each pixel of the first frame.

[00175] Example 3: In the method of example 1 or 2, the light flow estimation generates each motion field for the pixels of the first frame, and the generating the light flow reference frame comprises: forming a first warped reference frame Warping the first reference frame to the first frame using the motion fields; Warping the second reference frame to the first frame using the motion fields to form a second warped reference frame; And blending the first warped reference frame and the second warped reference frame to form a lightflow reference frame.

[00176] Example 4: In the method of example 3, the blending of the first warped reference frame and the second warped reference frame comprises: between the first reference frame and the second reference frame and between the current frame and the first reference frame and the second reference. Combining the collocated pixel values by scaling the collocated pixel values of the first warped reference frame and the second warped reference frame using the distances between each of the frames.

[00177] Example 5: In the method of example 3 or example 4, the blending of the first warped reference frame and the second warped reference frame comprises: collocated pixel values of the first warped reference frame and the second warped reference frame; Filling pixel positions of the lightflow reference frame by either combining or using a single pixel value of either the first warped reference frame or the second warped reference frame.

[00178] Example 6: In the method of any of examples 3-5, the method further comprises: detecting an occlusion in the first reference frame using the first warped reference frame and the second warped reference frame, Blending the first warped reference frame and the second warped reference frame includes: filling pixel locations of the lightflow reference frame corresponding to the occlusion with pixel values from the second warped reference frame.

[00179] Example 7: In the method of any of examples 1-6, performing the prediction process comprises: using the lightflow reference frame only for single reference inter prediction of blocks of the first frame.

[00180] Example 8: The method of any of examples 1-7, wherein the first reference frame is the closest reconstructed frame in display order of the video sequence relative to the first frame available for forward inter prediction of the first frame; The two reference frames are the reconstructed frames closest in display order to the first frame available for reverse inter prediction of the first frame.

[00181] Example 9: In the method of any one of examples 1-8, performing the prediction process comprises: determining a reference block in a lightflow reference frame in parallel with the first block of the first frame; And encoding a residual of the reference block and the first block.

[00182] Example 10 An apparatus includes: a processor; And a non-transitory storage medium comprising instructions executable by the processor to perform the method, the method comprising: determining a first frame to be predicted in a video sequence; Determining availability of the first reference frame for forward inter prediction of the first frame and availability of the second reference frame for reverse inter prediction of the first frame; In response to determining the availability of both the first reference frame and the second reference frame: each motion for the pixels of the first frame using the first reference frame and the second reference frame using the light flow estimation Creating a field; Warping the first reference frame to the first frame using the motion fields to form a first warped reference frame; Warping the second reference frame to the first frame using the motion fields to form a second warped reference frame; And blending the first warped reference frame and the second warped reference frame to form an optical flow reference frame for inter prediction of the blocks of the first frame.

[00183] Example 11: In the apparatus of Example 10, the method further comprises: performing a prediction process for the first frame using the lightflow reference frame.

[00184] Example 12: The apparatus of example 10 or 11, the method further comprising using the lightflow reference frame only for single reference inter prediction of blocks of the first frame.

[00185] Example 13: In the apparatus of any of Examples 10-12, generating each motion field comprises: a Lagrange for each pixel of the first frame using the first reference frame and the second reference frame. Calculating the output of the function.

[00186] Example 14: In the apparatus of Example 13, calculating the output of the Lagrange function includes: calculating a first set of motion fields for pixels of the current frame using the first value for the Lagrange parameter; And using the first set of motion fields as input to a Lagrange function using the second value for the Lagrange parameter to calculate a set of improved motion fields for the pixels of the current frame, The second value for the Lagrange parameter is smaller than the first value for the Lagrange parameter, and the first warped reference frame and the second warped reference are warped using the improved set of motion fields.

[00187] Example 15 An apparatus includes: a processor; And a non-transitory storage medium comprising instructions executable by the processor to perform the method, the method comprising: initializing motion fields for pixels of a first frame at a first processing level for light flow estimation; The first processing level represents downscaled motion within the first frame and includes one of a plurality of levels—a first reference frame from the video sequence and a second reference frame of the video sequence; Generating an optical flow reference frame for inter prediction of the first frame; For each of the plurality of levels: warping the first reference frame to the first frame using motion fields to form a first warped reference frame; Warping the second reference frame to the first frame using the motion fields to form a second warped reference frame; Estimating motion fields between the first warped reference frame and the second warped reference frame using light flow estimation; And updating the motion fields for the pixels of the first frame using the motion fields between the first warped reference frame and the second warped reference frame; For a final level of the plurality of levels: warping the first reference frame to the first frame using the updated motion fields to form a final first warped reference frame; Warping the second reference frame to the first frame using the updated motion fields to form a final second warped reference frame; And blending the final first warped reference frame and the second warped reference frame to form a lightflow reference frame.

[00188] Example 16: In the apparatus of example 15, the light flow estimation uses a Lagrange function for each pixel of the frame.

[00189] Example 17: The apparatus of example 16, the method further comprises: for each of the plurality of levels: warping a first reference frame, warping a second reference frame, estimating motion fields, and a motion field. Initializing a Lagrangian parameter of the Lagrange function to a maximum value for a first iteration of updating them; And using the smaller and smaller of a set of possible values for the Lagrange parameter, warping the first reference frame, warping the second reference frame, estimating motion fields, and updating the motion fields. The method further includes performing additional iteration of the step.

[00190] Example 18: In the apparatus of example 16 or 17, estimating the motion fields comprises: calculating derivatives of pixel values of the first warped reference frame and the second warped reference frame with respect to the horizontal axis, the vertical axis, and time. ; Downscaling the derivatives in response to the level being different than the final level; Solving the linear equations representing the Lagrange function using the derivatives.

[00191] Example 19: The apparatus of any of examples 15-18, the method further comprising inter predicting the first frame using the lightflow reference frame.

[00192] Example 20: The apparatus of any of examples 15-19, wherein the processor and the non-transitory storage medium form a decoder.

[00193] The foregoing embodiments, embodiments, and aspects have been described to facilitate understanding of the invention and do not limit the invention. On the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the widest interpretation so as to encompass all such modifications and equivalent structures permitted by law. do.

Claims (21)

Determining a first frame portion of a first frame to be predicted, wherein the first frame is in a video sequence;
Determining a first reference frame from the video sequence for forward inter prediction of the first frame;
Determining a second reference frame from the video sequence for reverse inter prediction of the first frame;
Generating an optical flow reference frame portion for inter prediction of the first frame portion by performing optical flow estimation using the first reference frame and the second reference frame; And
Performing a prediction process for the first frame portion using an optical flow reference frame portion,
Way. The method of claim 1,
Generating the light flow reference frame portion may include:
Performing the light flow estimation by minimizing a Lagrangian function for each pixel of the first frame portion,
Way. The method according to claim 1 or 2,
The light flow estimate generates respective motion fields for the pixels of the first frame portion,
Generating the light flow reference frame portion may include:
Warping pixels of the first reference frame co-located with the first frame portion to the first frame portion using the motion fields to form a first warped reference frame portion. step;
Warping pixels of the second reference frame in parallel with the first frame portion to the first frame portion using the motion fields to form a second warped reference frame portion; And
Blending the first warped reference frame portion and the second warped reference frame portion to form the lightflow reference frame portion;
Way. The method of claim 3, wherein
Blending the first warped reference frame portion and the second warped reference frame portion includes:
The first warped reference frame portion and the second warping using distances between the first reference frame and the second reference frame and between the current frame and each of the first reference frame and the second reference frame. Combining the juxtaposed pixel values by scaling juxtaposed pixel values of a portion of the referenced reference frame,
Way. The method according to claim 3 or 4,
Blending the first warped reference frame portion and the second warped reference frame portion includes:
Combining the collocated pixel values of the first warped reference frame portion and the second warped reference frame portion or a single pixel value of one of the first warped reference frame portion or the second warped reference frame portion Populating pixel positions of the lightflow reference frame portion by using one of
Way. The method according to any one of claims 1 to 5,
The first frame portion includes a current block of the first frame or one of the first frames, and the light flow reference frame portion is a block when the first frame portion includes the current block, and the first frame portion includes: When one frame part includes the first frame,
Way. The method according to any one of claims 1 to 6,
The first reference frame is a reconstructed frame closest to the first frame available for forward inter prediction of the first frame in display order of the video sequence, and the second reference frame is a reverse inter frame of the first frame. The reconstructed frame nearest to the display order for the first frame available for prediction,
Way. The method according to any one of the preceding claims.
The first frame portion is a current block to be decoded,
Performing the prediction process includes:
Using the motion vector used to encode the current block to identify a reference block location;
Adjusting boundaries of the reference block by a length of a subpixel interpolation filter; And
Identifying blocks comprising pixels in the adjusted boundaries of the reference block,
The generating of the light flow reference frame portion may include: light flow estimation on blocks of the first frame collocated with the identified blocks without performing light flow estimation on the remaining blocks of the first frame. Comprising the steps of:
Way. The method according to any one of claims 1 to 8,
The first frame portion is a current block to be encoded,
Generating the light flow reference frame portion may include performing light flow estimation on each block of the first frame as the current block to generate each collocated reference block for the light flow reference frame. Include,
Performing the prediction process includes:
Forming the lightflow reference frame by combining the juxtaposed reference blocks to their respective pixel positions;
Storing the optical flow reference frame in a reference frame buffer; And
Using the lightflow reference frame for motion search for the current block,
Way. As a device,
A processor; And
A non-transitory storage medium comprising instructions executable by the processor to perform a method,
The method is:
Determining a first frame to be predicted in the video sequence;
Determining availability of a first reference frame for forward inter prediction of the first frame and availability of a second reference frame for reverse inter prediction of the first frame;
In response to determining the availability of both the first reference frame and the second reference frame:
Using the first reference frame and the second reference frame as inputs to a light flow estimation process, generating respective motion fields for the pixels of the first frame portion;
Warping a first reference frame portion to the first frame portion using the motion fields to form a first warped reference frame portion, the first reference frame portion being equal to the pixels of the first frame portion; Includes pixels of the collocated first reference frame;
Warping a second reference frame portion into the first frame portion using the motion fields to form a second warped reference frame portion, wherein the second reference frame portion is coupled with the pixels of the first frame portion; Includes pixels of the collocated second reference frame; And
Blending the first warped reference frame portion and the second warped reference frame portion to form an optical flow reference frame portion for inter prediction of the block of the first frame;
Device. The method of claim 10,
The method is:
Performing a prediction process on the block of the first frame using the lightflow reference frame portion;
Device. The method according to claim 10 or 11, wherein
The method is:
Using the lightflow reference frame portion only for single reference inter prediction of blocks of the first frame,
Device. The method according to any one of claims 10 to 12,
Generating each motion field includes:
Calculating an output of a Lagrange function for each pixel of the first frame portion using the first reference frame portion and the second reference frame portion;
Device. The method of claim 13,
Computing the output of the Lagrange function is:
Calculating a first set of motion fields for pixels of the first frame portion using the first value for the Lagrange parameter; And
To calculate the set of refined motion fields for the pixels of the first frame portion, the input of the first motion fields as input to the Lagrange function using a second value for the Lagrange parameter. Using the set,
The second value for the Lagrange parameter is less than the first value for the Lagrange parameter, wherein the first warped reference frame and the second warped reference frame are warped using the improved set of motion fields. felled,
Device. As a device,
A processor; And
A non-transitory storage medium comprising instructions executable by the processor to perform a method,
The method is:
By initializing motion fields for pixels of a first frame portion at a first processing level for light flow estimation, wherein the first processing level represents downscaled motion within the first frame portion and indicates one of a plurality of levels. Including a level, wherein the first reference frame portion from the video sequence and the second reference frame portion of the video sequence are used to generate a light flow reference frame portion for inter prediction of a block of the first frame of the video sequence. step;
For each level of the plurality of levels:
Warping the first reference frame portion to the first frame portion using the motion fields to form a first warped reference frame portion;
Warping the second reference frame portion to the first frame portion using the motion fields to form a second warped reference frame portion;
Estimating motion fields between the first warped reference frame portion and the second warped reference frame portion using the light flow estimation; And
Updating the motion fields for pixels of the first frame portion using the motion fields between the first warped reference frame portion and the second warped reference frame portion;
For the final level of the plurality of levels:
Warping the first reference frame portion to the first frame portion using the updated motion fields to form a final first warped reference frame portion;
Warping the second reference frame portion to the first frame portion using the updated motion fields to form a final second warped reference frame portion; And
Blending the final warped reference frame portion with the second warped reference frame portion to form the lightflow reference frame portion;
Device. The method of claim 15,
The light flow estimation uses a Lagrange function for each pixel of the first frame portion,
Device. The method of claim 16,
The method, for each level of the plurality of levels:
The Lagrang for a first iteration of warping the first reference frame portion, warping the second reference frame portion, estimating the motion fields, and updating the motion fields. Initializing the Lagrangian parameters of the support function to a maximum value; And
Warping the first reference frame portion using the smaller and smaller of a set of possible values for the Lagrange parameter, warping the second reference frame portion, estimating the motion fields, And performing an additional iteration of updating the motion fields.
Device, The method according to claim 16 or 17,
Estimating the motion fields includes:
Calculating derivatives of pixel values of the first warped reference frame portion and the second warped reference frame portion with respect to a horizontal axis, a vertical axis, and time;
Downscaling the derivatives in response to the level being different than the final level;
Solving linear equations representing the Lagrange function using the derivatives,
Device. The method according to any one of claims 15 to 18,
The method is:
Inter predicting a current block of the first frame using the lightflow reference frame portion;
Device. The method according to any one of claims 15 to 19,
The processor and the non-transitory storage medium form a decoder;
Device. As a device,
A processor; And
A non-transitory storage medium comprising instructions executable by the processor to perform a method,
The method is:
Determining a first frame portion of a first frame to be predicted, wherein the first frame is in a video sequence;
Determining a first reference frame from the video sequence for forward inter prediction of the first frame;
Determining a second reference frame from the video sequence for reverse inter prediction of the first frame;
Generating an optical flow reference frame portion for inter prediction of the first frame portion by performing optical flow estimation using the first reference frame and the second reference frame; And
Performing a prediction process for a first frame portion using the lightflow reference frame portion;
Device.

Images