HK1241609B - Image decoding method, video decoder, and non-transitory computer-readable storage medium - Google Patents

Description

Image decoding method, video decoder, and non-transitory computer-readable storage medium

This application is a divisional application of PCT application with international application number PCT/US2012/033605 filed on April 13, 2012, and invention name “Multi-color channel multivariate regression prediction operator”. The date on which the PCT application entered the Chinese national phase is October 11, 2013, and the national application number is 201280018070.9.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Patent Application No. 61/475,359, filed April 14, 2011, which is hereby incorporated by reference herein in its entirety.

This application is also related to co-pending U.S. Provisional Patent Application No. 61/475,372, filed April 14, 2011, which is incorporated herein by reference in its entirety.

Technical Field

The present invention relates generally to images and, more particularly, to a multi-color channel, multivariate regression prediction operator between high dynamic range images and standard dynamic range images.

Background Art

As used herein, the term "dynamic range" (DR) refers to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., brightness) in an image, for example, from the darkest darks to the brightest highlights. In this sense, DR refers to intensities that are "relative to the scene." DR may also refer to the ability of a display device to appropriately or approximately present a range of intensities of a particular width. In this sense, DR refers to intensities that are "relative to the display." Unless a particular meaning is explicitly stated at any point in the description herein as having a specific meaning, it should be inferred that the term can be used (e.g., interchangeably) in either sense.

As used herein, the term high dynamic range (HDR) refers to a DR width that spans a certain 14-15 orders of magnitude across the human visual system (HVS). For example, a substantially normal, well-adapted person (e.g., in one or more statistical, biometric, or ophthalmological senses) has an intensity range that spans about 15 orders of magnitude. An adapted person can perceive faint light sources, as few as a handful of photons. However, the same person can perceive the nearly painful glare of the midday sun in the desert, sea, or snow (or even look at the sun, however briefly, to prevent damage). While this span is achievable for "adapted" people, their HVS, for example, has a time period for reset and adjustment.

In contrast, the DR of the extended width in the intensity range that humans can perceive simultaneously can be shortened to a certain extent compared to HDR. As used herein, the terms "visual dynamic range" or "variable dynamic range" (VDR) may refer individually or interchangeably to the DR that can be perceived simultaneously by an HVS. As used herein, VDR may refer to a DR that spans 5-6 orders of magnitude. Therefore, although HDR associated with real scenes may be narrowed to a certain extent, VDR still represents a wider DR width. As used herein, the term "simultaneous dynamic range" may refer to VDR.

Until recently, displays had a significantly narrower DR than either HDR or VDR. Televisions (TVs) and computer monitors using conventional cathode ray tube (CRT), liquid crystal display (LCD) with constant fluorescent white backlighting, or plasma screen technology were limited to about three orders of magnitude in their DR rendering capabilities. These traditional displays were therefore characterized by low dynamic range (LDR), also known as standard dynamic range (SDR) for VDR and HDR.

However, advances in their underlying technologies have allowed newer display designs to present image and video content with significant improvements in various quality characteristics relative to that content presented on less modern displays. For example, newer display devices may be capable of presenting high-definition (HD) content and/or content that is scalable according to various display capabilities (such as image scalers). Furthermore, some newer displays are capable of presenting content at a higher DR than the SDR of conventional displays.

For example, some modern LCD displays have a backlight unit (BLU) that includes an array of light-emitting diodes (LEDs). The LEDs of the BLU array can be modulated separately from the modulation of the polarization state of the active LCD elements. This dual modulation approach is extendable (e.g., to an N-modulation layer, where N is an integer greater than 2), such as by using a controllable intermediate layer between the BLU array and the LCD screen elements. The LED-based BLU and dual (or N) modulation effectively increase the display-related DR of LCD monitors having this feature.

With respect to traditional SDR displays, these so-called "HDR displays" (although in reality, their capabilities may be closer to the range of VDR) and their possible DR extensions represent a significant advancement in the ability to display images, video content, and other video information. The color gamut that these HDR displays can present can also significantly exceed the color gamut of most traditional displays, even to the point of being able to present wide color gamut (WCG). Scene-related HDR or VDR and WCG image content, such as that produced by "next-generation" film and TV cameras, can now be more realistically and effectively displayed using "HDR" displays (hereinafter referred to as "HDR displays").

In the context of scalable video coding and HDTV technologies, extending image DR typically involves a bifurcated approach. For example, HDR content associated with a scene, captured by a new HDR-capable camera, can be used to generate an SDR version of the content, which can be displayed on a conventional SDR display. In one approach, generating an SDR version from the captured VDR version may involve applying a global tone mapping operator (TMO) to intensity-related (e.g., brightness) pixel values in the HDR content. In a second approach (as described in International Patent Application No. PCT/US2011/048861, filed on August 23, 2011, and incorporated herein by reference for all purposes), generating the SDR image may involve applying a reversible operator (or prediction operator) to the VDR data. To conserve bandwidth or for other considerations, transmitting the actual captured VDR content may not be the best approach.

Therefore, an inverse tone mapping operator (iTMO) that is the inverse of the initial TMO or an inverse operator of the initial prediction operator can be applied to the generated SDR content version, which allows a version of the VDR content to be predicted. The predicted VDR content version can be compared with the initially acquired HDR content. For example, subtracting the predicted VDR version from the initial VDR version can produce a residual image. The encoder can send the generated SDR content as a base layer (BL) and package the generated SDR content version, any residual image, and the iTMO or other prediction operator as an enhancement layer (EL) or as metadata.

Sending the EL and metadata (with its SDR content, residual image and prediction operator) into the bitstream typically takes up less bandwidth than sending both HDR content and SDR content directly into the bitstream. A compatible decoder receiving the bitstream sent by the encoder can decode the SDR and present it on a conventional display. However, a compatible decoder can also use the residual image, the iTMO prediction operator or the metadata in order to calculate a predetermined version of the HDR content from them for use on a more versatile display. The object of the present invention is to provide a new method for generating a prediction operator that allows efficient encoding, transmission and decoding of VDR data using the corresponding SDR data.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, none of the approaches described in this section should be considered prior art by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not be considered established in any prior art based on this section, unless otherwise indicated.

Summary of the Invention

The following is a brief overview of the present invention to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention, is not intended to identify key or important aspects of the present invention, or to limit the scope of the present invention. Its purpose is simply to present certain concepts in a simplified form as a foundation for the detailed description that follows.

According to one aspect of the present invention, there is provided an image decoding method using a processor, the method comprising: receiving a first image; receiving metadata, the metadata comprising prediction parameters for a multi-channel, multivariate regression (MMR) prediction model, wherein the MMR model is adapted to predict a second image based on the first image; and applying the first image and the prediction parameters to the MMR prediction model to generate an output image that approximates the second image, wherein pixel values of at least one color component in the output image are calculated based on pixel values of at least two color components in the first image, wherein the MMR model comprises a first-order MMR model with cross multiplication according to the following formula:

where represents the predicted three color components of the i-th pixel of the output image, s _i = [s _i1 s _i2 s _i3 ] represents the three color components of the i-th pixel of the first image, is a 3×3 prediction parameter matrix and n is a 1×3 prediction parameter vector according to the following formula:

and n = [n ₁₁ n ₁₂ n ₁₃ ],

According to the following formula, it is a 4×3 prediction parameter matrix:

and

sc _i is a 1×4 vector according to sc _i = [s _i1 ·s _i2 s _i1 ·s _i3 s _i2 ·s _i3 s _i1 ·s _i2 ·s _i3 ].

According to one aspect of the present invention, a video decoder is provided, the decoder comprising: an input for receiving a first image and metadata, wherein the metadata comprises prediction parameters for a multi-channel, multivariate regression (MMR) prediction model, wherein the MMR model is adapted to predict a second image based on the first image; a processor; and a memory for storing an output image, wherein the processor is configured to apply the first image and the prediction parameters to the MMR prediction model to generate an output image that approximates the second image, wherein pixel values of at least one color component in the output image are calculated based on pixel values of at least two color components in the first image, wherein the MMR model comprises a first-order MMR model with cross-multiplication according to the following formula:

where represents the predicted three color components of the i-th pixel of the output image, s _i =[s _i1 s _i2 s _i3 ] represents the three color components of the i-th pixel of the first image,

According to the following formula, is a 3×3 prediction parameter matrix and n is a 1×3 prediction parameter vector:

and n = [n ₁₁ n ₁₂ n ₁₃ ],

According to the following formula, it is a 4×3 prediction parameter matrix:

and

sc _i is a 1×4 vector according to sc _i = [s _i1 ·s _i2 s _i1 ·s _i3 s _i2 ·s _i3 s _i1 ·s _i2 ·s _i3 ].

According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer-executable program for executing the above-mentioned image decoding method using one or more processors is stored.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals indicate similar elements and in which:

FIG1 depicts an example data flow of a VDR-SDR system according to an embodiment of the present invention;

FIG2 depicts an example VDR encoding system according to an embodiment of the present invention;

FIG3 depicts the input and output interfaces of a multivariate multiple regression prediction operator according to an embodiment of the present invention;

FIG4 depicts an example multivariate multiple regression prediction process according to an embodiment of the present invention;

FIG5 depicts an example process for determining a model for a multivariate multiple regression prediction operator according to an embodiment of the present invention;

FIG6 depicts an example image decoder with a prediction operator operating in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

This paper describes inter-color image prediction based on multivariate multivariate regression modeling. Given a pair of corresponding VDR and SDR images, i.e., images representing the same scene but at different dynamic range levels, this section describes a method that allows an encoder to approximate the VDR image based on a multivariate multivariate regression (MMR) prediction operator and the SDR image. In the following description, for purposes of explanation, numerous specific details are set forth to provide a full understanding of the present invention. However, it will be apparent that the present invention can be implemented without these specific details. In other cases, in order to avoid unnecessarily obscuring, obscuring, or confusing the present invention, known structures and devices are not described in detail.

Overview

The exemplary embodiments described herein relate to encoding images with high dynamic range.The embodiments create an MMR predictor that allows a VDR image to be expressed with respect to its corresponding SDR representation.

Example VDR-SDR System

FIG1 depicts an example data flow in a VDR-SDR system 100 according to an embodiment of the present invention. An HDR image or video sequence is acquired using an HDR camera 110. After acquisition, the acquired image or video is processed through a mastering process to create a target VDR image 125. The mastering process may include multiple processing steps, such as editing, primary and secondary color correction, color conversion, and noise filtering. The VDR output 125 of this processing represents the intended display of the acquired image on the target VDR display.

The mastering process may also output a corresponding SDR image 145 that represents the intended purpose of the person responsible for how the acquired image will be displayed on a legal SDR display. The SDR output 145 may be provided directly from the mastering circuitry 120 or may be generated by a separate VDR to SDR converter 140.

In this example embodiment, the VDR 125 and SDR 145 signals are input to an encoder 130. The purpose of the encoder 130 is to create an encoded bitstream that reduces the bandwidth required to transmit the VDR and SDR signals and also allows the corresponding decoder 150 to decode and render the SDR or VDR signals. In an example implementation, the encoder 130 can be a layered encoder, such as one of those defined by the MPEG-2 and H.264 coding standards, which represents its output as a base layer, an optional enhancement layer, and metadata. The term "metadata" as used herein refers to any auxiliary information that is transmitted as part of the encoded bitstream and helps the decoder render the decoded image. Such metadata may include (but is not limited to) such data as color space or color gamut information, dynamic range information, tone mapping information, or MMR prediction operators, such as those described herein.

At the receiver, the decoder 150 uses the received encoded bitstream and metadata to render either an SDR image or a VDR image, depending on the capabilities of the target display. For example, an SDR display may render an SDR image using only the base layer and metadata. In contrast, a VDR display may render a VDR signal using information and metadata from all input layers.

FIG2 illustrates in more detail an example implementation of an encoder 130 including the method of the present invention. In FIG2 , SDR′ represents an enhanced SDR signal. SDR video is now 8-bit, 4:2:0, ITU Rec.709 data. SDR′ may have the same color space (primaries and white point) as SDR, but may use high precision, say 12 bits per pixel, for all color components at full spatial resolution (e.g., 4:4:4 RGB). Based on FIG2 , SDR can be easily derived from the SDR′ signal using a set of forward transforms, which may include quantization from, say, 12 bits per pixel to 8 bits per pixel, a color conversion from, say, RGB to YUV, and color subsampling from, say, 4:4:4 to 4:2:0. The SDR output of the converter 210 is applied to a compression system 220. Depending on the application, the compression system 220 may be lossy (such as H.264 or MPEG-2) or lossless. The output of the compression system 220 may be transmitted as a base layer 225. To reduce the offset between the encoded and decoded signals, it is not uncommon for encoder 130 to immediately follow compression process 220 with a corresponding decompression process 230 and an inverse transform 240 corresponding to forward transform 210. Thus, predictor 250 may have the following inputs: VDR input 205, and SDR' signal 245 (which corresponds to an SDR' signal when received by a corresponding decoder) or input SDR' 207. Using the input VDR and SDR' data, predictor 250 creates signal 257, which represents an approximation or estimate of input VDR 205. Adder 260 subtracts predicted VDR 257 from original VDR 205 to form output residual signal 265. Residual 265 may then be encoded by another lossy or lossless encoder (not shown) and transmitted to a decoder as an enhancement layer.

Predictor 250 may also provide prediction parameters used in the prediction process as metadata 255. Since the prediction parameters may vary during the encoding process, for example, from frame to frame or scene to scene, these metadata may be transmitted to the decoder as part of the data that also includes the base layer and enhancement layer.

Since both the VDR 205 and the SDR' 207 represent the same scene, but for different displays with different characteristics (such as dynamic range and color gamut), it is expected that the two signals have a very close correlation. In an example embodiment of the present invention, a new multivariate, multiple regression (MMR) prediction operator 250 is developed that allows the SDR' signal corresponding to the VDR signal and a multivariate MMR operator to be used to predict the input VDR signal.

Example Prediction Model

Figure 3 shows the input and output interfaces of an MMR prediction operator 300 according to an example implementation of the present invention. According to Figure 3, the prediction operator 330 receives input vectors v 310 and s 320, which represent VDR image data and SDR image data, respectively, and the prediction operator 330 outputs a vector 340, which represents the predicted value of the input v.

Example notation and nomenclature

The three color components of the i-th pixel in the SDR image 320 are labeled:

s _i =[s _i1 s _i2 s _i3 ]. (1)

The three color components of the i-th pixel in the VDR input 310 are labeled:

V _i =[v _i1 v _i2 v _i3 ]. (2)

The predicted three color components of the i-th pixel in the predicted VDR 340 are labeled:

The total number of pixels in one color component is denoted as p.

In equations (1-3), color pixels can be RGB, YUV, YCbCr, XYZ, or any other color representation. Although equations (1-3) assume three color representations for each pixel in each image or video frame, as will be shown below, the methods described herein can be easily extended to image and video representations with more than three color components per pixel, or to image representations where one of the inputs may have pixels with a different number of color representations than the other inputs.

First-order model (MMR-1)

Using the multivariate multiple regression (MMR) model, the first-order prediction model can be expressed as:

where is a 3×3 matrix and n is a 1×3 vector defined as:

and n＝[n ₁₁ n ₁₂ n ₁₃ ]. (5)

It should be noted that this is a multi-color channel prediction model. In equation (4), each color component is represented as a linear combination of all color components in the input. In other words, unlike other single-channel color prediction operators (where each color channel is processed on its own and independently of each other for each output pixel), this model considers all color components of the pixel and thus fully exploits the correlation and redundancy between colors.

By using a single matrix-based representation, equation (4) can be simplified to:

in,

and S′ _i = [1 s _i1 s _i2 s _i3 ] (7)

By grouping together all p pixels of a frame (or other suitable fragment or portion of the input), we can have the following matrix representation:

in,

and represent the input and predicted output data, S' is a p x 4 data matrix, is a p x 3 matrix, and N ⁽¹⁾ is a 4 x 3 matrix. As used herein, N ⁽¹⁾ is interchangeably referred to as a multivariate operator or a prediction matrix.

Based on this linear system of equation (8), the MMR system can be formulated as two different problems: (a) a least squares problem, or (b) a total least squares problem; both problems can be solved using known numerical methods. For example, using the least squares method, the problem for solving M can be formulated as minimizing the residual or prediction mean square error, or

where V is a p-by-3 matrix formed using the corresponding VDR input data.

Given equations (8) and (10), the optimal solution of M ⁽¹⁾ is given by

M ⁽¹⁾ = (S′ ^T S′) ^-1 S′ ^T V, (11)

Where S' ^T represents the transposed matrix of S', and S' ^T S' is a 4×4 matrix.

If S' is full column rank, for example,

rank(S′)＝4≤p,

Then, a variety of alternative numerical techniques (including SVD, QR or LU decomposition) can be used to solve M ⁽¹⁾ .

Second-order model (MMR-2)

Equation (4) represents a first-order MMR prediction model. Higher-order predictions can also be considered as described below.

The second-order prediction MMR model can be expressed as:

where is a 3×3 matrix,

as well as

Equation (12) can be simplified by using a single prediction matrix,

in,

and

By grouping all p pixels together, the following matrix representation can be defined:

in,

The same optimization and solution methods described in the previous section can be used to solve equation (14). The optimal solution to the least squares problem M ⁽²⁾ is

M ⁽²⁾ = (S ^(2)T S ⁽²⁾ ) ^-1 S(2) ^T V, (19)

where S ^(2)T S ⁽²⁾ is now a 7×7 matrix.

Third-order or higher-order MMR models can also be constructed in a similar manner.

First-order model with cross-multiplication (MMR-1C)

In an alternative MMR model, the first-order prediction model of Equation (4) can be enhanced to include cross-multiplication between the color components of each pixel as follows:

where is a 3×3 matrix and n is a 1×3 vector, both as defined in equation (5), and

And sc _i =[s _i1 ·s _i2 s _i1 ·s _i3 s _i2 ·s _i3 s _i1 ·s _i2 ·s _i3 ]. (twenty one)

Following the same approach as before, the MMR-IC model of equation (20) can be simplified by utilizing a single prediction matrix MC as follows:

in,

and

By grouping all p pixels together, a simplified matrix representation can be derived as follows:

in,

and

SC is a p x (1 + 7) matrix and can be solved using the same least squares solution described previously.

Second-order model with cross-multiplication (MMR-2C)

The first-order MMR-1C can be extended to also include second-order data. For example,

in,

and

And the remaining components of equation (27) are the same as those defined previously in equation (5-26).

As before, Equation (27) is simplified by using the simple prediction matrix MC ⁽²⁾ ,

in,

and

By grouping all p pixels together, we can have a simplified matrix representation

in,

And SC ⁽²⁾ is a px(1+2*7) matrix and the same least squares solution as described before can be applied.

A third-order or higher-order model with cross-multiplication parameters can be constructed in a similar manner. Alternatively, as described in "Chaper 5.4.3 of "Digital Color Imaging Handbook", CRC Press, 2002, Edited by Gaurav Sharma", the following formula can also be used to describe the K-order representation of the MMR cross-multiplication model.

and

Here, K represents the highest order of the MMR prediction operator.

MMR based on spatial expansion (MMR-C-S)

In all the MMR models described so far, the value of the predicted pixel depends only on the corresponding, usually configured input value s _i . In the case of MMR-based prediction, it is also possible to benefit from taking into account data from neighboring pixels. This approach corresponds to integrating any linear type processing of the input in the spatial domain (such as FIR-type filtering) into the MMR model.

If all eight possible neighboring pixels in an image are considered, this method can add up to eight more first-order variables for each color component into the prediction matrix M. However, in practice, it is usually sufficient to add only the prediction variables corresponding to the two horizontally neighboring pixels and the two vertically neighboring pixels, ignoring the diagonal neighbors. This adds up to four variables for each color component into the prediction matrix, i.e., the four variables corresponding to the top, left, bottom, and right pixels. Similarly, parameters corresponding to higher-order numbers of neighboring pixel values can also be added.

To simplify the complexity and computational requirements of such MMR spatial models, one can consider adding spatial extensions to the traditional model for only a single color component, such as the luminance component (as in luma-chrominance representation) or the green component (as in RGB representation). For example, assuming that spatial-based pixel prediction is added only for the green color component, the general representation of the predicted green output pixel value according to equations (34-36) would be

First-order model with spatial extension (MMR-1-S)

As another example implementation, the first-order MMR model (MMR-1) of equation (4) may be considered again, but now enhanced to include spatial extension in one or more color components. For example, when applied to the four neighboring pixels of each pixel in the first color component,

where is a 3×3 matrix and n is a 1×3 vector, both as defined in equation (5),

and

Where m in equation (39) represents the number of columns in the input frame with m columns and n rows, or m×n=p total pixels. Equation (39) can be easily extended to apply these methods to other color components and to alternative neighbor pixel constructions.

Following the same approach as before, equation (38) can be easily formulated as a system of linear equations

It can be solved as described before.

Applications of VDR signals with more than three primary colors

All proposed MMR prediction models can be easily extended to signal spaces with more than three primary colors. As an example, consider the case where the SDR signal has three primary colors, say RGB, but the VDR signal is defined in a P6 color space with six primary colors. In this case, equations (1-3) can be rewritten as

s _i =[s _i1 s _i2 s _i3 ], (41)

v _i = [v _i1 v _i2 v _i3 v _i4 v _i5 v _i6 ], (42)

as well as

As before, the number of pixels in a color component is denoted as p. Now consider the first-order MMR prediction model (MMR-1) of equation (4),

Now it is a 3×6 matrix and n is a 1×6 vector, given by

And n=[n ₁₁ n ₁₂ n ₁₃ n ₁₄ n ₁₅ n ₁₆ ]. (46)

Equation (41) can be expressed using a single prediction matrix M ⁽¹⁾ as:

in,

and s′ _i = [1 s _i1 s _i2 s _i3 ]. (48)

By grouping all p pixels together, the prediction problem can be described as

in,

is a p×6 matrix, is a p×4 matrix, and M ⁽¹⁾ is a 4×6 matrix.

Higher-order MMR prediction models can also be expanded in a similar manner and the solution of the prediction matrix can be obtained via the method described previously.

Example processing of multi-channel and multivariate regression prediction

FIG4 illustrates an example process of multi-channel multiple regression prediction according to an example implementation of the present invention.

The process begins at step 410, where a predictor (such as predictor 250) receives input VDR and SDR signals. Given the two inputs, the predictor determines which MMR model to select in step 420. As previously described, the predictor can select from a variety of MMR models, including but not limited to: a first-order model (MMR-1), a second-order model (MMR-2), a third-order or higher-order model, a first-order model with cross-multiplication (MMR-1C), a second-order model with cross-multiplication (MMR-2C), a third-order model with cross-multiplication (MMR-3C), a third-order or higher-order model with cross-multiplication, or any of the above models with added spatial expansion.

The selection of the MMR model can be done using a variety of methods that take into account a number of criteria, including prior knowledge about the SDR and VDR inputs, available computational and memory resources, and target coding efficiency. Figure 5 shows an example implementation of step 420 based on the requirement that the residual be lower than a predetermined threshold.

As described previously, a set of linear equations of the form

Where M is the prediction matrix.

At step 430, a variety of numerical methods can be used to solve for M. For example, under the constraint of minimizing the mean square of the residual between V and its estimate,

M＝(S ^T S) ^-1 S ^T V。 (51)

Finally, at step 440, the operator outputs and M are predicted using equation (50).

FIG5 shows an example process 420 for selecting an MMR model during prediction. In step 510, the predictor 250 may start with an initial MMR model, such as an MMR model that has been used in a previous frame or scene, for example, a second-order model (MMR-2), or the simplest possible model, such as MMR-1. After solving for M, in step 520, the predictor calculates the prediction error between the input V and its predicted value. In step 530, if the prediction error is below a given threshold, the predictor selects the existing model and stops the selection process (540). Otherwise, in step 550, a check is made to see whether a more complex model should be used. For example, if the current model is MMR-2, the predictor may decide to use MMR-2-C or MMR-2-C-S. As described above, this decision may depend on a variety of criteria, including the value of the prediction error, processing power requirements, and target coding efficiency. If using a more complex model is feasible, the new model is selected in step 560 and the process returns to step 520. Otherwise, the predictor uses the existing model (540).

The prediction process 400 may be repeated at various intervals as needed to maintain coding efficiency while utilizing available computing resources. For example, when encoding a video signal, the process 400 may be repeated for each frame, a group of frames, or each time the prediction residual exceeds a certain threshold, based on each predefined video segment size.

The prediction process 400 can also use all available input pixels or a subsample of these pixels. In one example implementation, pixels from every k-th pixel row and every k-th pixel column of the input data can be used, where k is an integer equal to or greater than 2. In another example implementation, a decision can be made to skip input pixels that are below a particular clipping threshold (e.g., very close to 0) or pixels that are above a particular saturation threshold (e.g., very close to ²ⁿ -1 for n-bit data). In another implementation, a combination of such subsampling and thresholding techniques can be used to reduce the pixel sample size and accommodate the computational constraints of a particular implementation.

Image decoding

Embodiments of the present invention may be implemented at an image encoder or at an image decoder. FIG6 shows an example implementation of a decoder 150 according to an embodiment of the present invention. The decoding system 600 receives an encoded bitstream that may have a base layer 690, an optional enhancement layer (or residual) 665, and metadata 645, which are extracted after decompression 630 and multiple inverse transforms 640. For example, in a VDR-SDR system, the base layer 690 may represent an SDR representation of the encoded signal, and the metadata 645 may include information about the MMR prediction model and corresponding prediction parameters used in the encoder predictor 250. In an example implementation, when the encoder uses an MMR predictor according to the method of the present invention, the metadata may include an identification of the model used (e.g., MMR-1, MMR-2, MMR-2C, etc.) and all matrix coefficients associated with the specific model. Given the base layer 690 and the color MMR-related parameters extracted from the metadata 645, the prediction operator 650 can calculate the predicted VDR image using any of the corresponding equations described herein. For example, if the identified model is MMR-2C, it can be calculated using equation (32). If there is no residual, or the residual is negligible, the predicted value 680 can be directly output as the final VDR image. Otherwise, in the adder 660, the output of the prediction operator (680) is added to the residual 665 to output the VDR signal 670.

Example Computer System Implementation

Embodiments of the present invention may be implemented by a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD)), a discrete-time or digital signal processor (DSP), an application-specific IC (ASIC) and/or an apparatus comprising one or more such systems, devices or components. The computer and/or IC may execute, control or run instructions related to MMR-based predictions such as those described herein. The computer and/or IC may calculate any of a variety of parameters or values related to MMR predictions as described herein. The image and video dynamic range extension embodiments may be implemented in hardware, software, firmware and various combinations thereof.

A specific implementation of the present invention includes a computer processor that runs software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, an encoder, a set-top box, a code converter, etc. can implement the above-mentioned MMR-based prediction method by running software instructions in a program memory accessible to the processor. The present invention is also provided in the form of a program product. The program product may include any medium that carries a set of computer-readable signals including instructions that, when executed by a data processor, cause the data processor to perform the method of the present invention. The program product according to the present invention may be in any of a variety of forms. The program product may include, for example, physical media, such as magnetic data storage media including floppy disks and hard drives, optical data storage media including CD ROMs and DVDs, electronic data storage media including ROMs and flash RAMs, etc. The computer-readable signals on the program product may optionally be compressed or encrypted.

Unless otherwise indicated, references to the components referred to above (e.g., software modules, processors, accessories, devices, circuits, etc.) (including references to "means") should be interpreted as including any components that are equivalent to that component and perform the function of the described component (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the present invention.

Equivalence, extension, substitution, and diversification

Thus described are example embodiments relating to the application of MMR prediction in the encoding of VDR and SDR images. In the foregoing description, embodiments of the invention have been described with reference to many specific details that will vary from implementation to implementation. Therefore, the only indication of the technical solution of the invention, and of the technical solution to which the applicant regards the invention relates, is the set of claims arising from this application, in the specific form in which such claims are published, including subsequent amendments. Any express definitions given herein for terms contained in such claims shall encompass the meaning of such terms as used in the claims. Therefore, no limitation, element, attribute, feature, advantage or characterizing that is not expressly recited in a claim should in any way limit the scope of such claim. The specification and drawings are therefore to be regarded in an illustrative and not limiting sense.

The first set of notes:

1. A method comprising:

receiving a first image and a second image, wherein the second image has a different dynamic range than the first image;

Selecting a multi-channel, multiple regression (MMR) prediction model from the family of MMR models;

Solve for the prediction parameters of the selected MMR model;

calculating an output image representing a predicted value of the first image using the second image and prediction parameters of the MMR model;

Output the predicted parameters of the MMR model and the output image.

2. The method according to Note 1, wherein the first image comprises a VDR image and the second image comprises an SDR image.

3. The method according to Note 1, wherein the MMR model is at least one of a first-order MMR model, a second-order MMR model, a third-order MMR model, a first-order MMR model with cross-multiplication, a second-order MMR model with cross-multiplication, or a third-order MMR model with cross-multiplication.

4. The method according to Note 3, wherein any one of the MMR models further includes prediction parameters involving adjacent pixels.

5. The method according to Note 4, wherein the adjacent pixels considered include left adjacent pixels, right adjacent pixels, top adjacent pixels, and bottom adjacent pixels.

6. The method according to Note 2, wherein pixels in the VDR image have more color components than pixels in the SDR image.

7. The method according to Note 1, wherein solving the prediction parameters of the selected MMR model further includes applying a numerical method that minimizes the mean square error between the first image and the output image.

8. The method according to note 1, wherein selecting the MMR prediction model from the family of MMR models further comprises an iterative selection process, comprising:

(a) Select and apply the initial MMR model;

(b) calculating a residual error between the first image and the output image;

(c) If the residual error is less than a threshold and no other MMR model is available, select the existing MMR model; otherwise, select a new MMR model that is different from the previous model; and return to step (b).

9. An image decoding method, comprising:

receiving a first image having a first dynamic range;

receiving metadata, wherein the metadata defines an MMR prediction model and corresponding prediction parameters of the MMR prediction model;

The first image and the prediction parameters are applied to the MMR prediction model to calculate an output image representing a predicted value for a second image, wherein the second image has a dynamic range different from a dynamic range of the first image.

10. The method according to Note 9, wherein the MMR model is at least one of a first-order MMR model, a second-order MMR model, a third-order MMR model, a first-order MMR model with cross-multiplication, a second-order MMR model with cross-multiplication, or a third-order MMR model with cross-multiplication.

11. The method according to Note 10, wherein any one of the MMR models further includes prediction parameters involving neighboring pixels.

12. The method according to Note 9, wherein the first image comprises an SDR image and the second image comprises a VDR image.

13. A device comprising a processor and configured to perform any one of the methods described in Notes 1-12.

14. A computer-readable storage medium storing computer-executable instructions for executing the method according to any one of Notes 1 to 12.

Second set of notes:

1. A method comprising:

A plurality of multi-channel, multivariate regression (MMR) prediction models are provided, each MMR prediction model being adapted to approximate an image having a first dynamic range according to,

an image having a second dynamic range, and

prediction parameters of each of the MMR prediction models obtained by applying inter-color image prediction;

receiving a first image and a second image, wherein the second image has a different dynamic range than the first image;

selecting a multi-channel, multiple regression (MMR) prediction model from the plurality of MMR models;

determining values of prediction parameters of the selected MMR model;

calculating an output image that approximates the first image based on the second image and the determined values of prediction parameters applied to the selected MMR prediction model;

Outputting the determined values of the prediction parameters and the calculated output image, wherein the plurality of MMR models includes a first-order multi-channel, multivariate regression prediction model incorporating cross-multiplications between color components of each pixel according to the following formula,

wherein represents the three predicted color components of the i-th pixel of the first image,

s _i =[s _i1 s _i2 s _i3 ] represents the three color components of the i-th pixel of the second image,

According to the following formula, is a 3×3 matrix and n is a 1×3 vector

and n = [n ₁₁ n ₁₂ n ₁₃ ],

sc _i =[s _i1 ·s _i2 s _i1 ·s _i3 s _i2 ·s _i3 s _i1 ·s _i2 ·s _i3 ], and

The prediction parameters of the first-order multi-channel, multivariate regression prediction model are numerically obtained by minimizing the mean square error between the first image and the output image.

2. The method according to Note 1, wherein the first image comprises a VDR image and the second image comprises an SDR image.

3. The method according to Note 1, wherein the selected MMR prediction model is at least one of a first-order MMR model, a second-order MMR model, a third-order MMR model, a first-order MMR model with cross-multiplication, a second-order MMR model with cross-multiplication, or a third-order MMR model with cross-multiplication.

4. The method according to Note 3, wherein any one of the MMR models further includes prediction parameters involving adjacent pixels.

5. The method according to Note 4, wherein the adjacent pixels include left adjacent pixels, right adjacent pixels, top adjacent pixels, and bottom adjacent pixels.

6. The method according to Note 2, wherein pixels in the VDR image have more color components than pixels in the SDR image.

7. The method according to Supplementary Note 1, wherein selecting the MMR prediction model from the plurality of MMR prediction models further comprises an iterative selection process, comprising:

(a) Select and apply an initial MMR prediction model;

(b) calculating a residual error between the first image and the output image;

(c) If the residual error is less than the error threshold and no other MMR prediction model can be selected, select the initial MMR model; otherwise, select a new MMR prediction model from the multiple MMR prediction models, and the new MMR prediction model is different from the previously selected MMR prediction model; and return to step (b).

8. An image decoding method, comprising:

receiving a first image having a first dynamic range;

Receive metadata, wherein the metadata includes

a multivariate regression (MMR) prediction model adapted to approximate a second image having a second dynamic range according to,

the first image, and

prediction parameters of the MMR prediction model obtained by applying inter-color image prediction, the metadata further comprising previously determined values of the prediction parameters, and

applying the first image and previously determined values of the prediction parameters to the MMR prediction model to calculate an output image for approximating the second image, wherein the second dynamic range is different from the first dynamic range, wherein the MMR prediction model is a first-order multi-channel, multivariate regression prediction model incorporating cross-multiplications between color components of each pixel according to the following formula,

wherein represents the three predicted color components of the i-th pixel of the first image,

s _i =[s _i1 s _i2 s _i3 ] represents the three color components of the i-th pixel of the second image,

According to the following formula, is a 3×3 matrix and n is a 1×3 vector

and n = [n ₁₁ n ₁₂ n ₁₃ ],

sc _i =[s _i1 ·s _i2 s _i1 ·s _i3 s _i2 ·s _i3 s _i1 ·s _i2 ·s _i3 ], and

9. The method according to Note 8, wherein the first-order MMR prediction model is expanded into a second-order MMR prediction model or a third-order MMR prediction model with pixel cross multiplication.

10. The method according to Note 8 or 9, wherein the MMR prediction model further includes prediction parameters involving adjacent pixels.

11. The method according to Note 8, wherein the first image comprises an SDR image and the second image comprises a VDR image.

12. A device comprising a processor and configured to perform any one of the methods described in Notes 1-11.

13. A computer-readable storage medium storing computer-executable instructions for executing the method according to any one of Notes 1-11.

14. A method comprising:

A plurality of multi-channel, multivariate regression (MMR) prediction models are provided, each MMR prediction model being adapted to approximate an image having a first dynamic range according to,

an image having a second dynamic range, and

prediction parameters of each of the MMR prediction models obtained by applying inter-color image prediction;

receiving a first image and a second image, wherein the second image has a different dynamic range than the first image;

selecting a multi-channel, multiple regression (MMR) prediction model from the plurality of MMR models;

determining values of prediction parameters of the selected MMR model;

calculating an output image that approximates the first image based on the second image and the determined values of prediction parameters applied to the selected MMR prediction model;

Outputting the determined values of the prediction parameters and the calculated output image, wherein the multiple MMR models include second-order multi-channel, multivariate regression prediction according to the following formula,

wherein represents the three predicted color components of the i-th pixel of the first image,

s _i =[s _i1 s _i2 s _i3 ] represents the three color components of the i-th pixel of the second image,

According to the following formula, and are 3×3 matrices and n is a 1×3 vector,

n＝[n ₁₁ n ₁₂ n ₁₃ ],

as well as

The prediction parameters of the second-order multi-channel, multivariate regression prediction model are numerically obtained by minimizing the mean square error between the first image and the output image.

15. The method according to Note 14, wherein any one of the MMR models includes prediction parameters involving neighboring pixels.

16. The method according to Note 15, wherein the adjacent pixels include left adjacent pixels, right adjacent pixels, top adjacent pixels, and bottom adjacent pixels.

17. The method according to supplementary note 14, wherein selecting the MMR prediction model from the plurality of MMR prediction models further comprises an iterative selection process, comprising:

(a) Select and apply an initial MMR prediction model;

(b) calculating a residual error between the first image and the output image;

(c) If the residual error is less than the error threshold and no other MMR model can be selected, select the initial MMR model; otherwise, select a new MMR prediction model from the multiple MMR prediction models, and the new MMR prediction model is different from the previously selected MMR prediction model; and return to step (b).

18. An image decoding method, comprising:

receiving a first image having a first dynamic range;

Receive metadata, wherein the metadata includes

a multivariate regression (MMR) prediction model adapted to approximate a second image having a second dynamic range according to,

the first image, and

prediction parameters of the MMR prediction model obtained by applying inter-color image prediction, the metadata further comprising previously determined values of the prediction parameters; and

Applying the first image and previously determined values of the prediction parameters to the MMR prediction model to calculate an output image for approximating the second image, wherein the second dynamic range is different from the first dynamic range, wherein the MMR prediction model is a second-order multi-channel, multivariate regression prediction according to the following formula,

wherein represents the three predicted color components of the i-th pixel of the first image,

represents the three color components of the i-th pixel of the second image,

According to the following formula, and are 3×3 matrices and n is a 1×3 vector,

n＝[n ₁₁ n ₁₂ n ₁₃ ],

as well as

The prediction parameters of the second-order multi-channel, multivariate regression prediction model are numerically obtained by minimizing the mean square error between the first image and the output image.

19. The method according to Note 18, wherein the second-order MMR prediction model is expanded into a second-order MMR prediction model or a third-order MMR prediction model with pixel cross multiplication.

20. The method according to Note 18 or 19, wherein the MMR prediction model further includes prediction parameters involving neighboring pixels.

21. An apparatus comprising a processor and configured to perform any one of the methods described in Notes 14-17.

22. A computer-readable storage medium storing computer-executable instructions for executing the method according to any one of Notes 14-17.

Claims (10)

1. An image decoding method using a processor, the method comprising: Receive the first image; Receive metadata, the metadata including prediction parameters for a multi-channel, multivariate regression MMR prediction model, wherein the MMR model is adapted to predict a second image based on the first image; and The first image and the prediction parameters are applied to the MMR prediction model to generate an output image that approximates the second image, wherein the pixel value of at least one color component in the output image is calculated based on the pixel values of at least two color components in the first image, and wherein the MMR model comprises a first-order MMR model with cross-multiplication according to the following formula: in, This represents the predicted three color components of the i-th pixel in the output image. s _i = [s _i1 s _i2 s _i3 ] represents the three color components of the i-th pixel in the first image. According to the following formula, n is a 3×3 prediction parameter matrix and n is a 1×3 prediction parameter vector: and n = [n ₁₁ n ₁₂ n ₁₃ ], The following formula represents the 4×3 prediction parameter matrix: and sc _i is a 1×4 vector based on sc _i = [s _i1 · s _i2 s _i1 · s _i3 s _i2 · s _i3 s _i1 · s _i2 · s _i3 ]. 2. The method according to claim 1, wherein the first image and the second image have different dynamic ranges. 3. The method according to claim 1, wherein the dynamic range of the first image is lower than the dynamic range of the second image. 4. The method of claim 3, wherein the first image is a standard dynamic range image and the second image is a high dynamic range image. 5. The method of claim 1, wherein the MMR model comprises a second MMR model with cross-multiplication according to the following formula: in, sc _i =[s _i1 ·s _i2 s _i1 ·s _i3 s _i2 ·s _i3 s _i1 ·s _i2 ·s _i3 ], sc _i ² = [s _i1 ² ·s _i2 ² s _i1 ² ·s _i3 ² s _i2 ² ·s _i3 ² s _i1 ² ·s _i2 ² ·s _i3 ² ], And includes a 4×3 prediction parameter matrix, and and includes a 3×3 prediction parameter matrix. 6. A video decoder, the decoder comprising: An input terminal for receiving a first image and metadata, wherein the metadata includes prediction parameters for a multi-channel, multivariate regression MMR prediction model, wherein the MMR model is adapted to predict a second image based on the first image; A processor is configured to apply the first image and the prediction parameters to the MMR prediction model to generate an output image that approximates the second image, wherein the pixel value of at least one color component in the output image is calculated based on the pixel values of at least two color components in the first image, and wherein the MMR model comprises a first-order MMR model with cross-multiplication according to the following formula: in, This represents the predicted three color components of the i-th pixel in the output image. s _i = [s _i1 s _i2 s _i3 ] represents the three color components of the i-th pixel in the first image. According to the following formula, n is a 3×3 prediction parameter matrix and n is a 1×3 prediction parameter vector: and n = [n ₁₁ n ₁₂ n ₁₃ ], The following formula represents the 4×3 prediction parameter matrix: and sc _{_i} is a 1×4 vector based on sc _{_i} = [s _{_i1} · s _{_i2}s_{_i1} · s_{_i3}s_i2 · s_{_i3}s_i1 · s_i2 · s_{_i3}]; and A memory for storing the output image. 7. The decoder according to claim 6, wherein the dynamic range of the first image is lower than the dynamic range of the second image. 8. The decoder of claim 6, wherein the first image is a standard dynamic range image and the second image is a high dynamic range image. 9. The decoder of claim 6, wherein the MMR model comprises a second MMR model with cross-multiplication according to the following formula: in, sc _i =[s _i1 ·s _i2 s _i1 ·s _i3 s _i2 ·s _i3 s _i1 ·s _i2 ·s _i3 ], sc _i ² = [s _i1 ² ·s _i2 ² s _i1 ² ·s _i3 ² s _i2 ² ·s _i3 ² s _i1 ² ·s _i2 ² ·s _i3 ² ], And includes a 4×3 prediction parameter matrix, and and includes a 3×3 prediction parameter matrix. 10. A non-transitory computer-readable storage medium having stored thereon a computer-executable program for executing the method of claim 1 using one or more processors.