CN118922855A - Image optimization in mobile capture and editing applications - Google Patents

Abstract

An HDR color scale is sampled in an HDR color space parameterized by a parameter. A reference SDR color scale, an input HDR color scale, and a reference HDR color scale are generated from the sampled HDR color scale. An optimization algorithm is performed to generate an optimized forward reshaping map and an optimized reverse reshaping map. The input HDR image is forward reshaped into a forward reshaped SDR image using the optimized forward reshaping map, and the forward reshaped SDR image is reversely reshaped into a reverse reshaped HDR image using the optimized reverse reshaping map.

Description

Image optimization in mobile capture and editing applications

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63/321,363, filed on March 18, 2022, and European Patent Application 22162984.3, filed on March 18, 2022, each of which is incorporated herein by reference in its entirety.

Technical Field

The present disclosure relates generally to images and more particularly to video codecs for processing images.

Background Art

As used herein, the term "dynamic range (DR)" may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., brightness, luminance) in an image, e.g., from the darkest black (dark colors) to the brightest white (highlights). In this sense, DR is related to "scene-referred" intensities. DR may also relate to the ability of a display device to adequately or approximately render a range of intensities of a particular breadth. In this sense, DR is related to "display-referred" intensities. Unless a particular sense is expressly specified at any point in the description herein to have a particular meaning, it should be inferred that the terms may be used in either sense, e.g., interchangeably.

As used herein, the term "high dynamic range (HDR)" refers to a DR width that spans approximately 14 to 15 or more orders of magnitude across the human visual system (HVS). In practice, the DR, over a wide range of intensity ranges that humans can simultaneously perceive, may be slightly truncated relative to HDR. As used herein, the terms "enhanced dynamic range (EDR) or visual dynamic range (VDR)" may be individually or interchangeably related to such a DR that can be perceived within a scene or image by the human visual system (HVS) including eye movement, allowing for some light adaptation changes across the scene or image. As used herein, EDR may refer to a DR that spans 5 to 6 orders of magnitude. Therefore, while HDR may be slightly narrower relative to a real scene reference, EDR may represent a wide DR width and may also be referred to as HDR.

In practice, an image includes one or more color components of a color space (e.g., luminance Y and chrominance Cb and Cr), where each color component is represented by n bits of precision per pixel (e.g., n=8). Using nonlinear light intensity coding (e.g., gamma coding), images where n≤8 (e.g., color 24-bit JPEG images) are considered as images of standard dynamic range, while images where n>8 can be considered as images of enhanced dynamic range.

The reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between the color values (e.g., luminance) of an input video signal and the output screen color values (e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT.1886, "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011), defines a reference EOTF for flat panel displays, which is incorporated herein by reference in its entirety. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. The term "metadata" herein refers to any auxiliary information that is transmitted as part of an encoded bitstream and that assists a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters as described herein.

As used herein, the term "PQ" refers to the quantization of the perceived brightness amplitude. The human visual system responds to the increased light level in a very nonlinear manner. The ability of humans to observe stimuli is affected by the following factors: the brightness of the stimuli, the size of the stimuli, the spatial frequency that constitutes the stimuli, and the brightness level that the eyes adapt to at the specific moment of viewing the stimuli. In some embodiments, the perception quantizer function maps the linear input grayscale to the output grayscale that better matches the contrast sensitivity threshold in the human visual system. In SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays" (hereinafter referred to as "SMPTE"), an example PQ mapping function is described, which is incorporated herein by reference in its entirety, wherein, given a fixed stimulus size, for each brightness level (e.g., stimulus level, etc.), the minimum visible contrast step at the brightness level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

Displays supporting luminances of 200 to 1,000 cd/ ^m2 or nits represent a lower dynamic range (LDR) associated with EDR (or HDR), also known as standard dynamic range (SDR). EDR content can be displayed on an EDR display that supports a higher dynamic range (e.g., from 1,000 nits to 5,000 nits or more). Such a display can be defined using an alternative EOTF that supports high luminance capabilities (e.g., 0 to 10,000 or more nits). Examples of such EOTFs are defined in SMPTE 2084 and Rec. ITU-R BT.2100, “Image parameter values for high Dynamic range television for use in production and international programme exchange” (06/2017). As the inventors understand herein, it is desirable to use improved techniques for synthesizing video content data that can be used to support the display capabilities of various SDR and HDR display devices.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

1A-1E illustrate an example process flow for an application-optimized reshaping operation;

FIG. 2A illustrates example color distributions of irregular shapes associated with HDR and SDR color spaces such as R.2020, P3, and R.709 color spaces; FIGS. 2B-2E and 2H-2K illustrate example parameterized color spaces in the R.2020 color space; FIGS. 2F and 2G illustrate example percentile Cb and Cr values in a forward reshaped SDR color space corresponding to different parameterized HDR color spaces; FIG. 2L illustrates an example color table image; FIGS. 2M and 2N illustrate example checkerboard images; FIG. 2O illustrates an example original color table image displayed on a reference image display, an example captured image from the color table image, and a corrected captured image generated from the captured image; FIG. 2P illustrates an example RGB value distribution; FIGS. 2Q-2T illustrate example alpha shapes;

3A and 3C illustrate an example process flow for finding an optimized color space to represent HDR colors; FIG. 3B illustrates an example process flow for determining optimized values of programmable parameters in an ISP pipeline; FIG. 3D illustrates an example process flow for generating optimized reshaping operation parameters; FIG. 3E illustrates an example process flow for generating multiple different color tables; FIG. 3F illustrates an example process flow for matching SDR and HDR colors between image pairs of captured SDR and HDR images; FIG. 3G and 3H illustrate example encoder-side and decoder-side image/video editing operations; FIG. 3I to FIG. 3M illustrate example solutions for clipping out-of-range codewords in image/video editing applications;

4A-4E illustrate example process flows; and

FIG5 illustrates a simplified block diagram of an example hardware platform upon which a computer or computing device as described herein may be implemented.

DETAILED DESCRIPTION

In the following description, for the purpose of illustration, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is apparent that the present disclosure can be practiced without these specific details. In other cases, in order to avoid unnecessarily obscuring, obscuring or confusing the present disclosure, well-known structures and devices are not described in detail.

Overview

Image optimizations are described herein for video capture applications including but not limited to mobile (video) capture applications, such as those related to image reconstruction solutions based on tensor product B-splines (TPB). Solutions based on TPB can use robust predictors implemented with video codecs such as backward compatible video codecs to provide or generate relatively accurate reconstructed images. These predictors can operate with static mappings to calmly handle or recover from transmission errors such as lost frames, minimize power consumption such as battery power consumption, reduce data overhead and/or processing/transmission delays, etc. In some operating scenarios, some or all of the solutions based on TPB as described herein can be implemented with mobile applications that are subject to relatively stringent battery or power constraints.

The TPB-based image reshaping solution can be adapted to operate in different use cases. Some use cases allow the freedom to design and/or apply HDR to SDR mapping to generate SDR images to be encoded in (the base layer of) the video signal, as described herein. Some use cases rely on a mobile image signal processor (ISP) with relatively limited programmable registers to generate SDR images to be encoded in (the base layer of) the video signal. Different architectures can be used to implement the TPB-based solution as described herein. Additionally, optionally or alternatively, these architectures or TPB-based solutions can be used to provide or support backward compatibility.

Given a reference SDR image represented in an SDR color space or a color space in an SDR domain, a TPB-based solution may use a TPB optimization process to achieve or determine the maximum or widest HDR color space or a color space in an HDR domain to represent a corresponding reconstructed HDR image predicted from the SDR image. The wider the HDR color space, the more color deviations are introduced into the SDR color space. The TPB optimization process may be implemented to achieve or determine an optimized balance point or trade-off between achieving the maximum or widest HDR color space on the one hand and introducing SDR color deviations on the other hand. Additionally, optionally or alternatively, a TPB optimization process as described herein may be implemented to combine neutral color processing or perform constrained optimization to help preserve the grayscale represented in the SDR image in the reconstructed HDR image predicted from the SDR image.

As various camera-equipped computing or mobile devices capture more and more videos in practice, video editing has become more and more popular. Users operating these devices may be allowed to manually adjust the appearance of the captured video or simply apply a default theme template to the captured video. Depending on the available computing resources and/or the preferences of the video editing tool designer/provider, video editing can be done in a source domain such as the source HDR domain (in which the source image is represented) or a non-source domain such as the generated SDR domain (the pre-edited image can be converted from the source image in the source domain to the non-source domain).

In an operational scenario where an SDR image is to be encoded in a video signal, video editing operations in the HDR domain or source domain do not adversely affect the reconstruction of an HDR image from an SDR image decoded from the video signal at the decoder side by a device downstream of the video signal. This is because these video editing operations do not interfere with the HDR to SDR mapping that generates an SDR image from an HDR image or source image in the HDR domain at the encoder side, and do not interfere with the approximate SDR to HDR mapping that maps back to an HDR image or reconstructs an HDR image at the decoder side.

However, in these operating scenarios, video editing operations in the SDR domain will likely adversely affect the reversibility between the SDR domain and the HDR domain. For example, an edited SDR image may not be able to be mapped back to the original or source HDR image. In addition, the color space used to represent the SDR image may be limited, and its pixel values or codewords cannot exceed a predefined range such as the SMPTE range. Color space conversions such as YUV to RGB conversion for video editing operations may cause pixel values or codewords to be sheared and thus damage or harm reversibility. The shearing techniques described herein can be used to reduce or prevent adverse editing effects on video editing operations (including but not limited to those performed with mobile devices). Two-level TPB boundary shearing can be implemented in the RGB domain to support or retain the maximum edited colors in the SDR domain, which can be used to propagate or map back to the reconstructed images in the HDR domain.

The example embodiments described herein relate to image generation. Sampled HDR color space points distributed in an HDR color space are constructed. The HDR color space is parameterized by a color source scaling parameter having a candidate value selected from a plurality of candidate values. The color source scaling parameter is used to calculate the color space coordinates of at least one of a plurality of color sources outlining the HDR color space. Reference SDR color space points represented in a reference SDR color space, input HDR color space points represented in an input HDR color space, and reference HDR color space points represented in a reference HDR color space point are generated from the sampled HDR color space points in the HDR color space. A reshaping operation optimization algorithm is performed to generate an optimized forward reshaping mapping and an optimized reverse reshaping mapping chain. The reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as input. The optimized forward reshaping mapping is used to forward reshape the input HDR images in the input HDR color space into forward reshaped SDR images in the forward reshaped SDR color space, and the optimized reverse reshaping mapping is used to reverse reshape the forward reshaped SDR images in the forward reshaped SDR color space into reverse reshaped HDR images.

The example embodiments described herein relate to image generation. Sampled HDR color space points distributed in an HDR color space are constructed. The HDR color space is parameterized by a color source scaling parameter having a candidate value selected from a plurality of candidate values. The color source scaling parameter is used to calculate the color space coordinates of at least one of a plurality of color sources outlining the HDR color space. Input SDR color space points represented in an input SDR color space and reference HDR color space points represented in reference HDR color space points are generated from the sampled HDR color space points in the HDR color space. A reshaping operation optimization algorithm is performed to generate an optimized reverse reshaping mapping. The reshaping operation optimization algorithm receives the input SDR color space points and the reference HDR color space points as input. The reverse reshaping mapping is used to reversely reshape the SDR image in the input SDR color space into a reversely reshaped HDR image.

The example embodiments described herein relate to image generation. A set of SDR image feature points are extracted from a training SDR image, and a set of HDR image feature points are extracted from a training HDR image. A subset of one or more SDR image feature points in the set of SDR image feature points is matched with a subset of one or more HDR image feature points in the set of HDR image feature points. A geometric transformation is generated using the subset of the one or more SDR image feature points and the subset of the one or more HDR image feature points to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image. A set of paired SDR color scales and HDR color scales are determined from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image has been spatially aligned with the training HDR image by the geometric transformation. An optimized SDR to HDR mapping is generated based at least in part on the set of paired SDR color scales and HDR color scales obtained from the training SDR image and the training HDR image. The optimized SDR to HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

Example embodiments described herein relate to image generation. A corresponding camera distortion correction operation is performed on each training image in a pair of training standard dynamic range (SDR) images and training high dynamic range (HDR) images to generate a corresponding undistorted image in a pair of undistorted training SDR images and undistorted training HDR images. A corresponding projection transformation in a pair of SDR image projection transformations and HDR image projection transformations is generated using corner pattern markers detected from each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images. Each projection transformation in the pair of SDR image projection transformations and HDR image projection transformations is applied to the corresponding undistorted image in the pair of undistorted training SDR images and undistorted training HDR images to generate a corresponding corrected image in a pair of corrected training SDR images and corrected training HDR images. A set of SDR color labels is extracted from the corrected training SDR images, and a set of HDR color labels is extracted from the corrected training HDR images. An optimized SDR to HDR mapping is generated based at least in part on the set of SDR color labels and the set of HDR color labels obtained from the training SDR images and the training HDR images. The optimized SDR-to-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

The example embodiments described herein relate to a shearing operation on an edited image. Sampled HDR color space points distributed in an HDR color space for representing a reconstructed HDR image are constructed. The sampled HDR color space points are converted to SDR color space points in a first SDR color space, in which an SDR image to be edited by an editing device is represented. A delimiting SDR color space rectangle is determined based on extreme SDR codeword values of the SDR color space points in the first SDR color space. An irregular three-dimensional (3D) shape is determined from the distribution of the SDR color space points. Sampled SDR color space points distributed in the delimiting SDR color space rectangle in the first SDR color space are constructed. A boundary shearing 3D lookup table (3D-LUT) is generated using the sampled SDR color space points and the irregular shape. The boundary shearing 3D-LUT uses the sampled SDR color space points as lookup keys. A shearing operation is performed on the edited SDR image in the first SDR color space based at least in part on the boundary shearing 3D-LUT to generate a boundary sheared edited SDR image in the first SDR color space.

Reshaping optimization in image/video capture applications

Reshaping optimization processes such as TPB and/or non-TPB optimization processes may be implemented or incorporated into video capture applications that are run on computing devices such as mobile devices in various operating scenarios. Video capture applications with reshaping optimization processes may be used to generate or output video signals such as base layers or SDR images encoded therein. The reshaping optimization process may implement different solutions in different operating scenarios to generate or optimize reshaping operation parameters such as TPB and/or non-TPB coefficients for use with base layers or SDR images to generate, construct, or reconstruct non-base layers or HDR images with optimized picture quality.

For illustrative purposes, the reshaping optimization processes or solutions implemented therein can be classified into different types based on the specific SDR generation process or sub-process adopted by the video capture application and based on the specific reshaping path (forward (reshaping) path and/or reverse (reshaping) path) for performing the reshaping optimization.

1A illustrates a first example reshaping optimization process implementing a white-box joint forward and reverse TPB optimization design/solution (referred to as "WFB") with a (single) video capture device. The white-box joint forward and reverse TPB optimization design/solution can be implemented for the following operation scenarios: the SDR generation process or sub-process converts a reference HDR image to a reference SDR image using a white-box conversion operation, and both the forward and reverse (reshaping) paths can be subjected to TPB optimization.

As used herein, "white-box conversion" refers to an HDR to SDR mapping or conversion performed using well-defined conversion or mapping functions/equations, such as those specified or documented in a standard-based or proprietary video coding specification (e.g., publicly, etc.). In contrast, "black-box conversion" refers to an HDR to SDR mapping or conversion performed using a conversion or mapping operation that is not based on a well-defined conversion or mapping function/equation. For example, a black-box conversion may be implemented as internal image signal processing performed by an image signal processor that does not rely or relies little on any well-defined conversion or mapping function or equation specified or documented in a standard-based or proprietary video coding specification (e.g., publicly, etc.).

By way of example and not limitation, in FIG1A , an input video signal including an HDR image may be represented in an input color space, such as a full R.2020 color space. A generated SDR signal including a reference SDR image may be represented in a reference SDR color space. The reference SDR image may be an image generated from the HDR image using a known HDR to SDR mapping process, function, and/or equation.

As shown in FIG1A , in the forward reshaping path, the reference HDR image may be forward reshaped into a (forward) reshaped SDR image that is approximate to the reference SDR image based at least in part on optimized forward reshaping operation parameters generated from the joint forward and backward TPB optimization solution (denoted as “forward TPB optimization”). The reshaped SDR image may be represented in a reshaped SDR color space and generated by an upstream device to be included/encoded in a video signal or its base layer (BL) output by the upstream device.

A downstream receiving device or video decoder of the video signal may decode the reshaped SDR image from the video signal. The decoded reshaped SDR image at the decoder side may be identical to the reshaped SDR image at the encoder side, but errors may be introduced in compression/decompression, encoding operations, and/or data transmission.

In a reverse reshaping path as implemented by a downstream device, the reshaped SDR image may be reversely reshaped into a (reverse) reshaped HDR image based at least in part on optimized reverse reshaping operation parameters generated from the same joint forward and reverse TPB optimization solution (denoted as "Reverse TPB Optimization"). The reshaped HDR image generated with the optimized reverse reshaping operation parameters in the output HDR color space represents an approximate or reconstructed version of the reference HDR image.

The joint forward and backward TPB optimization solution can generate optimized forward and backward reshaping operation parameters to cover as wide as possible output HDR color space in which the reshaped or reconstructed HDR images generated at the decoder side are represented.

One or both of the optimized forward and reverse reshaping operation parameters, such as optimized forward and reverse TPB coefficients, may be implemented or represented in a three-dimensional lookup table or 3D-LUT to reduce processing time in the reshaping operation.

Since the forward and reverse reshaping operation parameters such as forward and reverse TPB coefficients are jointly or simultaneously designed or optimized in the joint forward and backward TPB optimization process, the supported reshaping SDR and HDR color spaces for representing the reshaping SDR and HDR images can be jointly or simultaneously designed or optimized in the same process.

In some operational scenarios, the optimized forward and reverse reshaping operation parameters generated from the joint forward and reverse TPB optimization process can be applied in a static single layer backward compatible (SLBC) framework. Under this static framework, it is not necessary to obtain (e.g., dynamically, etc.) optimized image-specific or image-dependent optimized forward and reverse operation parameters, such as image-specific or image-dependent (or content-dependent) forward and reverse TPB coefficients, such as obtained on the fly when processing the image.

Instead, under the static SLBC framework, the same or statically optimized forward and reverse operation parameters, such as the same optimized forward and reverse TPB coefficients, can be obtained or generated at one time (e.g., offline or before performing a reshaping operation on any of the (input) reference HDR images or the reshaped SDR images) for forward reshaping all (input) reference HDR images and reverse reshaping the reshaped SDR images. In an example, a single set of statically optimized forward and reverse operation parameters can be generated offline by a system as described herein, and configured/deployed in or used by a capture device as described herein to perform an image reshaping or reconstruction operation. In another example, multiple sets of statically optimized forward and reverse operation parameters can be generated offline by a system as described herein, and configured/deployed in or used by a capture device as described herein to select a specific set of statically optimized forward and reverse operation parameters to perform an image reshaping or reconstruction operation.

Thereafter, the upstream device may apply the optimized static forward TPB coefficients to forward reshape some or all (input) reference HDR images (e.g., a continuous or sequential (input) reference HDR image sequence, etc.) to generate a reshaped SDR image to be encoded in a (SLBC) video signal, while the downstream receiving device of the video signal may apply the optimized static reverse TPB coefficients to some or all reshaped SDR images decoded from the video signal (e.g., a continuous or sequential reshaped SDR image sequence, etc.) to generate or reconstruct a reshaped HDR image.

1B illustrates a second example reshaping optimization process implementing a white-box reverse-only TPB optimization design/solution (referred to as "WB") with a (single) video capture device. The white-box reverse-only TPB optimization design/solution can be implemented for the following operation scenarios: the SDR generation process or sub-process converts a reference HDR image to a reference SDR image using a white-box conversion operation, and only the reverse (reshaping) path is subjected to TPB optimization.

By way of example and not limitation, in FIG1B , an input video signal including an HDR image may be represented in an input color space such as a P3 color space. A generated SDR signal including a reference SDR image may be represented in a reference SDR color space. The reference SDR image may be an image generated from the HDR image using a known HDR to SDR mapping process, function, and/or equation.

As shown in FIG. 1B , a programmable image signal processor or ISP may process a reference HDR image into an ISP SDR image that approximates a reference SDR image (e.g., using a given image signal processing function/equation/operation, etc.) based at least in part on optimized ISP operating parameters generated from an ISP optimization solution (denoted as “ISP parameter optimization”). The ISP SDR image may be represented in an ISP SDR color space and generated by an upstream device to be included/encoded in a video signal or a base layer (BL) thereof output by the upstream device.

A downstream receiving device or video decoder of the video signal can decode the ISP SDR image from the video signal. The decoded ISP SDR image on the decoder side may be the same as the ISP SDR image on the encoder side, but errors may be introduced in compression/decompression, encoding operations and/or data transmission.

In a reverse reshaping path as implemented by a downstream device, the ISP SDR image may be reverse reshaped into a (reverse) reshaped HDR image based at least in part on optimized reverse reshaping operation parameters generated from a reverse-only TPB optimization solution (denoted as "Reverse TPB Optimization"). The reshaped HDR image generated with the optimized reverse reshaping operation parameters in the output HDR color space represents an approximate or reconstructed version of the reference HDR image.

Only the inverse TPB optimization solution can generate optimized inverse reshaping operation parameters to cover as wide as possible an output HDR color space in which the reshaped or reconstructed HDR images generated at the decoder side are represented.

The optimized inverse reshaping operation parameters, such as optimized inverse TPB coefficients, may be implemented or represented in a three-dimensional lookup table or 3D-LUT to reduce processing time in the reshaping operation.

In some operational scenarios, the inverse reshaping operational parameters generated from the inverse TPB-only optimization process may be applied in a static single-layer inverse display mapping (SLiDM) framework. Under this static framework, there is no need to obtain (e.g., dynamically, etc.) inverse operational parameters that optimize image-specific or image-dependent optimizations, such as image-specific or image-dependent (or content-dependent) inverse TPB coefficients.

Instead, under the static SLiDM framework, the same or statically optimized reverse operation parameters, such as the same optimized reverse TPB coefficients, may be obtained or generated once (e.g., offline or before performing a reshaping operation on any of the reshaped SDR images) for reverse reshaping the ISP SDR image. In an example, a single set of statically optimized reverse operation parameters may be generated offline by a system as described herein, and configured/deployed in or used by a capture device as described herein to perform an image reshaping or reconstruction operation. In another example, multiple sets of statically optimized reverse operation parameters may be generated offline by a system as described herein, and configured/deployed in or used by a capture device as described herein to select a particular set of statically optimized reverse operation parameters to perform an image reshaping or reconstruction operation.

Thereafter, a downstream receiving device of the video signal may apply the optimized static inverse TPB coefficients to some or all ISP SDR images decoded from the video signal (e.g., a continuous or sequential ISP SDR image sequence, etc.) to generate or reconstruct a reshaped HDR image.

The ISP SDR image encoded in the video signal may or may not be the same as the desired SDR appearance as represented by the reference image. The operating parameters or settings in the programmable ISP can be optimized to approximate the reference image. Therefore, in the "WB" operating scenario of Figure 1B, the SDR image used for reverse reshaping to the reshaped HDR image is given or fixed as an ISP SDR image. By comparison, in the WFB operating scenario of Figure 1A, the SDR image used for reverse reshaping to the reshaped HDR image is a forward reshaped SDR image, which can be optimized by applying the optimized forward reshaping operating parameters generated in the joint forward and reverse TPB optimization process of generating optimized forward and reverse reshaping parameters. In other words, TPB optimization is not used in the forward path to generate ISP SDR images for improving or enhancing both the desired appearance of the SDR image encoded in the video signal and the desired appearance of the reshaped HDR image. Relatively large deviations from the expected appearance may occur in these operating scenarios.

1C illustrates a third example reshaping optimization process implementing a black-box reverse-only TPB optimization design/solution (referred to as "BB1") with a (single) video capture device. The black-box reverse-only TPB optimization design/solution can be implemented for the following operation scenarios: generating SDR images from HDR images without using white-box conversion operations, and only the reverse (reshaping) path is subjected to TPB optimization.

The reshaping operation parameters in these operation scenarios can be generated using training SDR images and training HDR images generated or acquired by the same video capture device, such as the same mobile device, etc. These training SDR images and training HDR images form a plurality of SDR and HDR image pairs, each of which includes a training SDR image and a training HDR image corresponding to the training SDR image.

The training SDR image and the training HDR image in the same SDR and HDR image pair may be acquired at different moments/time points (e.g., a few milliseconds apart, a few fractions of a second apart, a few seconds apart, etc.) using the same capture device using the same ISP. The training SDR image and the training HDR image may not be precisely spatially aligned or temporally aligned with each other because it is difficult, if not impossible, to maintain the same shooting position and process the training SDR image and the training HDR image in the same manner at different moments/time points. For example, the training SDR image may be locally tone mapped or locally enhanced, while the training HDR image may be generated or obtained from multiple camera exposures. Therefore, in the "BB1" operating scenario, the relationship between the pixel or codeword value in the training SDR image and the corresponding pixel or codeword value in the training HDR image in the same image pair may be considered or assumed to be a black box.

The training SDR image and the training HDR image in each of the image pairs may first be spatially aligned. The spatially aligned training SDR image and the training HDR image in the image pair may be used to determine or find matching color pairs. These matching color pairs may then be used to generate optimized reshaping operation parameters such as optimized TPB and/or non-TPB coefficients for reshaping or mapping back the (e.g., non-training, reference, etc.) SDR image to an inversely reshaped or reconstructed HDR image that approximates the (e.g., non-training, reference, etc.) HDR image.

As shown in Figure 1C, an upstream device such as a capture device may generate an SDR image such as a reference SDR image to be included/encoded in a video signal or a base layer (BL) thereof output by the upstream device. The example reference SDR images as described herein may include, but are not necessarily limited to, SDR images generated by a programmable image signal processor of an upstream device or a capture device operating in conjunction with the upstream device.

A downstream receiving device or video decoder of the video signal can decode the reference SDR image from the video signal. The decoded reference SDR image at the decoder side may be the same as the reference SDR image at the encoder side, but errors may be introduced in compression/decompression, encoding operations and/or data transmission.

In a reverse reshaping path as implemented by a downstream device, the reference SDR image may be reverse reshaped into a (reverse) reshaped HDR image based at least in part on optimized reverse reshaping operation parameters generated from a black-box-only reverse TPB optimization solution (denoted as "Reverse TPB Optimization"). The reshaped HDR image generated with the optimized reverse reshaping operation parameters in the output HDR color space represents an approximate or reconstructed version of the reference HDR image that may or may be generated by the same capture device.

The black-box inverse-only TPB optimization design/solution can generate optimized inverse reshaping operation parameters to cover as wide as possible output HDR color space in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

The optimized inverse reshaping operation parameters, such as optimized inverse TPB coefficients, may be implemented or represented in a three-dimensional lookup table or 3D-LUT to reduce processing time in the reshaping operation.

In some operational scenarios, the inverse reshaping operational parameters generated from the black-box inverse-only TPB optimization design/solution can be applied in the SLiDM framework. Under this static framework, there is no need to obtain (e.g., dynamically, etc.) inverse operational parameters that optimize image-specific or image-dependent optimizations, such as image-specific or image-dependent (or content-dependent) inverse TPB coefficients.

Instead, under the static SLiDM framework, the same or statically optimized inverse operation parameters, such as the same optimized inverse TPB coefficients, can be obtained or generated once (e.g., offline or before performing a reshaping operation on any of the reshaped SDR images) for inversely reshaping spatially aligned training SDR images generated by an upstream capture device into reshaped or reconstructed HDR images that are approximately the same as the spatially aligned training HDR images generated by the same capture device.

Thereafter, a downstream receiving device of the video signal may apply the optimized static inverse TPB coefficients to some or all (e.g., non-training, etc.) reference SDR images decoded from the video signal (e.g., a continuous or sequential reference SDR image sequence, etc.) to generate or reconstruct a reshaped HDR image that is approximate to a reference HDR image that may or may be generated from the same upstream capture device that generated or captured the reference SDR images.

Since only TPB optimization is used in the reverse path, in the "BB1" operation scenario, the resulting appearance of the reshaped HDR image generated from the reverse reshaping of the reference SDR image may have a relatively large deviation from the desired appearance of the reference HDR image.

In some "BB1" operating scenarios, as illustrated in FIG. 1D , a camera raw image generated from a camera ISP may be encoded directly in a video signal output by an upstream (capture) device, rather than a reference SDR image generated from a programmable image signal processor. During a training phase, a training camera raw image, which may or may not be an SDR image, may be spatially and/or temporally aligned with a training HDR image in the same image pair of a camera raw and HDR image. Inverse reshaping operation parameters, such as inverse TPB coefficients, may be optimized using matching color pairs from the spatially aligned image pairs of camera raw and HDR images. During a non-training or deployment phase, a downstream device encoding a video signal with a non-training camera raw image may use the optimized inverse reshaping operation parameters to inversely reshape a decoded non-training camera raw image from a video signal into an inversely reshaped or reconstructed (non-training) HDR image that approximates a reference HDR image that may or may be generated by the same upstream (capture) device.

FIG. 1E illustrates a fourth example reshaping optimization process implementing a black-box reverse-only reshaping optimization design/solution (referred to as "BB2") with a different video capture device. The "BB2" reshaping optimization design/solution may be implemented for an operating scenario in which an SDR image is generated by a first capture device, and an HDR image to be generated from the SDR image is generated by a second, different capture device (e.g., intended to be, etc.), and only the reverse (reshaping) path is performed for reshaping optimization. Additionally, the reshaping optimization in the "BB2" operating scenario may or may not be a TPB optimization.

The reshaping operation parameters in the "BB2" operation scenario can be generated using training SDR images and training HDR images generated or acquired by two video capture devices (such as two mobile devices of different brands and/or models). These training SDR images and training HDR images form a plurality of SDR and HDR image pairs, each of which includes a training SDR image and a training HDR image corresponding to the training SDR image.

The training SDR images and training HDR images in the same SDR and HDR image pair may be acquired using the same image, such as the same checkerboard image, rendered on the same or similar image display. In the "BB1" operating scenario, the relationship between the pixel or codeword values in the training SDR image and the corresponding pixel or codeword values in the training HDR image in the same image pair may be considered or assumed to be a black box.

The training SDR image and the training HDR image in each of the image pairs may first be spatially aligned. The spatially aligned training SDR image and the training HDR image in the image pair may be used to determine or find matching color scales that are rendered on the same image display or the same image display type together with a test image such as a checkerboard image. These matching color scales may then be used to construct a three-dimensional lookup table (3D-LUT) to map the SDR pixel or codeword values of the color scale in the test image to the corresponding HDR pixel or codeword values of the same color scale.

A 3D-LUT derived partially or entirely based on a test image may be used to generate optimized static or dynamic reshaping operation parameters for reshaping or mapping back a (e.g., non-training, reference, etc.) SDR image to an inversely reshaped or reconstructed HDR image that approximates a (e.g., non-training, reference, etc.) HDR image.

As shown in Figure 1E, an upstream device such as a first capture device may generate an SDR image such as a reference SDR image to be included/encoded in a video signal or a base layer (BL) thereof output by the upstream device. The example reference SDR image as described herein may include, but is not necessarily limited to, an SDR image generated from a programmable image signal processor of the first capture device or upstream device.

A downstream receiving device or video decoder of the video signal may decode the reference SDR image acquired with the first capture device from the video signal. The decoded reference SDR image at the decoder side may be identical to the reference SDR image at the encoder side, but errors may be introduced during compression, decompression, encoding operations and/or data transmission.

In the reverse reshaping path as implemented by a downstream device, the reference SDR image may be reverse reshaped into a (reverse) reshaped HDR image based at least in part on optimized reverse reshaping operation parameters generated from the "BB2" reshaping optimization design/solution (denoted as "dynamic reverse function optimization"). The reshaped HDR image generated with the optimized reverse reshaping operation parameters in the output HDR color space represents an approximate or reconstructed version of the reference HDR image that may or may be generated by a second, different capture device.

The "BB2" reshaping optimization design/solution can generate optimized inverse reshaping operation parameters to cover as wide as possible output HDR color space, in which the reshaped or reconstructed HDR image generated at the decoder side is represented.

In some operational scenarios, the inverse remodeling operational parameters generated from the “BB2” remodeling optimization design/solution can be applied in a static or dynamic SLiDM framework.

Since only the "BB2" reshaping optimization is used in the reverse path, the resulting appearance of the reshaped HDR image generated from the reverse reshaping of the reference SDR image may have a relatively large deviation from the desired appearance of the reference HDR image. The first capture device and the second capture device may be mobile devices that may operate in different ways (e.g., with different exposure times, different video processing operations, different mapping functions/relationships, etc.) in their respective video capture applications.

In a dynamic SLiDM frame, a dynamic mapping using image-dependent or image-specific reshaping operation parameters can be used to reversely reshape an SDR image of a first capture device into an HDR image that can or can be generated by a second capture device, particularly in an operating scenario where the first capture device and the second capture device operate with relatively large differences between the corresponding video/image capture applications running on the first capture device and the second capture device and/or between the corresponding ISPs used in the first capture device and the second capture device. For example, in a training phase, multiple sets of training SDR images and training HDR images or image pairs can be used to obtain multiple sets of optimized reshaping operation parameters for multiple sets of training SDR images and training HDR images or image pairs, respectively. In a deployment or application phase, specific image characteristics, such as the overall brightness of some or all regions of a specific image, can be dynamically evaluated when processing a specific image. Based on specific image characteristics of a specific image that are related to or compared to corresponding image characteristics of multiple sets of training SDR images and training HDR images or image pairs, a specific set of optimized reshaping operation parameters can be adaptively and/or dynamically selected for a specific image from multiple sets of optimized reshaping operation parameters.

White-box joint TPB forward and backward optimization (WFB)

Multiple design factors of reshaping optimization can be considered to help increase the coverage of the supported color space in which the reshaped image is to be represented. First, a tensor product bi-spline (TPB) predictor with optimized TPB coefficients generated from the reshaping optimization can be used to obtain or achieve relatively high prediction accuracy compared to other types of predictors including but not limited to MMR predictors. The multi-knot and continuity characteristics of the tensor product bi-spline predictor or prediction function can be utilized or used to cover a relatively wide color range or color space portion with relatively high accuracy. An example TPB predictor can be found in U.S. Provisional Patent Application Serial No. 62/908,770, "Tensor-product B-spline predictor" filed by Guan-Ming Su, Harshad Kadu, Qing Song, Qing Song and Neeraj J. Gadgil on October 1, 2019, the contents of which are incorporated herein by reference as fully set forth herein.

In contrast, although a multi-segment MMR predictor or prediction function may be better than a single-segment MMR predictor or prediction function, discontinuities between different MMR segments may be introduced into the multi-segment MMR predictor or prediction function, thereby adversely affecting and even prohibiting the use of the multi-segment MMR predictor or prediction function in many operating scenarios. Example MMR-based operations are described in U.S. Patent 8,811,490, which is incorporated herein by reference in its entirety as if fully set forth herein.

The TPB predictor can be advantageously used in a SLBC-based video codec. More specifically, the forward TPB predictor can be used to generate a forward reshaped SDR image that is approximate to a reference SDR image, while the reverse TPB predictor can be used to generate a reverse reshaped or reconstructed HDR image that is approximate to a reference HDR image. An example use of the TPB predictor with a SLBC-based video codec can be found in U.S. Provisional Patent Application Serial No. 63/255,057, "Tensor-product B-spline prediction for HDR video in mobile applications," filed by H. Kadu et al. on October 13, 2021, the contents of which are incorporated herein by reference in their entirety as fully set forth herein. In addition, a BESA (Backward Error Subtraction for Signal Adjustment) algorithm/method modified to support neutral color preservation can be used in a chained reshaping function pipeline to optimize the forward and reverse TPB predictors together to achieve relatively high reversibility or relatively accurate approximation of a reference HDR image through a reverse reshaped or reconstructed HDR image. Example BESA algorithms/methods can be found in U.S. Provisional Patent Application Serial No. 63/013,063, “Reshaping functions for HDR imaging with continuity and reversibility constraints,” filed by G-M. Su on April 21, 2020, U.S. Provisional Patent Application Serial No. 63/013,807, “Iterative optimization of reshaping functions in single-layer HDR image codec,” filed by G-M. Su and H. Kadu on April 22, 2020, and PCT Application Serial No. PCT/US2021/028475, filed on April 21, 2021, the contents of which are incorporated herein by reference in their entirety as if fully set forth herein.

Although the TPB predictor has better prediction accuracy than other types of predictors, transmitting TPB coefficients in a video signal may incur relatively large signal or bit rate overhead. In addition, constructing the TPB equations or basis functions used in the TPB predictor may incur relatively large computational costs.

In some operating scenarios, in order to avoid or reduce signal or bit rate overhead and computational cost, some or all images in the whole video sequence can be reshaped using a built-in or static TPB predictor without changing the TPB coefficients used in the static TPB predictor during the playback of the video sequence. More specifically, some or all (built-in or static) TPB coefficients can be cached or stored in video applications such as video capture/editing applications, without the need to explicitly transmit these TPB coefficients by encoding a video signal or bit stream with an image to be reshaped by a built-in or static TPB predictor or in the video signal or bit stream. In an example, some or all TPB coefficients can be preloaded or preconfigured in a downstream receiving device before a downstream receiving device receives and processes a video signal or bit stream. Additionally, optionally or alternatively, multiple groups of TPB coefficients can be preloaded or preconfigured in a downstream receiving device before a downstream receiving device receives and processes a video signal or bit stream. For example, based on a simple indicator (e.g., a binary indicator, a multi-bit indicator, etc.) signaled or transmitted in a video signal or bitstream, a video application can simply pick or select a particular set of TPB coefficients from multiple sets of TPB coefficients to use for a static TPB predictor. An example static TPB predictor or prediction function can be found in the aforementioned U.S. Provisional Patent Application Serial No. 63/255,057.

In some operating scenarios, a mobile device can be used to host or run video capture and/or editing applications. Users of mobile devices can edit captured images in these video capture and/or editing applications. For example, edited images such as edited HDR images that may be intended to be displayed or viewed on an image display of a non-mobile device with a display capability higher than or larger than the image display of the mobile device may have a brightness range and/or color range or color distribution that is higher, larger, wider and/or wider than the (original) captured images such as HDR images originally captured from the camera sensor of the mobile device. For example, the captured HDR image may typically be limited by the camera sensor and ISP output of the mobile device to be represented in a relatively narrow color space or color gamut such as P3, while the edited HDR image can be represented in a relatively wide color space or color gamut such as the entire R.2020 color space.

In order to design a static mapping for a reshaping operation, it is highly desirable to optimize the forward/backward TPB coefficients to cover as high a dynamic range as possible and as wide a color space or gamut as possible while achieving as much bitrate and computational efficiency as possible.

Full HDR reversibility between a reconstructed HDR image and a reference HDR image that is the same as the reconstructed HDR will require distinguishing colors in the original or reconstructed HDR domain (or color space) that are to be distinguished or distinguishable in the reshaped SDR domain (or color space) so that the distinguished colors in the SDR domain can be mapped back to distinguishable colors in the HDR domain. Therefore, full reversibility will likely require a one-to-one mapping from HDR to SDR and back from SDR to HDR. The wider the supported HDR domain or color space, the more codewords the SDR domain or color space needs to have. Given that the total number of available SDR pixel or codeword values in the SDR domain (e.g., corresponding to a lower bit depth than the HDR domain, etc.) is typically less than the total number of required HDR pixel or codeword values in the HDR domain, full HDR reversibility may not be achieved in some operating scenarios. In practice, some captured or edited images may contain diffuse and/or specular color combinations that form irregularly shaped color distributions as shown in FIG. 2A , which exceed the R.2020 color space (“BT.2020”) and therefore cannot be fully represented by that color space, let alone smaller color spaces or SDR color spaces such as the R.709 color space (“BT.709”) or the (DCI) P3 color space.

In many operating scenarios, a nonlinear reshaping mapping or function can be used to relatively efficiently distribute the available pixel or codeword values and generate a (forward) reshaped SDR that approximates the reference SDR image as closely as possible and helps the reconstructed HDR image approximate the reference HDR image as closely as possible, although the reshaped SDR generated by the nonlinear reshaping mapping or function may be different from the reference SDR image, but may contain some deviations from the reference SDR image.

Additionally, optionally, or alternatively, in some operating scenarios, in order to maintain the same or similar appearance of the reference SDR and HDR images in the reshaped SDR and HDR images, reshaping optimizations as described herein may be performed under neutral color constraints that preserve neutral colors in the color mapping performed with the reshaping operations.

Given that a TPB predictor or function has the ability to approximate functions with relatively high nonlinearity, the TPB predictor or function may be incorporated into the reshaping operation to support or enable a relatively wide HDR color space for representing the inversely reshaped or reconstructed HDR image.

A potential disadvantage is that the TPB predictor may require a relatively large number of TPB coefficients to represent or approximate the non-linear SDR to HDR and/or HDR to SDR mapping functions. To prevent possible ambiguous conditions, numerical instabilities, slow convergence issues, etc. that may arise when solving the optimization problem of the TPB coefficients, static TPB predictors may be pre-generated and deployed with video codecs (such as those used by upstream devices and/or downstream devices) and then used to process video sequences such as captured and/or edited video sequences.

For illustrative purposes only, in an operational scenario as shown in FIG. 1A , the (forward reshaped) SDR domain or color space may be that of R.709, while the (original or reverse reshaped) HDR domain or color space may be that of R.2020. As illustrated in FIG. 2A , the R.2020 color space is much larger than the R.709 color space. Therefore, while it may be difficult to map the entire R.2020 to R.709 and then map R.709 back to R.2020, the reshaping optimization techniques as described herein may be used to optimize the coverage of the HDR color space used to represent the reverse reshaped or reconstructed HDR image, and maintain the same or similar appearance of the reference SDR and HDR images in the reshaped SDR and HDR images. These techniques may be used to simultaneously optimize reshaping operation parameters such as TPB coefficients for both forward reshaping operations and reverse reshaping operations.

While it may not be possible to cover the entire R.2020 color space in all scenarios, a subset or subspace in the R.2020 color space (e.g., a specific color gamut outlined as a triangle formed by color primaries in a color coordinate system, etc.) can be selected or picked through a reshaping optimization operation to maintain HDR reversibility of the reference HDR image and an acceptable SDR approximation of the reference SDR image in the reshaped HDR and SDR images.

By way of illustration and not limitation, a subset or subspace in an R.2020 color space (or an HDR color space used to represent a reshaped HDR image) may be defined or characterized by a specific white point and three specific chromatic algebras (red, green, blue). The specific white point may be selected or fixed to be a D65 white point. There may be a great deal of design freedom to select three specific chromatic algebras from many possible chromatic algebra combinations for a subset or subspace in R.2020 to be supported by the reshaping operation. The reshaping optimization techniques as described herein may be used to determine or select specific chromatic algebras for a subset or subspace in R.2020 as optimized chromatic algebras for achieving or attaining maximum perceived quality and/or color coding efficiency and/or reversibility between SDR images and HDR images.

The MacAdam ellipses represent only noticeable color differences, since the HVS may not be able to distinguish color differences within the same ellipse. The color gamut of the pointer may represent all diffuse colors that the HVS can perceive. In the MacAdam ellipses and the color gamut of the pointer, green is less important or less distinguishable/perceivable by the HVS than red and blue. Therefore, in some operating scenarios, a specific optimized color source for a subset or subspace in R.2020 used to represent the reshaped or reconstructed HDR image may be selected to cover more red and blue than green, especially when the SDR to HDR and HDR to SDR mappings cannot support a one-to-one mapping relationship in the reshaping operation.

As used herein, a color source may also be referred to as a primary color and may be used to define the angles of a polygon such as a triangle that represents a specific color space or color gamut (such as a color space specified by a standard, a color space supported by a display, a color space supported by a video signal, etc.). For example, a standard color space with a standard specified white point may be represented by a triangle whose angles are specified by three standard specified color sources in a CIExy color space coordinate system or a CIExy chromaticity diagram. The CIExy coordinates of the color sources (red or R, green or G, blue or B) and white points that define R.709, P3, and R.2020 color spaces, respectively, are specified in Table 1 below.

Table 1

Each color source and the CIExy coordinates of the corresponding white point in the standard color space in Table 1 can be expressed as and Among them, (c) is the standard color space.

Therefore, the P3 color space can be characterized by the following CIExy coordinates of the P3 color primaries and the P3 white point: Similarly, the R.2020 color space can be characterized by the following CIExy coordinates of the R.2020 color primaries and the R.2020 white point:

The inversely reshaped HDR color space used to represent the inversely reshaped or reconstructed HDR image is denoted as the (a) color space. Therefore, the CIExy coordinates of the color primaries and white point defining the (a) color space can be expressed as and

As mentioned above, green is less important or less distinguishable/perceivable by the HVS than red and blue. In some operating scenarios, the red and blue chromatic algebra of the (a) color space may be selected to match the red and blue chromatic algebra of the R.2020 color space as follows:

In addition, like all R.709, P3, and R.2020 color spaces that are specified with a D65 white point, the (a) color space can also be specified with a D65 white point, as follows:

(a) The green color source of the color space can be the green color source along the P3 color space in the CIExy coordinate system or the chromaticity diagram Green color source with R.2020 color space Any point along the line between the two green primaries in the P3 color space and the R.2020 color space can be represented as a linear combination of the two green primaries with a weight factor a, as follows:

Therefore, the optimization problem of finding the maximum support for the R.2020 color space from the (a) color space can be simplified to the problem of selecting the weight factor a. When a=0, the (a) color space becomes the entire R.2020 color space. When a=1, as illustrated in Figure 2B (where the (a) color space is represented as "TPB" or "colors covered by the current TPB"), the red and blue corners or red and blue color primaries of the (a) color space are the same as the red and blue corners or red and blue color primaries of the R.2020 color space, and the green corner or green color primaries of the (a) color space are the same as the green corner or green color primaries in the P3 color space. Figures 2C to 2E illustrate three example (a) color spaces, where a=0.9, 0.5, and 0.25, respectively.

Joint color space and TPB optimization

Since the range or coverage of the inverse reshaped (a) color space on the R.2020 color space can be controlled by parameter a, the overall TPB (reshaping-based) optimization problem becomes how to optimize the TPB coefficients in both the forward path and the backward path so that (i) the forward reshaped SDR domain (or color space) used to represent the reshaped SDR image is close to the reference SDR domain (or color space) used to represent the reference SDR image, especially in the neutral colors or color space parts that are more sensitive to HVS than the non-neutral colors or color space parts, and (ii) the inverse reshaped or reconstructed HDR domain (or color space) used to represent the inverse reshaped or reconstructed HDR image is as closely identical as possible to the reference HDR domain (or color space) used to represent the reference HDR image. Ideally, the inverse reshaped or reconstructed HDR image is identical to the reference HDR image if perfect reconstruction or complete reversibility can be achieved.

As illustrated in Figures 2B to 2E, in order to make the inversely reshaped or reconstructed HDR domain or color space cover as large an R.2020 color space as possible, the TPB optimization problem boils down to finding or searching for the minimum possible value of parameter a so that the above SDR and HDR quality conditions are met.

Figure 3A illustrates an example process flow for finding the minimum possible value of parameter a for maximizing SDR and HDR quality in reshaped SDR and HDR images.The process flow of Figure 3A may iterate through multiple candidate values of parameter a in an iteration order that may be a sequential order or a non-sequential order.

Block 302 includes selecting a current (e.g., to be iterated, etc.) value of parameter a to a next candidate value (e.g., initially a first candidate value, etc.) among a plurality of candidate values for parameter a. Given a candidate inverse reshaped HDR color space having a current value of parameter a, optimized reshaping operation parameters, such as optimized TPB coefficients, may be obtained in one or more subsequent process flow blocks of FIG. 3A.

Block 304 includes constructing sample points or preparing two sample data sets in a candidate inversely reshaped HDR color space. By way of example and not limitation, the candidate inversely reshaped HDR color space may be a hybrid log-gamma (HLG) RGB color space (referred to as "(a) RGB color space HLG").

The first of the two sampling datasets is a uniformly sampled color label dataset. Each color label in the uniformly sampled color label dataset consists of a corresponding RGB color uniformly sampled from (a) the RGB color space HLG (denoted as ) characterizes or represents, the (a) RGB color space HLG includes three dimensions of R axis, G axis, B axis represented as R, G and B component colors respectively. Each axis or dimension of the (a) RGB color space HLG is normalized to the value range [0,1] and is sampled by _NR , _NG and _NB units or divisions respectively. Here, each of _NR , _NG and _NB represents a positive integer greater than one (1). Therefore, the total number of color scales or sampled data points in the uniformly sampled color scale data set is given as follows:

_{Nu ＝ NRNGNB} ( ₄ )

Each uniformly sampled data point or RGB color in the uniformly sampled color scale dataset is given as follows

where i∈[0,…, _NR -1], j∈[0,…, _NG -1], k∈[0,…, _NB -1]

For simplicity, (i, j, k) in the above expression (5) can be vectorized or simply represented as p. Accordingly, the uniformly sampled data points or RGB colors It can be simply expressed as All _Nu nodes (each of which represents a unique combination of a specific R-axis cell/partition, a specific B-axis cell/partition, a specific G-axis cell/partition) can be grouped or collected into a vector/matrix as follows:

The second of the two sample data sets prepared or constructed in block 304 is a neutral color data set. The second data set includes a plurality of neutral colors or neutral color scales (also referred to as grays or gray scales).

The second data set can be used to preserve the input gray scales in the input domain as output gray scales in the output domain when the input gray scales in the input domain are mapped to or reshaped into output gray scales in a reshaping operation as described herein. Increased weighting can be given to the input gray scales in the input domain (or input color space) in the optimization problem compared to other scales to reduce the likelihood that these input gray scales are mapped to non-gray scales in the output domain (or output color space) by the reshaping operation.

The second data set (gray data set or gray data set) can be prepared or constructed by uniformly sampling R, G, B values along a line connecting a first gray (0, 0, 0) and a second gray (1, 1, 1) in an RGB domain (e.g., (a) RGB color space HLG, etc.) to generate N _n nodes or gray color scales, as follows:

where i∈[0,…,N _n -1]

All N _n nodes in the second data set can be grouped or collected into a neutral color vector/matrix as follows:

The neutral color vector/matrix in expression (8) above may be repeated N _t (a positive integer not less than (1)) times to generate N _n N _t neutral color labels in the (now repeated) second data set, as follows:

Repeating the neutral colors in the repeated second neutral color data set increases the weight of the neutral colors or grays compared to other colors. Therefore, the neutral colors can be retained more in the optimization problem than other colors.

The first (all sampled) color data set and the second neutral color (repeated) data set in expressions (6) and (9) can be collected or placed together in a single combined vector/matrix as follows:

Combine vectors/matrices The total number of vector/matrix elements (repeating and non-repeating color scales) in is N=N _n N _t +N _u . Each vector/matrix element or color scale (row) in can be represented as follows:

Block 306 includes converting the combined vector/matrix in (a) RGB color space (or (a) RGB color space HLG) The color values of the color scales (rows) represented in the vector/matrix elements are converted to the corresponding color values in the standard R.2020 color space or R.2020RGB color space HLG.

The three primary colors of red, green and blue in the (a) RGB color space HLG corresponding to the three corners or points of the triangle defining the (a) RGB color space HLG in the CIExy chromaticity diagram can be converted from CIE xy values to CIE XYZ values via the following equations:

Y＝1 (12-1)

In the given In the case of red, green, and blue primaries as (x, y) or CIE xy values, the same red, green, and blue primaries as (X, Y, Z) or CIE XYZ values can be obtained using the conversion equations in the above expression (12), which are respectively expressed as Similarly, given In the case of a white point as (x,y) or CIE xy values, the same conversion equation in equation (12) above can be used to obtain the same white point as (X,Y,Z) or CIE XYZ values, expressed as

The 3×3 transformation matrix denoted as P ^(a)→XYZ can be constructed by combining the vector/matrix The color values of the color scale (row) represented by the vector/matrix elements of the (x,y) or CIE xy values are converted to the corresponding (X,Y,Z) or CIE XYZ values as follows

in, represents the element-wise multiplication between the two matrices on the right side (RHS) in Expression (13); and the two matrices on the RHS in Expression (13) can be constructed as follows:

in,

The overall 3×3 transformation matrix, denoted as P ^(a)→R2020, can be constructed as the combined vector/matrix in the aRGB color space HLG The color values of the color scales (rows) represented in the vector/matrix elements are converted to the corresponding color values in the R.2020RGB color space HLG as follows:

P ^(a)→R2020 =P ^(a)→XYZ P ^XYZ→R2020 (17)

Among them, P ^XYZ→R2020 represents a 3×3 conversion matrix that converts XYZ color values to corresponding values in the R.2020RGB color space HLG.

Thus, in block 306, the combined vector/matrix in aRGB color space (or (a)RGB color space HLG) may be The color values of the color scales (rows) represented in the vector/matrix elements are converted to the corresponding color values in the standard R.2020 color space or R.2020RGB color space HLG as follows:

Block 308 includes converting the vector/matrix into the R.2020RGB color space HLG The color value of the color scale (row) represented in the vector/matrix element of is converted to the corresponding color value in the R.2020YCbCr color space HLG (expressed as ),as follows:

in, Represents a conversion function or matrix that converts a (R.2020RGB) color value in the R.2020RGB color space HLG to a corresponding (HDR R.2020YCbCr) color value in the R.2020YCbCr color space HLG.

Each color scale (row) in can be represented as follows:

Block 310 includes converting the vector/matrix into the R.2020RGB color space HLG The color value of the color scale (row) represented by the vector/matrix element is converted or mapped to the corresponding color value in the R.709SDR YCbCr color space (expressed as ),as follows:

Among them, f _{HLG→W_SDR()} means converting the HDR R.2020RGB color value in the R.2020RGB color space HLG Convert to the corresponding R.709SDR RGB color value in the R.709SDR RGB color space (expressed as )’s white-box HDR to SDR mapping function; and Represents then converting the R.709SDRRGB color value in the R.709SDR RGB color space Convert to the corresponding color value in R.709SDR YCbCr color space A conversion function or matrix. Non-limiting examples of white box HDR to SDR mapping functions are described in BT.2390-10, “High dynamic range television for production and international programme exchange” (November 2021), which is incorporated herein by reference in its entirety. Other examples of white box HDR to SDR mapping functions may include, but are not necessarily limited to, any of the following: a standard-based HDR to SDR conversion function, a proprietary HDR to SDR conversion function, an HDR to SDR conversion function with a linear and/or non-linear domain, an HDR to SDR mapping function using a gamma function, a power function, etc.

Each color scale (row) in can be represented as follows:

Box 312 includes converting the vector/matrix in the R.2020RGB color space HLG The color value of the color scale (row) represented in the vector/matrix element is converted or mapped to the corresponding color value in the R.2020YCbCr color space PQ (expressed as ),as follows:

Wherein, f _HLG→PQ() represents, for example, the transfer function described in the aforementioned SMPTE 2084 and Rec.ITU-R BT.2100 for converting HDR R.2020RGB HLG color values in the R.2020RGB color space HLG to Converted to the corresponding HDR R.2020RGB PQ color value in the R.2020HDR RGB color space PQ (expressed as )’s HLG to PQ conversion function; and The HDR R.2020 RGB PQ color values in the R.2020 HDR RGB color space PQ are then converted, for example, using the transfer functions described in the aforementioned SMPTE 2084 and Rec.ITU-R BT.2100 Convert to the corresponding color value in R.2020YCbCr color space PQ The transfer function or matrix of .

Each color scale (row) in can be represented as follows:

Block 314 includes using and Forward and reverse reshaping operation parameters, such as forward and reverse TPB coefficients, may be obtained or generated as input to a (TPB) BESA algorithm that may be enhanced or modified with neutral color preservation.

The BESA algorithm is an iterative algorithm, where each current iteration may modify the reference SDR signal based on the inverse prediction error measured or determined in the previous iteration.

The modified BESA algorithm as described herein can implement neutral color preservation and avoid modifying the neutral color scale. To this end, a set of neutral color scale indices that identify or correspond to the neutral color scale can be generated as follows:

Wherein, δ represents a neutral color value threshold value that determines or specifies a maximum color deviation range in absolute value, within which the color value meets the condition of neutral color. Example values of the neutral color value threshold value δ may include, but are not necessarily limited to, any of the following items: 1/2048, 1/1024, etc.

Let ch denote the channel in the forward reshaped SDR domain or color space (e.g., three channels of , etc.). Let F denote the forward reshaping or forward path. Let B denote the reverse reshaping or reverse path.

Expressed as The per-channel forward generation matrix (also called the design matrix) can be used to generate an input HDR color scale represented by the corresponding input HDR codeword as obtained by expression (19) Predict the forward reshaped SDR color scale as represented by the corresponding forward reshaped SDR codeword. The forward generator matrix can be derived from the cross-channel forward TPB basis function to is generated or precomputed, stored or cached in computer memory, and fixed in all iterations for the forward path as follows:

Among them, the cross-channel forward TPB basis function to Adopting or accepting the above expression (20) The color scales (or rows) in are taken as input parameters.

The forward reshaped SDR color scale can be predicted by multiplying the forward generation matrix in expression (28) with the forward TPB coefficients. Example predicted reshaped codewords with TPB coefficients (which are equivalent or similar to the color scales herein) are described in the aforementioned U.S. Provisional Patent Application Serial No. 62/908,770.

The forward reshaped SDR color scale or codeword may be reversely reshaped into a reverse reshaped HDR color scale or codeword, for example, by a TPB-based reverse reshaping operation.

In the reverse path, at each iteration (e.g., kth iteration, etc.) in the BESA algorithm, it is expressed as The per-channel reverse generation matrix of can be used to predict the reverse reshaped HDR color scale as represented by the corresponding reverse reshaped HDR codeword from the forward reshaped SDR color scale as represented by the corresponding forward reshaped SDR codeword obtained in the forward path. The reverse generation matrix for each iteration can be obtained from the cross-channel forward TPB basis function to is generated, and not fixed across all iterations for the reverse path, as follows:

in, to Indicates the forward reshaped SDR color code or codeword.

The inverse reshaped HDR color scale can be predicted by multiplying the inverse generation matrix in expression (29) with the inverse TPB coefficients.

The reverse prediction error for each iteration in the BESA algorithm can be determined by comparing the reverse reshaped HDR color scale or codeword collected in the vector/matrix with the reference HDR color scale or codeword in the per-channel reverse observation vector/matrix obtained from expression (25). The per-channel reverse observation vector/matrix can be generated or pre-computed from expression (25), stored or cached in computer memory, and fixed in all iterations as follows:

The reference SDR signal or the reference SDR color scale or codeword can be used as a prediction target for the forward reshaping operation to generate the forward reshaped SDR color scale or codeword into an approximate reference SDR color scale or codeword. (where k represents an iteration index starting from zero (0) as the first iteration in the BESA algorithm) may be initialized to the SDR color scale or codeword obtained using the above expression (22) for the first iteration (k=0), and thereafter may be modified at the end of each iteration in part or in whole based on the prediction error determined for that iteration. The modified reference SDR color scale or codeword may be used as a prediction target for a forward reshaping operation in the next iteration in the BESA algorithm to generate the forward reshaped SDR color scale or codeword into an approximately modified reference SDR color scale or codeword.

The per-channel reference SDR color scale or codeword can be used to construct the vector/matrix as follows at the kth iteration

Before the first iteration, the vector in the above expression (31) is set to the original reference SDR signal represented by expressions (22) and (23), as follows:

At the kth iteration, the forward TPB coefficients (expressed as ) that minimizes the difference between the forward reshaped SDR color scale or codeword and the reference SDR color scale or codeword determined for that iteration, as follows:

The predicted SDR color scale or codeword for channel ch at the kth iteration may be calculated from the optimized values of the (e.g., per-channel, etc.) forward TPB coefficients as follows:

At the kth iteration, the inverse TPB coefficients (expressed as) can be generated (eg, per channel, etc.) via the least squares solution of the optimization problem. ) that minimizes the difference between the inversely reshaped HDR color scale or codeword and the reference HDR color scale or codeword (which is fixed for all iterations in the BESA algorithm), as follows:

The predicted HDR color scale or codeword for channel ch at the kth iteration may be calculated from the optimized values of the (eg, per-channel, etc.) inverse TPB coefficients as follows:

The inverse prediction error is calculated and propagated back to the reference non-neutral SDR signal.

As described above, the reverse prediction error for each iteration in the BESA algorithm can be determined by comparing the reverse reshaped HDR color scale or codeword with the reference HDR color scale or codeword obtained from expression (25) (which is fixed for all iterations in the BESA algorithm).

Among these backward prediction errors, the backward prediction errors of non-neutral colors (or non-gray colors) can be propagated back to update or modify the reference SDR color labels or codewords of these non-neutral colors (or non-gray colors). The modified SDR color labels or codewords of non-neutral colors (or non-gray colors) can be combined with the (unmodified) reference SDR color labels or codewords of neutral colors (or gray colors) to be used as the prediction target of the forward path in the next or upcoming iteration in the BESA algorithm.

In some operational scenarios, the inverse prediction error may be calculated as the difference between the original or reference HDR signal (or a reference HDR color label/codeword therein) and the predicted HDR signal (or an inversely reshaped HDR color label/codeword therein) at the kth iteration as follows:

In response to determining that the color label i is in the neutral color set Φ, the reference SDR codewords of the color label in the Cb and Cr channels of the reference SDR signal may be set to a gray value, such as 0.5, for the upcoming (k+1)th iteration as follows:

Otherwise, in response to determining that the color label i is not in the neutral color set Φ, the reference SDR codewords of the color labels in the Cb and Cr channels of the reference SDR signal may be set to updated or modified values according to the reverse prediction error for the upcoming (k+1)th iteration as follows:

Wherein, ε represents a reverse prediction error threshold, above which the reverse prediction error can be propagated to update the color scale of the non-neutral color; ^αch represents a fixed or configurable learning rate for scaling the reverse prediction error; λ represents an upper limit or maximum allowed value of the reference codeword for modifying the color scale of the non-neutral color. An example value of ε may be, but is not necessarily limited to, 0.000005. An example value of ^αch may be, but is not necessarily limited to, 1. An example value of λ may be, but is not necessarily limited to, 2. (This means that there is no limitation, as the SDR and/or HDR codewords herein may be normalized to a normalization domain or range of 0 to 1).

Neutral color preservation as performed by expressions (38) to (40) can be used to ensure that neutral colors in the reference SDR signal do not deviate to or from non-neutral colors in all iterations of the BESA algorithm, thereby improving the perceived quality of grayscale related in the reshaped image.

The BESA algorithm may be iterated up to a total number of iterations. In some operating scenarios, the total number of iterations of the BESA algorithm may be specifically selected to balance color bias in the reshaped SDR and/or HDR images. For example, different total numbers of iterations in the BESA algorithm may generate forward and reverse TPB coefficients that produce reshaped SDR images of different SDR appearances and reshaped HDR images of different HDR appearances. A total number of iterations may be selected for the BESA algorithm to generate reshaped SDR and HDR images of relatively high quality appearance, such as ten times, fifteen times, etc.

The BESA algorithm with neutral color preservation may be performed for each candidate value of parameter a.For example, in block 314, the BESA algorithm with neutral color preservation may be performed for the current candidate value of parameter a.

Block 316 includes determining whether the current candidate value for parameter a is the last candidate value among the plurality of candidate values for parameter a. If so, process flow proceeds to block 318. Otherwise, process flow returns to block 302.

Block 318 includes selecting an optimal or optimized value for parameter a, and calculating (or simply selecting already calculated) optimized forward and reverse TPB coefficients corresponding to the optimized value for parameter a for the (a) RGB color space.

Choosing the optimal value of parameter a can be formulated as finding The optimization problem of (a specific set of values of) is such that (1) (a) the RGB color space represents the maximum HDR color space (e.g., for covering the largest part of the R.2020RGB color space, etc.) to achieve The HDR prediction error is minimized, and (2) the SDR color scale or codeword is forward reshaped and represented as The relatively small or acceptable SDR color deviations are as follows:

Here, w is a weighting factor used to balance between HDR error and SDR error.

This optimization problem can be solved by using the corresponding optimal or optimized TPB coefficients for each aRGB color space. and The results from this color space are checked to solve. Among all possible or candidate a values, the smallest a value (corresponding to the maximum color space coverage in the R.2020 color space) can achieve the minimum HDR error and a relatively small SDR error. HDR prediction error and SDR color deviation It can be a subjective (e.g., with input from a human observer, etc.) or an objective distortion function (e.g., with no or little input from a human observer, etc.), such as those measured with an image dataset from a database. Example distortion functions may include, but are not necessarily limited to, any of the following: mean square error (or deviation) or MSE, root mean square error (or deviation) or RMSE, sum of absolute differences or SAD, peak signal-to-noise ratio or PSNR, structural similarity index (SSIM), etc.

2F and 2G illustrate example percentile Cb and Cr values in the forward reshaped SDR domain or color space for different values of parameter a. Different values of parameter a determine different color space coverage of the reverse reshaped HDR domain or color space in the R.2020 domain or color space, and thus affect the reconstruction or generation of the reverse reshaped HDR image in the R.2020 domain or color space in different ways, and result in different distortions of the reverse reshaped HDR image compared to a reference HDR image to which the reverse reshaped HDR image is to be approximated (e.g., as measured by a distortion metric expressed or measured).

The vertical axis of FIG. 2F represents the 3 percentile values in the Cb and Cr channels in the forward reshaped SDR domain for different values of the parameter a value as represented by the horizontal axis of FIG. 2F (e.g., obtained with the BESA algorithm with neutral color preservation in ten iterations, etc.). The lower the value of the parameter a, the greater the coverage of the a color space in the R.2020 color space. The 3 percentile value can be used explicitly or implicitly to represent the minimum value in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space. The smaller the 3 percentile value, the smaller the minimum value, and therefore the larger the range of codeword values in the forward reshaped SDR (YCbCr) domain or color space extends. As can be shown in FIG. 2F, the lower the value of the parameter a, the lower the 3 percentile value in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space.

The vertical axis of FIG. 2G represents the 97th percentile values in the Cb and Cr channels in the forward reshaped SDR domain for different values of the parameter a value as represented by the horizontal axis of FIG. 2G (e.g., obtained with the BESA algorithm with neutral color preservation in ten iterations, etc.). The 97th percentile value can be used explicitly or implicitly to represent the maximum value in the Cb and Cr channels of the forward reshaped SDR (e.g., YCbCr, etc.) domain or color space. The larger the 97th percentile value, the larger the maximum value, and therefore the larger the range of codeword values in the forward reshaped SDR (YCbCr) domain or color space extends. As can be shown in FIG. 2G, the 97th percentile value in both the Cb and Cr channels of the forward reshaped SDR domain or color space remains at an almost constant value.

In some operational scenarios, in block 318, the image data set used to determine distortions in the reshaped SDR and HDR images and select an optimized value for parameter a based in part or in whole on these distortions may include a video bitstream acquired with one or more specific video capture devices such as mobile phones (e.g., supporting a video coding profile in SMPTE ST 2094, supporting Dolby Vision Profile 8.4, etc.). A non-limiting example of an optimized value for parameter a may be, but is not necessarily limited to, 0.5, which determines the maximum color space coverage of the inverse reshaped HDR domain or color space in the R.2020 color space. As wider inverse reshaped color spaces are used or supported (e.g., corresponding to values less than the optimized value of parameter a, etc.), the reshaped SDR colors and codewords may begin to deviate from the reference SDR colors and codewords with relatively larger distortions. This is because increasing the reverse reshaped HDR domain or color space means squeezing the distinguished forward reshaped SDR colors (or codewords) more tightly into the forward reshaped SDR domain or color space, causing the forward reshaped SDR colors to move from the original 3D positions represented in the reference SDR colors (or codewords) to be approximated by the forward reshaped SDR colors.

As described above, forward and reverse TPB coefficients for a TPB-based reshaping operation may be determined for a given value of parameter a, such as an optimized value. These TPB coefficients may be multiplied in a mathematical equation with a forward or reverse generation matrix constructed with TPB basis functions using an input codeword as an input parameter to generate a forward or reshaped codeword, which may require performing numerous calculations involving calculating TPB basis function values for each of numerous pixels of the image.

In some operational scenarios, in order to speed up or reduce computation, forward and reverse 3D-LUTs may be constructed from pre-computed TPB basis function values for sampled values and optimized forward and reverse TPB coefficients generated for optimized values of parameter a. The forward and reverse 3D-LUTs or the lookup nodes/entries therein may be pre-built before being deployed at runtime to process input images and applied to relatively simple lookup operations in the forward and reverse paths or corresponding forward and reverse reshaping operations performed therein on input images at runtime.

The optimized value of parameter a is represented as a ^opt . The corresponding optimal forward and reverse TPB coefficients are represented as and

The forward 3D-LUT can be used to forward reshape the input HDR colors (or codewords) in an input HDR domain or color space, such as the R.2020 domain or color space (including the a color space), into forward reshaped SDR colors (or codewords) in a forward reshaped SDR domain or color space.

The alpha color space identified inside an input HDR domain or color space, such as an R 2020 (container) domain or color space, can be used to cut out HDR colors or codewords represented outside the alpha color space. The forward TPB coefficients can be applied to the input HDR colors or codewords in the alpha color space identified inside the input HDR domain or color space, such as an R 2020 (container) domain or color space, to generate predicted or forward reshaped SDR colors or codewords for each lookup node or entry in the forward 3D-LUT. Therefore, the forward 3D-LUT includes a plurality of lookup nodes or entries, each of which maps or forward reshapes a corresponding input (cross-channel or three-channel) HDR color or codeword to a corresponding predicted or forward reshaped (cross-channel or three-channel) SDR color or codeword.

In some operating scenarios, the forward 3D-LUT construction process includes a first step in which a 3D uniform sampling grid is prepared in an input HDR domain or color space, such as the R 2020 (container) domain or color space.

By way of illustration and not limitation, the input HDR domain or color space is an R.2020YCbCr color space HLG including three dimensions or channels (i.e., Y axis, Cb axis, Cr axis). The input HDR values (denoted as ) can be sampled uniformly in each dimension or channel as follows:

where i∈[0,…,N _Y -1], j∈[0,…,N _Cb -1], k∈[0,…,N _Cr -1]

The total number of sample values along each axis of the YCbCr axis is denoted as _NY , _NCb, and _NCr , respectively. Therefore, the combined total _number of sample values on all these axes is _Nu = _{NYNCbNCr .}

Although the valid values in the entire R.2020 YCbCr color space may be limited (not covering the entire 3D cube of YCbCr values), the entire sample value grid can be used to cover the entire 3D cube of YCbCr values to reduce the possibility of codeword bias caused by the compression operation.

For simplicity, (i,j,k) can be vectorized or represented as p. Therefore, the sample value in R.2020YCbCr It can be expressed as The vector/matrix can be constructed by collecting all _Nu nodes together, as follows:

The forward 3D-LUT construction process includes a second step, in which the sampled values in the R.2020YCbCr color space HLG can be converted to corresponding values (or colors) in the R.2020RGB color space HLG as follows:

The forward 3D-LUT construction process includes a third step, in which the converted value in the R.2020RGB color space HLG can be converted to a corresponding value (or color) in the aRGB color space HLG corresponding to the optimized value of the parameter a (denoted as a ^opt ), as follows:

The forward 3D-LUT construction process includes a fourth step, in which the converted values in (a) RGB color space HLG may be clipped as follows:

The forward 3D-LUT construction process includes a fifth step, in which the clipped values in the aRGB color space HLG can be converted to corresponding values (or colors) in the R.2020RGB color space HLG as follows:

The forward 3D-LUT construction process includes a sixth step, in which the value obtained by the above expression (47) in the R.2020RGB color space HLG can be converted to the corresponding value (or color) in the R.2020YCbCr color space HLG as follows:

The forward 3D-LUT construction process includes a seventh step, in which, for each lookup node/entry in the forward 3D-LUT, the optimized forward TPB coefficients are combined with the input HDR color or codeword (or codeword value) obtained using the above expression (48) in the R.2020YCbCr color space HLG as the input parameter of the forward TPB basis function to construct a forward generation matrix Used together to obtain mapped or forward reshaped SDR colors or codewords (or codeword values) as follows:

In the above expression (49), the input HDR color or codeword (or codeword value) can be used As input to the forward TPB basis function to construct or construct the forward generator matrix as follows:

Mapped or forward reshaped SDR color or codeword (or codeword value) The input HDR colors or codewords (or codeword values) used as the input parameters of the forward TPB basis functions can be used as the lookup keys of the lookup nodes/entries of the forward 3D-LUT. At runtime, these mapped or forward reshaped SDR colors or codewords can be simply looked up in the forward 3D-LUT with the lookup keys as the input HDR colors or codewords to be forward reshaped into the mapped or forward reshaped SDR colors or codewords.

As mentioned earlier, the reverse 3D-LUT can be used to reversely reshape the reshaped SDR colors (or codewords) in the forward reshaped SDR domain or color space into reverse reshaped HDR colors (or codewords) in a reverse reshaped HDR domain or color space such as the R.2020 domain or color space (including the a color space).

In some operating scenarios, a reverse 3D-LUT construction process that may be simpler than the forward 3D-LUT construction process discussed above may be implemented or executed to construct or construct a reverse 3D-LUT. In some operating scenarios, in order to speed up or reduce calculations, a reverse 3D-LUT may be constructed from pre-computed TPB basis function values of sample values and optimized reverse TPB coefficients generated for optimized values of parameter a. The reverse 3D-LUT or the lookup nodes/entries therein may be pre-built prior to being deployed at runtime to process input images such as forward reshaped SDR images, and applied to relatively simple lookup operations in the reverse path or corresponding reverse reshaping operations performed therein on the input images at runtime.

The pre-built reverse 3D-LUT can be deployed on the decoder side. A receiving downstream device on the decoder side can receive and decode a video signal encoded with a forward reshaped SDR image in a forward reshaped SDR domain or color space, and apply reverse reshaping to the forward reshaped SDR image using the 3D-LUT to generate a reverse reshaped HDR image in a reverse reshaped HDR domain or color space (such as a color space contained in the R.2020 domain or color space).

Although the shape formed or outlined by the complete boundaries of the forward reshaped SDR domain or color space mapped by utilizing the TPB-based forward reshaping operation may not be a simple 3D cube, the forward reshaped SDR values in the forward reshaped SDR domain or color space can be clipped to the tightest or smallest 3D cube (e.g., a 3D rectangle, a 3D cube rescaled from a 3D rectangle, etc.) that contains or supports all the forward reshaped SDR values represented in all lookup nodes/entries of the forward 3D-LUT, for example without clipping these SDR values represented in the forward 3D-LUT.

In some operational scenarios, the inverse 3D-LUT construction process includes a first step in which a complete grid of sample values may be prepared in a forward reshaped SDR domain or color space, such as the full R.709 SDR YCbCr color space.

The reverse 3D-LUT construction process includes a second step, in which the minimum and maximum values of each dimension or (color) channel in the forward reshaped SDR domain or color space can be determined among the forward reshaped SDR values in the forward 3D-LUT and used to limit the (input) data range of the forward reshaped SDR (e.g., YCbCr, etc.) colors or codewords used as input to the reverse path.

The inverse 3D-LUT construction process includes a third step in which, for each lookup node/entry in the inverse 3D-LUT, the optimized inverse TPB coefficients are combined with an inverse generation matrix constructed using the forward reshaped SDR colors or codewords (or codeword values) in the clipped or limited (input) data range obtained in the second step of the inverse 3D-LUT construction process as input parameters of the inverse TPB basis function. (e.g., as illustrated in expression (29) above, etc.) to obtain a mapped or inversely reshaped HDR color or codeword (or codeword value).

The mapped or reverse reshaped HDR colors or codewords (or codeword values) can be used as lookup values for lookup nodes/entries of the reverse 3D-LUT, while the input or forward reshaped SDR colors or codewords (or codeword values) used as input parameters of the reverse TPB basis functions can be used as lookup keys for lookup nodes/entries of the reverse 3D-LUT. At runtime, these mapped or reverse reshaped HDR colors or codewords can be simply looked up in the reverse 3D-LUT with the lookup keys as inputs to be reverse reshaped into mapped or reverse reshaped HDR colors or codewords or forward reshaped SDR colors or codewords.

White Box Backward Optimization (WB)

Due to the cost and/or computational overhead involved in implementing or operating forward TPB-based reshaping or forward 3D-LUT in video capture and/or editing applications, forward TPB-based reshaping or corresponding forward 3D-LUT may not be used in all video capture and/or editing devices.

In some operational scenarios, HDR to SDR conversion, such as HDR HLG to SDR image generation, may be performed in a video capture and/or editing device using a hardware-based solution such as implemented with an available ISP. An existing ISP pipeline deployed with the device may operate with one or more programmable parameters to generate, convert, and/or output an SDR image based in part or in whole on a corresponding HDR (e.g., HLG, etc.) image captured by the device.

In the forward path, programmable parameters for the ISP pipeline may be specifically set or configured to cause the ISP pipeline to output an SDR image that approximates as closely as possible a possible reference SDR image generated with a white-box HDR to SDR conversion function.

In the reverse path, the reverse TPB-based reshaping or the corresponding reverse 3D-LUT can be used to generate a reverse reshaped HDR image from the SDR image output from the ISP pipeline. The reverse TPB coefficients used in the reverse path can be optimized to, for example, cover as much of the reverse reshaped HDR domain or color space as possible, such as the R.2020 color space.

3B illustrates an example process flow for determining or generating optimized values for one or more programmable parameters in an ISP pipeline for generating or outputting (ISP) SDR images from HDR images.

In some operational scenarios, the ISP pipeline may be a relatively limited programmable module that is implemented with the ISP (hardware) to approximate a white-box HDR to SDR conversion function such as a white-box (e.g., known, well-defined, etc.) conversion function for converting an HDR HLG image to a corresponding SDR image in block 330.

The ISP pipeline may include or implement: (1) a first set of three one-dimensional lookup tables (1D-LUTs) for converting from HLG RGB to linear RGB, (2) followed by a 3×3 matrix for converting from an HDR image in an HDR domain or color space, such as the R.2020 color space, to an SDR domain or color space, such as the R.709 color space, and (3) followed by a second set of three 1D-LUTs implementing BT.1886 linear to non-linear (gamma) conversion using an (e.g., standards-based, etc.) HLG optical-to-optical transfer function (OOTF).

For illustration purposes only, the HDR image may be an HDR HLG image taken from an image dataset or database as shown in FIG3B .The one or more programmable parameters for the ISP pipeline may be design parameters (denoted as γ ^BT1886 ) specified in the BT.1886 standard.

3B may be used to search for an optimized value of the design parameter γ ^BT1886 for the ISP SDR image to best approximate the reference SDR image generated in block 330. The process flow may iterate through multiple candidate values of the design parameter γ ^BT1886 in an iteration order that may be a sequential order or a non-sequential order.

Block 322 includes selecting a current (e.g., to be iterated, etc.) value of the design parameter γ ^BT1886 to be a next candidate value (e.g., initially a first candidate value, etc.) among a plurality of candidate values of the design parameter γ ^BT1886 . Given the current value of the design parameter γ ^BT1886 , the ISP SDR image may be compared to a reference SDR image in one or more subsequent process flow blocks of FIG. 3B .

Block 324 includes applying a first set of 1D-LUTs to the HDR HLG (RGB) image retrieved from the database to generate a corresponding HDR linear RGB image, as follows:

Among them, ch represents the red, green or blue channel; Represents the HDRHLG codeword in the (input) HDR HLG image; represents the HDR linear codeword in the corresponding HDR linear image generated from the first set of 1D-LUT as expressed in Expression (51); parameters (a, b, c) can be set to (0.17883277, 0.28466892, 0.55991073).

Block 326 includes applying a 3×3 matrix to the corresponding HDR linear RGB image to generate a corresponding SDR linear RGB image. The 3×3 matrix may be given as follows:

P ^R2020→R709 ＝P ^R2020→XYZ P ^XYZ→R709 (52)

The SDR linear RGB codeword in the corresponding SDR linear RGB image generated by the 3×3 matrix in the above expression (52) can be expressed as

Block 328 includes applying the second set of 1D-LUTs to the SDR linear RGB codewords in the corresponding SDR linear RGB image. To generate the corresponding ISP SDR codeword in the corresponding ISP SDR image (expressed as ).

In some operating scenarios, as described above, the second set of 1D-LUTs merges or combines the linear to non-linear SDR conversion with the HLG OOTF given as follows:

in, Represents the SDR linear RGB codeword from the corresponding SDR linear RGB image Generated intermediate SDR codeword: _Lw = 100; _Lb = 0; and

For the intermediate SDR codeword from the above expression (53) Generate ISP SDR codewords The linear to non-linear SDR conversion can be a linear to gamma conversion as defined in BT.1886, as follows:

Among them, γ ^BT1886 represents the design parameter mentioned above.

Boxes 322 to 328 of FIG. 3B as performed with the ISP pipeline (or a combination of the first set of 1D-LUTs represented in the above expression (51), the 3×3 matrix represented in the above expression (52), and the second set of 1D-LUTs represented in the above expressions (53) to (56)) jointly implement an HDR HLG to SDR conversion function (expressed as f _HLG→ISPSDR ) with specific values of the design parameter γ ^BT1886 (e.g., current candidate values, etc.).

Block 332 includes determining the difference (e.g., quality, etc.) between the ISPSDR image generated from the HDR HLG to SDR conversion function f _HLG→ISPSDR in blocks 322 to 328 and the reference SDR image generated from the white-box HDR to SDR conversion function in block 330. Quality assessment functions such as MSE, RMSE, SAD, PSNR, SSIM, etc. may be used to calculate these differences.

Block 334 includes determining whether the current candidate value for the design parameter γ ^{BT 1886} is the last candidate value among the plurality of candidate values for the design parameter γ ^{BT 1886.} If so, process flow proceeds to block 336. Otherwise, process flow returns to block 322.

Block 336 includes selecting an optimal or optimized value for the design parameter γ ^BT1886 . Selecting the optimized value for the design parameter γ ^BT1886 can be formulated as an optimization problem of finding a specific value of the design parameter γ ^BT1886 (denoted as γ ^BT1886,opt ) from a plurality of candidate values that minimizes the difference between the ISP SDR image and the reference SDR image as calculated using a relatively large image data set or database, as follows:

γ ^BT1886,opt ＝argmin∑ _t D(f _HLG→ISPSDR (V _t ,γ ^BT1886 )-f _{HLG→W_SDR} (V _t ))

(ST)

Wherein, D() represents the quality assessment function used in block 332 .

An example value of γ ^BT1886,opt may be, but is not necessarily limited to, 2.115.

SDR to PQ TPB Optimization

As described above, in a "WFB" use case or operating scenario, the BESA algorithm may be used under the SLBC framework to generate optimized reshaping operation parameters that are used by forward and reverse reshaping operations to generate forward reshaped SDR images and reverse reshaped HDR images. In contrast, in a "WB" use case or operating scenario, only reverse reshaping may be performed under the SLiDM framework on non-forward reshaped SDR images encoded in the output video signal (e.g., ISP SDR images generated by an ISP pipeline implemented with a video capture and/or editing device) to generate corresponding reverse reshaped or reconstructed HDR images.

In the "WB" use case or operating scenario, forward TPB reshaping from an input HDR domain or color space such as R.2020 HDR color space HLG to a forward reshaped SDR domain or color space such as R.709 SDR YCbCr color space cannot be used to help utilize the full codeword range supported by the R.709 SDR YCbCr color space. The ISP SDR codewords encoded in the ISP SDR images in the output video signal are typically hard-constrained in time in the R.709 color space portion generated or supported by the video capture and/or editing device or the ISP pipeline implemented therein. The ISP SDR codewords in the hard-constrained R.709 color space portion may be difficult to further extend or map back to the reverse reshaped or reconstructed HDR domain or color space by reverse reshaping such as reverse TPB-based reshaping.

However, as in the "WFB" use case or operating scenario, in the "WB" use case or operating scenario, optimized reshaping operation parameters such as TPB coefficients may be generated in the reshaping optimization to help increase the coverage of supported color spaces in which the inverse reshaped HDR images are to be represented. In many "WB" operating scenarios, the inverse reshaped or reconstructed HDR domain or color space achieved with inverse TPB-based reshaping may be only slightly larger than the R.709 color space, compared to the maximum (a) color space achievable in the "WFB" use case or operating scenario.

For illustrative purposes only, in an operating scenario as shown in FIG. 1B , the ISP SDR domain or color space may be the domain or color space of R.709 or a hard-limited portion thereof in the ISP pipeline, and the (original or reverse-reshaped) HDR domain or color space may be the domain or color space of R.2020.

By way of illustration and not limitation, a subset or subspace in the R.2020 color space (or HDR color space used to represent the reshaped HDR image) may be defined or characterized by a specific white point and three specific color primaries (red, green, blue). The specific white point may be selected or fixed to be the D65 white point.

The inversely reshaped HDR color space used to represent the inversely reshaped or reconstructed HDR image is denoted as the (b) color space. Therefore, the CIExy coordinates of the color primaries and white point defining the (b) color space can be expressed as and (b) White point of color space Can be specified as D65 white point.

(b) Each of the color primaries of the color space may be a corresponding color primaries along the P3 color space in the CIExy coordinate system or chromaticity diagram Corresponding color primaries to the R.2020 color space Any point along the line between two corresponding chromatic algebras in the P3 color space and the R.2020 color space can be represented as a linear combination of the two corresponding chromatic algebras with a weight factor b, as follows:

Therefore, the optimization problem of finding the maximum support for the R.2020 color space from the (b) color space can be simplified to the problem of selecting a weight factor b. When b = 0, the (b) color space becomes the entire R.2020 color space. When b = 1, as illustrated in Figure 2K (where the (b) color space is represented as "TPB" or "TPB-covered colors (b color space)"), the (b) color space becomes the P3 color space. Figures 2H to 2J illustrate three example (b) color spaces, respectively, where b = 0.25, 0.50, and 0.75.

As illustrated in FIGS. 2H to 2K , in order to make the inversely reshaped or reconstructed HDR domain or color space cover as large an R.2020 color space as possible, the TPB optimization problem boils down to finding or searching for the minimum possible value of the parameter b.

Figure 3C illustrates an example process flow for finding the minimum possible value for parameter b. The process flow of Figure 3C may iterate through multiple candidate values for parameter b in an iteration order that may be a sequential order or a non-sequential order.

Block 342 includes selecting a current (e.g., to be iterated, etc.) value of parameter b to a next candidate value (e.g., initially a first candidate value, etc.) among a plurality of candidate values for parameter b. Given a candidate inversely reshaped HDR color space having a current value of parameter b, optimized reshaping operation parameters, such as optimized TPB coefficients, may be obtained in one or more subsequent process flow blocks of FIG. 3C .

Block 344 includes constructing sample points or preparing two sample data sets in a candidate inversely reshaped HDR color space. By way of example and not limitation, the candidate inversely reshaped HDR color space may be a hybrid log-gamma (HLG) RGB color space (referred to as "(b)RGB color space HLG").

Similar to block 304 of FIG. 3A , the first of the two sampled data sets is a uniformly sampled color label data set. Each color label in the uniformly sampled color label data set consists of a corresponding RGB color uniformly sampled from (b) the RGB color space HLG (denoted as ) characterizes or represents, the (b) RGB color space HLG includes three dimensions of R axis, G axis, B axis represented as R, G and B component colors, respectively. Each axis or dimension of the (b) RGB color space HLG is normalized to the value range [0,1] and is sampled by _NR , _NG and _NB units or divisions, respectively. Here, each of _NR , _NG and _NB represents a positive integer greater than one (1). Therefore, the total number of color scales or sampled data points in the uniformly sampled color scale data set is given as _Nu = _NR N _{G NB} . Each uniformly sampled data point or RGB color in the uniformly sampled color scale data set is Given as where i∈[0,…, _NR -1], j∈[0,…, _NG -1], k∈[0,…, _NB -1].

For simplicity, (i,j,k) can be vectorized or simply represented as p. Correspondingly, the uniformly sampled data points or RGB colors It can be simply expressed as All _Nu nodes (each of which represents a unique combination of a specific R-axis cell/partition, a specific B-axis cell/partition, a specific G-axis cell/partition) can be grouped or collected into a vector/matrix as follows:

The second of the two sample data sets prepared or constructed in block 344 is a neutral color data set. The second data set includes a plurality of neutral colors or neutral color scales (also referred to as grays or gray scales).

The second data set can be used to preserve the input gray scales in the input domain as output gray scales in the output domain when the input gray scales in the input domain are mapped to or reshaped into output gray scales in a reshaping operation as described herein. Increased weighting can be given to the input gray scales in the input domain (or input color space) in the optimization problem compared to other scales to reduce the likelihood that these input gray scales are mapped to non-gray scales in the output domain (or output color space) by the reshaping operation.

The second data set (gray data set or gray data set) can be prepared or constructed by uniformly sampling R, G, B values along a line connecting a first gray (0,0,0) and a second gray (1,1,1) in an RGB domain (e.g., (b) RGB color space HLG, etc.) to generate N _n nodes or gray color scales, as follows:

where i∈[0,…,N _n -1]

All N _n nodes in the second data set can be grouped or collected into a neutral color vector/matrix as follows:

The neutral color vector/matrix in expression (8) above may be repeated N _t (a positive integer not less than (1)) times to generate N _n N _t neutral color labels in the (now repeated) second data set, as follows:

Repeating the neutral colors in the repeated second neutral color data set increases the weight of the neutral colors or grays compared to other colors. Therefore, the neutral colors can be retained more in the optimization problem than other colors.

The first (all sampled) color data set and the second neutral color (repeated) data set in expressions (61) and (64) can be collected or placed together in a single combined vector/matrix as follows:

Combine vectors/matrices The total number of vector/matrix elements (repeating and non-repeating color scales) in is N=N _n N _t +N _u . Each vector/matrix element or color scale (row) in can be represented as follows:

Block 346 includes converting the combined vector/matrix in (b) RGB color space (or (b) RGB color space HLG) The color values of the color scales (rows) represented in the vector/matrix elements are converted to the corresponding color values in the standard R.2020 color space or R.2020RGB color space HLG as follows:

Wherein, P ^{(b) → R2020} represents the conversion matrix from the (b) color space to the R.2020RGB color space HLG, which can be constructed in a similar manner to the way P ^{(a) → R2020} is constructed in the process flow of FIG. 3A , as follows:

p ^(b)→XYZ (68-1)

P ^(b)→R2020 =P ^(b)-X _Y ^Z P ^XYZ→R2020 (68-2)

Wherein, P ^(b)→XYZ represents the conversion matrix from the (b) color space to the XYZ color space, and the conversion matrix can be constructed in a similar manner to the way in which P ^(a)→XYZ is constructed in the process flow of FIG. 3A .

Box 348 includes converting the vector/matrix in the R.2020RGB color space HLG The color values of the color scales (rows) represented in the vector/matrix elements are converted or mapped to the corresponding ISP SDR color values in the R.709SDR RGB color space (expressed as ) and then further to the corresponding color value in the R.709SDR YCbCr color space (expressed as ),as follows:

Wherein, f _{HLG→ISPSDR()} represents the ISP equation, as implemented by the ISP pipeline with the optimized value of the design parameter γ ^BT1886 generated from the process flow of FIG. 3B .

Each color scale (row) in can be represented as follows:

Box 350 includes converting the vector/matrix in the R.2020RGB color space HLG The color value of the color scale (row) represented in the vector/matrix element is converted or mapped to the corresponding color value in the R.2020YCbCr color space PQ (expressed as ),as follows:

Each color scale (row) in can be represented as follows:

Block 352 includes using p As input to the inverse TPB optimization algorithm, inverse TPB coefficients corresponding to the current value of parameter b for the (b) color space are generated for TPB-based reshaping optimization.

To this end, it is possible to The TPB basis functions and SDR codewords represented in the above construct the inverse generation matrix as follows:

The reverse prediction error can be determined by comparing the reverse reshaped HDR color scale or codeword collected in the vector/matrix with the reference HDR color scale or codeword in the per-channel reverse observation vector/matrix obtained and minimized when solving the TPB optimization problem. The per-channel reverse observation vector/matrix can be generated or pre-computed from the above expression (71), stored or cached in computer memory, and fixed in all iterations as follows:

The inverse TPB coefficients (expressed as ) that minimizes the difference between the inversely reshaped HDR color scale or codeword and the reference HDR color scale or codeword, as follows:

The per-channel predicted (or inversely reshaped or reconstructed) HDR codeword for each channel ch may be calculated as follows:

Block 354 includes determining whether the current candidate value for parameter b is the last candidate value among the plurality of candidate values for parameter b. If so, process flow proceeds to block 356. Otherwise, process flow returns to block 342.

Block 356 includes selecting an optimal or optimized value for parameter b, and calculating (or simply selecting already calculated) optimized forward and reverse TPB coefficients corresponding to the optimized value for parameter b for the (b)RGB color space.

Similar to the "WFB" use case or operation scenario, in the "WB" use case or operation scenario, (b) color space is also part of the optimization. The (b) color space and the inverse TPB coefficients can be optimized jointly. Therefore, the optimization problem can be formulated as finding (a specific value of ) such that (b) the color space is generated to cover the largest HDR color space or color space portion while achieving the expression The minimized HDR prediction error is as follows:

An example optimized value for parameter b may be, but is not necessarily limited to, 1, which corresponds to the P3 color space.

Black-box TPB reverse optimization in a single device (BB1)

Some (dual mode) video capture devices or mobile devices support both SDR and HDR capture modes and can therefore output SDR or HDR images. Some (single mode) video capture devices or mobile devices only support SDR capture mode. Under the technology as described herein, SDR to HDR mappings can be modeled or generated with dual mode video capture devices. Some or all of these SDR to HDR mappings can then be applied to SDR images captured by dual mode video capture devices or single mode video capture devices, for example in a downloaded and/or installed video capture application, to "up-convert" SDR images acquired with the SDR capture mode to corresponding HDR images, regardless of whether these video capture devices support HDR capture mode.

In some operating scenarios, since the computing environment in the mobile device or the processing power of the mobile device may be limited, a static (inverse) 3D-LUT and optimized TPB coefficients obtained at least in part from the TPB basis function can be used to represent a static SDR to HDR mapping, which maps or inversely reshapes SDR images such as all SDR images in an SDR video sequence to generate corresponding HDR images such as all HDR images in a corresponding HDR video sequence.

Static SDR to HDR mapping can be modeled based at least in part on multiple image pairs of HDR and SDR images acquired with a specific camera of a dual-mode video capture device. Each image pair of the HDR and SDR images includes an SDR image and an HDR image corresponding to the SDR image. The SDR image and the HDR image depict the same visual scene in the real world as having spatial alignment errors caused by spatial movement, which may occur between a first time point when a specific camera operating in the SDR capture mode captures the SDR image and a second time point when the same camera operating in the HDR capture mode captures the HDR image. For example, for each visual scene in the physical world, the video capture device can operate in HDR mode to capture the HDR image in the image pair and use the SDR mode to capture the SDR image in the same image pair. This operation can be repeated to generate multiple image pairs of HDR and SDR images to generate a relatively large number of different color scales (or a relatively large number of different colors or different codewords) for generating a static SDR to HDR mapping.

There are several challenges in modeling static SDR to HDR mapping using captured SDR and HDR images. First, although both SDR and HDR capture modes can use the same ISP inside the same video capture device, the video capture device can apply different image capture settings (e.g., exposure settings of a specific camera, etc.) for the same scene to obtain the designed optimized picture quality in the captured SDR and HDR images. Therefore, static SDR to HDR mapping may be able to model or approximate the HDR image capture settings actually implemented by the video capture device to some extent. Secondly, the spatial alignment between the captured SDR and HDR images of the image pair to be used to obtain the SDR and HDR images may not be accurate. For example, the selection of a specific capture mode between the SDR capture mode and the HDR capture mode can be performed via the screen of the touching video capture device, which may cause the video capture device to lose or move away from its previous spatial position and/or orientation. In addition, the temporal alignment between the captured SDR video/image sequence and the captured HDR video/image sequence may easily occur because these SDR and HDR sequences are captured at different times or durations. Some of the visual objects depicted in these sequences can move in or out of the camera field of view. Some local areas in the camera's field of view may be occluded or unoccluded from time to time.

In some operational scenarios, a registration operation may be performed to resolve spatial and/or temporal alignment issues between a captured SDR image and a corresponding captured HDR image to generate an aligned SDR image and a corresponding aligned HDR image and include the aligned SDR image and the corresponding aligned HDR image in an image pair of SDR and HDR images. The image pair may then be used to determine or establish SDR color scales or colors in the aligned SDR image and corresponding HDR color scales or colors in the corresponding aligned HDR image. These SDR and HDR color scales or colors may be used to derive at least some of a set of corresponding SDR and HDR color scales or colors for deriving or generating a static SDR to HDR mapping.

The SDR and HDR video capture processes may be performed to provide comprehensive coverage of different scenes and/or exposures at different times of day (e.g., from early morning to late evening) for both indoor and outdoor scenes. For each scene, the same video capture device may be used in both SDR mode and HDR mode to capture SDR and HDR video sequences. In some operational scenarios, specific frames/images, such as the first frame/image, of each of the video sequences may be selected or extracted for use in constructing or obtaining multiple image pairs of SDR and HDR images to be included in a training image dataset or database.

3D illustrates an example process flow for generating optimized reshaping operation parameters such as optimized or optimal TPB coefficients using corresponding (or matched) SDR and HDR color scales or colors in a captured SDR image and a captured HDR image (corresponding to the captured SDR image) captured by a video capture device operating in SDR and HDR capture modes, respectively.

Block 362 includes receiving a captured SDR image and finding or extracting a set of SDR image feature points in the captured SDR image. Block 364 includes receiving a captured HDR image and finding or extracting a set of HDR image feature points in the captured HDR image. The set of SDR image feature points and the set of HDR image feature points may be the same set of feature point types.

The set of feature point types may include feature point types among various different feature point types, including but not necessarily limited to any, some or all of the following: BRISK features detected using Binary Robust Invariant Scalable Keypoints or BRISK algorithm; corners detected using Features-from-Accelerated-Segment-Test or FAST algorithm; features detected from KAZE algorithm; corners detected using minimum eigenvalue algorithm; features generated using Maximum Stable Extreme Region or MSER algorithm; key points detected using oriented FAST and rotation or ORB algorithm; features extracted from scale-invariant feature transform or SIFT algorithm; features extracted from accelerated robust features or SURF algorithm; and the like.

Each of some or all of the set of SDR image feature points and the set of HDR image feature points may be represented by a feature vector or descriptor such as an array of (eg, numerical values, etc.) feature values.

Block 366 includes matching some or all of the feature points in the set of SDR image feature points with some or all of the feature points in the set of HDR image feature points. For each feature point of a particular type in the set of SDR image feature points in the SDR image, a matching metric between the feature point in the SDR image and each of the feature points of the same type in the set of HDR image feature points in the HDR image can be calculated. In a non-limiting example, the matching metric can be calculated as the sum of absolute differences or SAD (or another metric function for measuring the difference between two feature points) between a first feature vector representing a feature point in the SDR image and a second feature vector representing a feature point in the HDR image. The particular feature point with the lowest SAD or matching metric can be selected from the feature points of the same type in the HDR image as a possible match for the feature point in the SDR image. In response to determining that the lowest SAD or matching metric is below a matching difference threshold, the particular feature point in the HDR image can be identified or determined as a match for the feature point in the SDR image; the feature point in the SDR image and the particular feature point in the HDR image form a pair of matched SDR and HDR feature points. Otherwise, the particular feature point may not be identified as such a match. The matching operation may be repeatedly performed for all feature points in the set of SDR image feature points, thereby producing a set of paired matched SDR and HDR feature points.

Block 368 includes computing a geometric transformation, such as a 3×3 2D affine transformation, between the SDR image and the HDR image based in part or in whole on the set of paired matched SDR and HDR feature points. The geometric transformation may be computed or derived using coordinates (e.g., pixel rows and pixel columns, etc.) of matched feature points from the set of paired matched SDR and HDR feature points in the SDR and HDR images.

For each (e.g., kth, etc.) pair of matched SDR and HDR feature points, the 2D coordinates of the HDR feature point in the pair can be expressed as (τ _ix ,τ _iy ) and included in a first vector as τ _i = [τ _ix τ _iy 1], while the 2D coordinates of the SDR feature point can be expressed as (η _ix ,η _iy ) and included in a second vector η _i = [η _ix η _iy 1].

The total number of paired matched SDR and HDR feature points in the set of paired matched SDR and HDR feature points is denoted as N. Vectors generated from the 2D coordinates of feature points in all paired matched SDR and HDR feature points in the set of paired matched SDR and HDR feature points may be collected together as follows:

As mentioned above, the geometric transformation can be represented as a 3×3 matrix as follows:

Mathematically, the first vector and the second vector respectively representing the SDR and HDR feature points in each pair of the set of matched SDR and HDR feature points may be related to each other by a 3×3 matrix representing a geometric transformation as follows:

Alternatively, for all N pairs of matched feature points in the set of pairs of matched SDR and HDR feature points, the vectors representing the SDR and HDR feature points may be related by a 3×3 matrix representing the geometric transformation as follows:

The values of the matrix elements in the 3×3 matrix representing the geometric transformation can be generated or obtained as a solution (e.g., a least square solution, etc.) to an optimization problem that minimizes the transformation or alignment error. More specifically, the optimization problem can be expressed as follows:

The optimized or optimal value of the 3×3 matrix representing the geometric transformation can be obtained via the least squares solution of the optimization problem in the above expression (), as follows:

F ^opt = (T ^T T) ^-1 (T ^T H) (84)

Block 370 includes applying a geometric transformation to one of the SDR and HDR images. In the present example, for example, for each of the three channels such as Y, Cb and Cr in which the HDR image is represented, a transformation is applied to the HDR image so that the HDR pixel positions in the HDR image are shifted to coincide with the SDR pixel positions in the SDR image. Thus, most of the HDR pixels in the HDR image are spatially aligned with the corresponding SDR pixels in the SDR image. The remaining HDR pixels and the remaining SDR pixels (with which no HDR pixels are spatially aligned) can be excluded (e.g., assigned out-of-range pixel values, etc.) for use as (valid) color scales or colors for generating a static SDR to HDR mapping or TPB coefficients or a static inverse 3D-LUT.

Block 372 includes finding corresponding valid color labels or colors in the SDR and HDR images. The valid color labels or colors can be obtained from the codeword values of the spatially aligned pixels in the SDR and HDR images. As described above, the spatially aligned pixels can be generated by applying a geometric transformation, which in turn is generated from the matched SDR and HDR feature points.

In some operational scenarios, in order to enhance the spatial alignment accuracy or reliability of the geometric transformation, after identifying the (initial) matched SDR and HDR feature points with a matching metric below a minimum matching difference threshold, another matching threshold can be applied to select or distinguish a subset of final matched SDR and HDR feature points from the (initial) matched SDR and HDR feature points.

For each spatially aligned pixel in the SDR image, an SDR codeword, such as an SDR Y/Cb/Cr value, may be determined for the spatially aligned pixel in the SDR image. For co-located or spatially aligned pixels in a spatially transformed HDR image generated using a geometric transformation (corresponding to the spatially aligned pixels in the SDR image), a corresponding HDR codeword, such as a corresponding HDR Y/Cb/Cr value, may be determined for the spatially aligned pixel in the HDR image. If the transformed HDR pixel is unusable (e.g., has a value of 0, etc.), the HDR pixel may be discarded or prevented from being considered as part of a matched color scale or codeword.

In each pair (e.g., the i-th pair, etc.) of matched SDR and HDR color labels or colors as generated from a pair of spatially aligned SDR and HDR pixels in the SDR and HDR images, the matched SDR and HDR color labels or colors (or codeword values), respectively, may be given as follows:

All pairs of matched SDR and HDR color labels or colors (or codeword values) in all pairs of spatially aligned SDR and HDR pixels in the SDR and HDR images in all image pairs of SDR and HDR image pairs may be collected together from all images and merged into two matrices as follows:

From SDR images

From HDR images

Block 374 includes using the matrices in expressions (88) and (89) above and As input to solve or generate optimized or optimal TPB coefficients for inverse reshaping or static SDR to HDR mapping.

From the matrix The inverse TPP basis functions and codewords in generate the inverse generation matrix for each channel as follows:

From the matrix Construct a per-channel observation matrix for SDR to HDR mapping as follows:

The optimized or optimal inverse TPB coefficients for channel ch can be solved via a least squares solution as follows:

The predicted or inversely reshaped HDR value for channel ch may be calculated as follows:

Black-box reverse reshaping optimization in two devices (BB2)

In a "BB2" operating scenario, a (reverse) reshaping mapping may be used to map an (ISP captured) SDR image captured by a first video capture device in an SDR capture mode to generate an (ISP mapped) HDR image that simulates the HDR appearance of an (ISP captured) HDR image captured by a second, different video capture device operating in an HDR capture mode. In some operating scenarios, the first video capture device may be a relatively low-end mobile phone that may only capture SDR images or pictures, while the second video capture device may be a relatively high-end phone that may capture HDR images or pictures. Although the first and second video capture devices may operate with different hardware configurations and capabilities, the reshaping mapping is used to reshape the (ISP captured) SDR image captured by the first device into an (ISP mapped) HDR image that approximates the (ISP captured) HDR image captured by the second device.

The reshaping (SDR to HDR) mapping may be modeled based at least in part on a plurality of image pairs formed by (training) SDR images captured by a first device and (training) HDR images captured by a second device. Each of the image pairs of HDR and SDR images includes an SDR image and an HDR image corresponding to the SDR image.

In some "BB2" operational scenarios, some or all of the plurality of image pairs may include (e.g., initially, prior to an image alignment operation, etc.) captured SDR and HDR images that depict real-world visual scenes, such as natural indoor/outdoor scenes, as having spatial and/or temporal alignment errors associated with the first and second video capture devices. As in the "BB1" use cases or operational scenarios, in these "BB2" use cases or operational scenarios, image alignment operations between corresponding SDR and HDR images in the image pairs may be performed, for example, using geometric transformations constructed using selected feature points extracted from the SDR and HDR images. Subsequently, SDR and HDR color scales or colors determined from the aligned SDR and HDR images in the image pairs may be used to generate a reshape (SDR to HDR) mapping as described herein.

In some "BB2" operating scenarios, instead of or in addition to using images from natural scenes, some or all of the plurality of image pairs may include captured SDR and HDR images (e.g., initially, before image alignment operations, etc.), for example, in a laboratory environment, that depict color tables displayed on one or more reference image displays of the same type. For example, the color tables may be generated as 16-bit full HD RGB (color table) TIFF images. These TIFF images containing the color tables may be displayed as perceptually quantized (PQ) video signals on a reference image display such as a PRM TV. The color tables displayed on the reference image display may then be captured by the first and second video capture devices, respectively.

Fig. 2N illustrates an example TIFF image including a color table. As shown, the color table can be a central square in the TIFF image, and the central square includes a plurality of color blocks with a set of distinct colors to be captured and matched between the first device and the second device. Each color in the set of distinct colors can be different from all other colors in the set of distinct colors. The set of distinct colors can be displayed by a reference image display with a set of distinct pixels/codeword values. Therefore, each color in the set of distinct colors can correspond to a corresponding pixel or codeword value in the set of distinct pixels/codeword values.

Different TIFF images may include different color tables with different pluralities of colors or different sets of distinct colors. Each color table in a corresponding one of the different TIFF images may correspond to a corresponding one of the different pluralities or sets of distinct colors.

Each of the four corner rectangles in the TIFF image of Figure 2N includes a checkerboard pattern. The same corner rectangle or the same checkerboard pattern may be included in different TIFF images containing different color tables. The checkerboard pattern at the four corners of the TIFF image can be used as a spatial key or reference mark for estimating a projective transformation, as will be discussed in detail later. Additionally, optionally or alternatively, the TIFF image may include one or more numbers, binary codes, QR codes, etc., as a unique identifier (ID) assigned to or used to identify the TIFF image or the color table therein or a group or multiple colors therein, etc.

In order to find the correspondence/mapping relationship between as many colors as possible between the first video capture device and the second video capture device, the color table in the TIFF image as described herein can include as many different (groups or multiple) colors as possible (corresponding to as many different pixel/codeword values as possible), depending on the display capability of the reference image display to distinguish different colors. In addition, the color table can be displayed with different overall intensities or illuminations to enable the first and second video capture devices to perform different exposure settings under different lighting conditions. Therefore, the same color in the color table displayed in the TIFF image can be displayed with different intensities or illuminations on the reference image display.

FIG. 3E illustrates an example process flow for generating a plurality of different color tables to be included in a plurality of different TIFF images. To generate each color table, the mean and variance values of the colors in the color table (e.g., RGB, etc.) may be determined and used to randomly generate colors using a statistical distribution (type). In some operating scenarios, the statistical distribution represents a Beta distribution or a distribution type. Different combinations of mean and variance values of the statistical distribution or distribution type may be used to generate different color tables having different complex or groups of distinct colors.

Block 382 includes defining or determining a set of possible means in the PQ domain or color space (denoted as ).

In an operating scenario where the reference image display used to display the TIFF image containing the color table is a PRM TV, the maximum and minimum luminance values supported by the reference image display may be within a range between 1000 nits and 0 nits, or even greater dynamic range and contrast. By way of example and not limitation, the minimum and maximum PQ values that the reference image display can display without clipping may be given by: P ₀ =L2PQ(0) and P ₁ =L2PQ(1000), respectively, where L2PQ(·) represents a mapping function from (linear or non-PQ) luminance to (non-linear or PQ) luminance codewords in the PQ domain or color space.

In a non-limiting example, the set A set of different PQ luminance codewords may be defined as corresponding to (linear) luminance values uniformly distributed in a range of (linear) luminance values from L _{lower limit} to L upper limit on a logarithmic scale. The lower limit L _{lower limit} may be set to a value such as 10 ^-4 , while the upper limit L _{upper limit} may be set to a value such as 10 ^2.99 (e.g., a fractional exponent value selected for numerical stability, etc.), which depends in part or in whole on the image capture capabilities of the first device and the second device (e.g., capturing the dimmest and brightest luminances, etc.). The logarithmic scale used as luminance codewords in the captured SDR and HDR images may be approximately linear with the logarithm of the luminance in the range of luminance values. The total number of possible means (or magnitude) in the set of possible means ) can be, but is not necessarily limited to, the following sets:

Block 384 includes defining or determining a set of possible shape coefficients for generating possible variance values for a statistical distribution or type of distribution (eg, a Beta distribution, etc.).

In some operating scenarios, the Beta distribution used to randomly select a color or codeword value is defined or has support in a scaled or closed value interval [0, ₁ ]. Given a PQ value in the PQ codeword value range [P ₀ , P 1 ], the scaled mean μ (where 0<μ<1 or in a closed value interval of the Beta distribution) can be calculated as follows:

The scaled variance (denoted as σ ² ) of the beta distribution within the scaled or closed value interval [0,1] may be adaptively set to be proportional to μ(1-μ) to avoid over/under exposure caused by large (e.g., brightness, etc.) differences in color patches during capture of SDR and HDR images by the first device and the second device, as follows:

σ ² = μ(1-μ)/θ (95)

Wherein θ represents a shape coefficient that affects or determines the shape of the Beta distribution (e.g., more compressed, more expanded, etc.). The value of the shape coefficient may be selected from a set of possible shape coefficients θ to allow the generated color table to have a relatively high diversity of codeword values and/or colors generated therefrom. A non-limiting example of the set of possible shape coefficients may be: θ={3, 6, 9, 12}.

Block 386 includes using the set of possible means and the set of shape coefficients to generate a plurality of all unique combinations of scaled means and shape coefficients for different instances of a statistical (e.g., Beta, etc.) distribution. In the present example, the total number of unique combinations is given as follows: Where |Θ| represents the total number or magnitude of elements in the set Θ. In some operational scenarios, a different TIFF image with a different color table may be generated for each unique combination of the scaling mean and the shape coefficient in a plurality of all unique combinations of the scaling mean and the shape coefficient, so that the total number of different color tables or corresponding different TIFF images is given as

Block 388 includes selecting the current combination of scaled means and shape coefficients as the next combination from the plurality of all unique combinations of scaled means and shape coefficients. A set or more different pixel or codeword values that respectively specify or define a set or more different colors of the current color table corresponding to the current combination of scaled means and shape coefficients may be generated from a Beta distribution, taking into account the scaled mean μ in the current combination and the current scaled variance σ ² in the current combination.

Block 390 includes calculating or defining a Beta distribution as follows:

f (x; α, β) = x ^α-1 (1-x) ^β-1 /B (α, β), where x∈[0,1] (96)

Among them, α and β are the parameters of the Beta distribution; B(α,β) represents the normalization constant. The Beta distribution parameters α and β can be obtained from the mean and variance of the Beta distribution as follows:

α=μν and β=(1-μ)ν, where ν=μ(1-μ)/σ ² -1 (97)

Block 392 includes generating a set of different (e.g., 144, etc.) colors (or pixel/codeword values) for a current color table or current color table image based on a set (e.g., 144, etc.) of random numbers generated from a Beta distribution. Each random number, denoted x (where 0≤x≤1), may be scaled back or converted to a corresponding per-channel pixel or codeword value (referred to as a “PQ value”) within a PQ value range [P ₀ ,P ₁ ] as follows:

_xPQ ＝x( _P1 - _P0 )+ _P0 (98)

A set of PQ values generated from scaling or converting the set of random numbers back into the PQ value range can then be generated from the set of random numbers and used as a set of per-channel codeword values or codewords for the set of colors in the current color table. In some operating scenarios, for each color table, the same Beta distribution (with the same mean and shape coefficient), for example, with three different sets of random numbers, is used to generate per-channel codewords for each of a plurality of (e.g., RGB, etc.) channels of a color space in which the display image is to be rendered by the reference image display. Thus, the average color generated from the averaged colors represented in all color patches in the color table is close to a neutral color or gray.

Block 394 includes generating a current color table or central block of the corresponding TIFF image. The current color table may include a set (e.g., 2D squares, etc.) of color blocks, each of which is a single color given by a specific (cross-channel or composite) pixel or codeword value having three per-channel pixel or codeword values. Three sets of per-channel codewords generated from the same Beta distribution (but with three different sets of random numbers) may be used by the reference image display to drive rendering of the red, blue, and green color block channels in the set of color blocks in the current color table or current TIFF image, respectively.

Additionally, optionally or alternatively, the RGB values of the background color used to specify or generate the background of the color table image may be set to a PQ mean μ _PQ from which the scaled mean of the Beta distribution is derived.

Block 396 includes determining whether the current combination of the scaling mean and the shape coefficient is the last combination of all unique combinations of the scaling mean and the shape coefficient. If so, the process flow ends to generate a plurality of different color tables or different color table images. Otherwise, the process flow returns to block 388.

By way of illustration and not limitation, a total number of different colors or color patches that can be generated for each color table, denoted as n _c , may be 144. The total number of colors or color patches generated from a plurality of different color tables and color table images may be given as: Colors or color blocks, which can be used to determine the correspondence/matching relationship between SDR color labels or codewords captured by the first video capture device in the SDR capture mode and the corresponding HDR color labels or codewords captured by the second video capture device in the HDR capture mode.

In some operational scenarios, multiple TIFF images each containing multiple color tables may be iteratively displayed or rendered on a reference image display such as a PRM TV at a constant playback frame rate and may be captured in SDR and HDR images, respectively, by a first and second video capture device. Since the illumination of the reference image display may vary with viewing angle, the SDR and HDR images may be captured separately by a first and second video capture device (e.g., two phones, etc.) placed at the same position and orientation relative to or with reference to the reference image display. Thus, SDR and HDR video signals or bitstreams containing the captured SDR and HDR images of the color tables or TIFF images as rendered on the reference image display may be generated by the first and second devices or cameras, respectively. Since the playback frame rate of the reference image display may remain constant, it may be relatively easy to determine the number of frames of the color table to establish an image pair in which the SDR image captured by the first video capture device and the HDR image captured by the second video capture device depict the same color table of the multiple color tables. The captured SDR and HDR images may be represented in SDR and HDR YCbCr (or YUV) color spaces, such as in YUV image/video files.

3F illustrates an example process flow for matching SDR and HDR colors between an image pair of captured SDR and HDR images for each color table in a plurality of color tables. To extract the corresponding colors of individual color blocks from captured SDR and HDR images of color table images containing color tables, the captured SDR and HDR images may be converted to the same layout as the (original) color table images displayed on a reference image display.

Block 3002 includes receiving captured (SDR or HDR) checkerboard images that may be obtained by a camera of one of the first and second video capture devices from a displayed checkerboard image displayed on a reference image display. Blocks 3002 to 3006 of the same process flow of FIG. 3F may be performed for captured checkerboard images that are obtained by a camera of the other of the first and second video capture devices from the same displayed checkerboard image displayed on a reference image display.

Block 3004 includes detecting checkerboard corners from a checkerboard image captured by a camera of the device.

Box 3006 includes calculating or calibrating the camera distortion coefficients of the camera using the chessboard angles detected from the captured chessboard image. The 3D (reference) coordinates of the chessboard image displayed on the reference image display may first be determined in a 3D coordinate system fixed to the reference image display. Camera parameters such as the distortion coefficients and other intrinsic parameters of the camera used by the device to generate the captured chessboard image may be calculated as optimized values that generate the best mapping (e.g., minimum error or minimum mismatch, etc.) between the 3D reference coordinates of the chessboard angles in the captured chessboard image and the 2D (image) coordinates of the chessboard angles. The calibration process may be performed in a YUV or RGB color space, and the displayed chessboard image or the captured chessboard image may be converted to or represented in the color space.

2M and 2N illustrate two example checkerboard images detected from captured (HDR and SDR, respectively) images of the checkerboard image. Multiple (e.g., 100, etc.) captured checkerboard images can be used to determine, calculate, or calibrate distortion coefficients or other intrinsic parameters of respective cameras used in the first and second video capture devices to achieve relatively high accuracy (e.g., within half a pixel for camera reprojection error, etc.) in projecting images captured by respective cameras of a displayed image on a reference image display onto a displayed image on the reference image display.

The (camera-specific) distortion coefficients and intrinsic parameters of the cameras of the first and second video capture devices obtained in processing blocks 3002 to 3006 may then be used to analyze or correlate the captured SDR and HDR images from TIFF images containing corresponding color tables as follows.

Block 3008 includes receiving a captured (SDR or HDR) image containing a color table and a TIFF image of a checkerboard corner (or a checkerboard pattern in a corner) (displayed on a reference image display). The captured (SDR or HDR) image may be captured by a camera, having previously captured a checkerboard image (which may not contain a color table) in block 3006 to pre-obtain distortion coefficients for these images.

Block 3010 includes correcting or de-distorting the captured image of the displayed TIFF image, for example using distortion coefficients obtained during a camera calibration process such as block 3006, to correct for camera lens distortion of the camera.

Block 3012 includes detecting checkerboard corners in the captured image. The checkerboard corners are captured from checkerboard corners (eg, four corners thereof, etc.) in a displayed TIFF image on a reference image display, which may be a PPM TV.

Block 3014 includes estimating a projective transformation between image coordinates of the captured image and image coordinates of an original TIFF image displayed on a reference image display and captured in the received captured image.

Block 3016 includes using the estimated projective transformation to correct the captured image to the same layout as the original TIFF image.With the estimated projective transformation, the captured image can be corrected to the same spatial layout as the original TIFF image.

FIG2O illustrates (a) a captured image from a (displayed) TIFF image, (b) a corrected captured image generated by correcting the captured image with camera distortion correction and a projective transformation, and (c) an original TIFF color table image displayed on a reference image display. Since the corrected captured image of FIG2O(b) is generated from a dedistortion or correction operation performed on the captured image of FIG2O(a), some pixels in the corrected image of FIG2O(b) may be undefined.

Block 3018 includes locating and extracting (a set of) individual color blocks from the color table in the corrected captured image. As used herein, a color block having a single corresponding (e.g., RGB, YCbCr, composite, etc.) pixel or codeword value is specified. Corresponding individual codewords that respectively specify individual color blocks in the captured image can be determined based on the pixel or codeword values of the pixels in these individual color blocks in the corrected captured image. In addition, individual original codewords that respectively specify individual original color blocks in the original TIFF image (corresponding to individual color blocks of the corrected captured image) can be determined from the RGB or YUV file of the original TIFF image.

The correspondence/mapping relationship between SDR color blocks or codewords extracted from the (corrected) captured SDR image of the original color table image and HDR color blocks or codewords extracted from the (corrected) captured HDR image of the same original color table image can be established partially or entirely based on separate original color blocks or codewords in the original TIFF image (from which both the (corrected) SDR and HDR images are obtained).

Using multiple TIFF images, multiple SDR images captured from displayed TIFF images by a first video capture device in SDR capture mode, and multiple HDR images captured from the same displayed TIFF images by a second video capture device in HDR capture mode, multiple correspondences or mapping relationships between a set of SDR colors and a set of HDR images can be established.

Several methods can be used to map an SDR image to an HDR image.

In the first method, as in some "BB1" operating scenarios, in some "BB2" operating scenarios, mapped SDR and HDR color scales or colors can be used to generate (e.g., static, etc.) SDR to HDR reshaping operation parameters such as TPB coefficients. These TPB coefficients can be combined with TPB basis functions using the SDR codewords of the SDR image as input parameters to predict the corresponding HDR codewords of the reconstructed HDR image. The static 3D-LUT can be pre-built and deployed to a video capture device (e.g., a first video capture device, etc.) for mapping the SDR image captured by the first video capture device to a reconstructed HDR image that simulates the HDR appearance of the second video capture device. Since the HDR color blocks or codewords used to generate the reshaping operation parameters are extracted from the captured (training) HDR image of the second video capture device, the predicted HDR codewords generated with the reshaping operation parameters are likely to provide a mapped HDR appearance similar to the actual HDR appearance of the actual HDR image captured by the second video capture device in the reconstructed HDR image.

In a second method, in some "BB2" operating scenarios, non-TPB optimization can be used to generate non-TPB reshaping operation parameters, which are used in the non-TPB reshaping operation to map the captured SDR image obtained by the first video capture device to a reconstructed HDR image that simulates the HDR appearance of the second video capture device. The 3D-LUT for the non-TPB reshaping operation can be directly constructed without the relatively high continuity and smoothness supported by the TPB reshaping operation. The non-TPB second method provides a relatively high degree of design freedom and can be relatively flexible, even if different colors represented in nearby 3D-LUT entries/nodes may or may not have relatively high continuity and smoothness. In some operating scenarios, with relatively high degrees of design freedom, the non-TPB second method can be more suitable than the first method for dual-device or "BB2" operating scenarios, in which the SDR image from one device is to be mapped to an HDR image that simulates the HDR appearance of another device.

A non-TPB second method may utilize a 3D mapping table (3DMT) to construct a reverse reshaping map or a reverse lookup table (BLUT) representing the reverse reshaping map. The BLUT may be used to map SDR (e.g., across channels, three channels, etc.) codewords of an SDR image to HDR (e.g., chroma channels, per chroma channels, etc.) codewords of a reconstructed HDR image. Example operations for constructing a reverse reshaping map such as a BLUT from a 3DMT may be found in U.S. Patent Application Serial No. 17/054,495 filed on May 9, 2019, the contents of which are incorporated herein by reference in their entirety as if fully set forth herein.

For illustration purposes only, SDR codewords may be represented in three dimensions or channels Y, Cb, Cr in an SDR YCbCr color space. A 3D mapping table may be generated from a 3D histogram having a plurality of 3D histogram bins. The plurality of 3D histogram bins may correspond to a plurality of color space partitions generated by splitting each dimension or channel by a corresponding positive integer among three positive integers _Qy , _Qcb , _QCr , resulting in a total of ( _Qy × _Qcb × _Qcr ) 3D histogram bins.

Each 3D histogram bin (denoted as q) of a plurality of 3D histogram bins (denoted as Ω ^Q,s ) of a 3D histogram may be assigned a corresponding bin index or three corresponding quantized channel values q=(q _y ,q _Cb ,q _Cr ) in a dimension or channel of an SDR (YCbCr) color space, and stores pixel counts of all SDR color labels or codewords within the SDR color space partition represented by the 3D histogram bin. All bin indices (or quantized channel values) of the plurality of 3D histogram bins may be collected into a set of bin index values denoted as Q, where Q=[Q _y ,Q _C0 ,Q _Cr ].

Additionally, the sum of the HDR codewords mapped to the SDR codewords in each of the plurality of 3D histograms may be calculated and stored for the 3D histogram bin. and Represents the sum of HDR codewords (also referred to as “mapped HDR luma and chroma values”).

An example procedure for generating SDR pixel counts and mapped HDR luminance and chrominance values for 3D histogram bins of a histogram is illustrated in Table 2 below.

Table 2

let represents the (representative) SDR codeword at the center of the qth 3D histogram bin. The representative SDR codewords for all 3D histogram bins may be fixed for all SDR images/frames and pre-computed with an example procedure as shown in Table 3 below.

Table 3

Next, 3D histogram bins having non-zero (SDR) pixel counts may be identified and retained among the plurality of 3D histogram bins, wherein all other 3D histogram bins having zero (SDR) pixel counts are discarded or removed from the plurality of 3D histogram bins.

Let q ₀ ,q ₁ ,…q _k-1 , denote all k 3D histogram bins, each with a non-zero (SDR) pixel count or Where k is a positive integer. For example, using the procedure shown in Table 4 below, the mapped HDR luminance and chrominance values may be calculated for each of the k 3D histogram bins with non-zero (SDR) pixel counts by dividing the sum of the HDR luminance and chrominance values mapped by the SDR codeword for the SDR pixel by the number of SDR pixels in the 3D histogram bin and the pixel count. and The average value of .

Table 4

The 3D-LUT may be constructed from a 3DMT represented by a 3D histogram storing corresponding SDR pixel counts and corresponding mapped HDR codeword values in 3D histogram bins. Each node/entry in the 3D-LUT or 3DMT may be constructed with a mapping key represented by a specific SDR codeword ( _qy , qCb, qCr) represented in each of the k 3D histogram bins and a mapping key represented by a specific SDR codeword (qy, _qCb , _qCr ) calculated for the 3D histogram bin as in Table 4 above. Additionally, optionally or alternatively, for those 3D histogram bins with zero SDR pixel counts, the nearest or neighboring 3D bin with the closest bin index may be used to fill in the missing mapped HDR values, for example by trilinear interpolation.

As described above, example operations for constructing a 3D-LUT from a 3DMT for use as a non-TPB inverse reshaping map or BLUT (e.g., for predicting HDR chroma codewords per chroma channel from cross-channel SDR codewords) are described in the aforementioned U.S. patent application serial number 17/054,495.

In some "BB2" operating scenarios, a luma reverse reshaping map may be used to reverse reshape the SDR luma codewords of an SDR image into predicted or reverse reshaped HDR luma codewords of a corresponding HDR image using a GPR-based model. Example generation of luma reshaping maps using a GPR-based model may be found in U.S. Provisional Patent Application Serial No. 62/887,123, "Efficient user-defined SDR-to-HDR conversion with model templates," filed by Guan-Ming Su and Harshad Kadu on August 15, 2019, and PCT Application Serial No. PCT/US2020/046032, filed on August 12, 2020, the contents of which are incorporated herein by reference in their entirety as fully set forth herein. For example, a CDF matching curve may be generated from each of a plurality of training SDR-HDR image pairs. Using a set (e.g., 15, etc.) of uniformly sampled SDR points within an SDR codeword range (e.g., the entire SDR codeword range, etc.), a corresponding mapped HDR codeword can be found in each of the CDF matching curves. For each sample SDR point in the set of uniformly sampled SDR points, a GPR model can be constructed based on a histogram with multiple (e.g., 128, etc.) luma bin histograms and used to generate a corresponding luma reshaping map.

TPB optimization in editing

Image/video editing is a common application in video capture devices such as mobile devices to allow users to adjust color, contrast, brightness or other user preferences. Image/video editing can be performed or executed on the encoder side before the captured images are compressed into a (compressed) video signal or bitstream. Additionally, optionally or alternatively, image/video editing can be performed or executed on the decoder side after the video signal bitstream is decoded or decompressed.

Image/video editing operations as described herein may be performed in either or both of the HDR domain and the SDR domain. Image/video editing operations in the HDR domain may be performed relatively easily. For example, after editing HDR content or images, the edited HDR content or images may be passed as input or reference HDR images to a video pipeline that generates corresponding reshaped or ISP SDR content or images to be encoded in a video signal or bitstream. In contrast, image/video editing operations in the SDR domain may be relatively challenging. Although HDR to SDR and/or SDR to HDR mappings may be designed or generated to ensure or enhance reversibility between the SDR domain and the HDR domain, image/video editing operations performed with SDR images to be encoded in a video signal or bitstream may result in difficulty or challenging in reversing or reverse reshaping the edited SDR images to generate a reconstructed HDR image that is similar to the reference HDR image.

FIG3G illustrates an example image/video editing operation performed on the encoder side with an upstream device such as a video capture device or a video encoder. As shown, an input HDR image may be represented in an input HDR domain or color space (denoted as “HLG YCbCr”). A TPB-based forward reshaping operation (denoted as “forward TPB”) may be performed to forward reshape the input HDR image into a forward reshaped SDR image represented in a forward reshaped SDR domain or color space (denoted as “SDR YCbCr”). An image/video editing operation (denoted as “RGB domain editing”) may be performed to edit the forward reshaped SDR image so as to generate an edited SDR image represented in a forward reshaped SDR domain or color space (denoted as “edited SDR YCbCr”) after editing. The edited SDR image represented in the forward reshaped SDR domain or color space after editing may be compressed or encoded into a video signal or bitstream using one or more video codecs of an upstream device.

3H illustrates an example image/video editing operation performed on the decoder side with a downstream receiving device or video decoder. As shown, a video signal or bitstream is decoded by one or more video codecs of a downstream device into an SDR image represented in an SDR domain or color space (denoted as "SDR YCbCr"). An image/video editing operation (denoted as "RGB domain editing") may be performed to edit the SDR image so as to generate an edited SDR image represented in a forward reshaped SDR domain or color space (denoted as "edited SDR YCbCr") after editing. A TPB-based reverse reshaping operation (denoted as "reverse TPB") may be performed on the edited SDR image represented in the forward reshaped SDR domain or color space after editing to generate a reverse reshaped or reconstructed HDR image in a reconstructed HDR domain or color space ("PQ YCbCr").

As shown in Figures 3G and 3H, an SDR image encoded into a video signal at the encoder side or decoded from a video signal can be represented in an SDR domain or color space such as an SDR YCbCr domain or color space (e.g., after forward reshaping, etc.), with which the video codec can be programmed or developed to perform image processing operations relatively efficiently. By way of example and not limitation, image/video editing operations can occur in an SDR RGB domain or color space, as illustrated in Figures 3G and 3H. RGB to YCbCr conversion or YCbCr to RGB conversion can be performed by conversions such as SMPTE range conversions (e.g., specified by the standard, etc.).

As more colors are squeezed into the SDR domain or color space, codewords in a forward reshaped domain or color space, such as an SDR YCbCr color space, can often exceed a codeword value range, such as a SMPTE range, in an SDR RGB color space in which image/video editing operations are performed. Applying a YCbCr to RGB conversion, which converts YCbCr codewords in the YCbCr codeword value range (normalized within the value range [0,1]) of the SDR YCbCr color space to RGB codewords of the SDR RGB color space, can cause some RGB codewords to exceed or exceed the SMPTE range (normalized within the value range [0,1]) of the SDR RGB color space. Although these out-of-range codeword values can be clipped, the clipped codeword values may not be able to be restored to the non-clipped original HDR codeword in a reverse reshaping operation (e.g., based on TPB, etc.). Additional operations as described herein can be used to enhance reversibility and reduce or avoid visual artifacts in image/video editing applications.

3I-3M illustrate several example solutions for clipping out-of-range codewords in image/video editing applications.

In some operational scenarios, as illustrated in FIG3I , out-of-range SDRRGB codewords generated from a YCbCr to RGB conversion performed on an SDR YCbCr image at the encoder or decoder side (denoted as “YCbCr to RGB clipping conversion”) may be clipped (e.g., all codewords greater than one (1) are hard-clipped to 1, all codewords less than zero (0) are hard-clipped to 0, etc.) to a valid or specified codeword value range such as [0,1]. Image/video editing operations may then be performed with the SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space (“editing in the RGB domain”). The edited SDR RGB codewords in the SDR RGB color space may be converted and clipped into converted edited SDR YCbCr codewords in a valid or specified codeword value range such as [0,1] in a converted edited SDR YCbCr color space (denoted as "edited SDR YCbCr") through an RGB to YCbCr conversion (denoted as "RGB to YCbCr clipping conversion").

In some operational scenarios, as illustrated in FIG. 3J , out-of-range SDR RGB codewords generated from a YCbCr to RGB conversion performed on an SDR YCbCr image at the encoder or decoder side (denoted as “YCbCr to RGB unclipping conversion”) are not clipped to a valid or specified codeword value range such as [0, 1]. An image/video editing operation (“editing in the RGB domain”) may then be performed with the unclipped (input) SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr to RGB conversion performed with the unclipping operation allows the original information in the (input) SDR RGB codewords to be preserved in the (input) SDR RGB codewords received by the image/video editing operation. The image/video editing operation accepts all possible true codeword values in the input SDR RGB codewords (including less than 0 or greater than 1, e.g., in a limited but wider range of values than [0 1]). Therefore, some information in the input SDR YCbCr codeword can be retained in the input SDR RGB codeword and the edited SDR RGB codeword. An SDR RGB boundary operation (denoted as "RGB boundary [0 1] shearing") can be performed (e.g., optionally based on a user preference of a user operating a video editing application) to shrink or squeeze the edited SDR RGB codeword outside the valid or specified range [0,1] back to the inside of the valid or specified range [0,1], thereby generating a scaled edited SDR RGB codeword within the valid or specified range [0,1]. The scaled edited SDR RGB codeword in the SDR RGB color space can be converted and sheared into a converted scaled edited SDR YCbCr color space (denoted as "edited SDR YCbCr") in a valid or specified codeword value range such as [0,1] by an RGB to YCbCr conversion (denoted as "RGB to YCbCr shear conversion").

In some operational scenarios, as illustrated in FIG. 3K , out-of-range SDR RGB codewords generated from a YCbCr to RGB conversion performed on an SDR YCbCr image at the encoder or decoder side (denoted as “YCbCr to RGB unclipping conversion”) are not clipped to a valid or specified codeword value range such as [0, 1]. An image/video editing operation (“editing in the RGB domain”) may then be performed with the unclipped (input) SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr to RGB conversion performed with the unclipping operation allows the original information in the (input) SDR RGB codewords to be preserved in the (input) SDR RGB codewords received by the image/video editing operation. The image/video editing operation accepts all possible true codeword values in the input SDR RGB codewords (including less than 0 or greater than 1, e.g., in a limited but wider range of values than [0 1]). Therefore, some information in the input SDR YCbCr codeword can be retained in the input SDR RGB codeword and the edited SDR RGB codeword. Unlike those illustrated in Figure 3J, in the operating scenario illustrated in Figure 3K, SDR RGB boundary operations may not be performed. The edited SDR RGB codewords in the SDR RGB color space can be converted and sheared into converted edited SDR YCbCr color space (denoted as "Edited SDR YCbCr") in a valid or specified codeword value range such as [0,1] by RGB to YCbCr conversion (denoted as "RGB to YCbCr shear conversion").

In some operational scenarios, as illustrated in FIG. 3L , out-of-range SDR RGB codewords generated from a YCbCr to RGB conversion performed on an SDR YCbCr image at the encoder or decoder side (denoted as “YCbCr to RGB unclipping conversion”) are not clipped to a valid or specified codeword value range such as [0, 1]. An image/video editing operation (“editing in the RGB domain”) may then be performed with the unclipped (input) SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr to RGB conversion performed with the unclipping operation allows the original information in the (input) SDR RGB codewords to be preserved in the (input) SDR RGB codewords received by the image/video editing operation. The image/video editing operation accepts all possible true codeword values in the input SDR RGB codewords (including less than 0 or greater than 1, e.g., in a limited but wider range of values than [0 1]). Thus, some information in the input SDR YCbCr codewords may be retained in the input SDR RGB codewords as well as in the edited SDR RGB codewords. An SDR RGB boundary operation (denoted as "RGB Boundary Clipping 3D-LUT") may be performed, for example, optionally based on user preferences of a user operating a video editing application to shrink or squeeze edited SDR RGB codewords outside the (3D) boundary supported by the TPB-based reshaping operations back inside the boundary, thereby generating scaled edited SDR RGB codewords within the boundary supported by the TPB-based reshaping operations or well-defined in these reshaping operations. The boundaries supported by the TPB-based reshaping operations may not be regular shapes or cubes that are simply defined with a fixed value range [0,1]. The scaled edited SDR RGB codewords in the SDR RGB color space can be converted and sheared into converted scaled edited SDR YCbCr codewords in a valid or specified codeword value range such as [0,1] in the SDR RGB color space (denoted as "Edited SDR YCbCr") through an RGB to YCbCr conversion (denoted as "RGB to YCbCr shear conversion").

The (input) SDR YCbCr image may be obtained from the original HDR image by forward reshaping or ISP processing with a known (e.g., white box, ISP, etc.) HDR to SDR forward transform or mapping, and the boundaries may be determined and represented as a (TPB boundary clipping) 3D-LUT based in part or in whole on the HDR to SDR forward or mapping and/or any applicable color space conversion matrix. For example, using full grid sampling data covering the entire HDR domain or color space in which the original HDR image is represented, the boundary pixel or codeword values in the forward reshaping or ISP SDR color space may be determined.

After the image/video editing operation ("editing in the RGB domain"), the edited SDR RGB codewords may be scaled or squeezed (or irregularly sheared) into a 3D shape outlined or enclosed by a boundary. In some operating scenarios, since the converted scaled edited SDR YCbCr codewords are already placed within corresponding boundaries in the SDR YCbCr color space supported by the TPB-based inverse reshaping operation, the scaled edited SDR RGB codewords in the SDRRGB color space may be converted to converted scaled edited SDR YCbCr codewords by RGB to YCbCr conversion ("RGB to YCbCr shearing conversion") without further shearing by RGB to YCbCr conversion ("RGB to YCbCr shearing conversion"). In these operating scenarios as illustrated in FIG. 3J, a maximized number of colors may be retained in the converted scaled edited SDR YCbCr codewords and in the inverse reshaped HDR codewords generated from the converted scaled edited SDR YCbCr codewords by the TPB-based inverse reshaping operation while avoiding the generation of color artifacts.

In some operational scenarios, as illustrated in FIG. 3M , out-of-range SDR RGB codewords generated from a YCbCr to RGB conversion performed on an SDR YCbCr image at the encoder or decoder side (denoted as “YCbCr to RGB unclipping conversion”) are not clipped to a valid or specified codeword value range such as [0, 1]. An image/video editing operation (“editing in the RGB domain”) may then be performed with the unclipped (input) SDR RGB codewords in the valid or specified codeword value range in the SDR RGB color space to generate edited SDR codewords. The YCbCr to RGB conversion performed with the unclipping operation allows the original information in the (input) SDR RGB codewords to be preserved in the (input) SDR RGB codewords received by the image/video editing operation. The image/video editing operation accepts all possible true codeword values in the input SDR RGB codewords (including less than 0 or greater than 1, e.g., in a limited but wider range of values than [0 1]). Therefore, some information in the input SDR YCbCr codeword can be retained in the input SDR RGB codeword and the edited SDR RGB codeword. The edited SDR RGB codeword in the SDR RGB color space can be converted and sheared into a converted edited SDR YCbCr codeword in a valid or specified codeword value range such as [0,1] in a converted edited SDR YCbCr color space (denoted as "Edited SDR YCbCr") by an RGB to YCbCr conversion (denoted as "RGB to YCbCr shear conversion"). The converted edited SDR YCbCr codewords in the converted edited SDRYCbCr color space ("edited SDR YCbCr") may be cropped, for example, optionally based on user preferences of a user operating a video editing application by SDR YCbCr boundary operations (denoted as "YCbCr Boundary Cropping 3D-LUT"), which may be performed to shrink or squeeze the converted edited SDR YCbCr codewords outside the (3D) boundary supported by the TPB-based reshaping operations back into the boundary, thereby generating scaled converted edited SDR YCbCr codewords in the scaled converted edited SDR YCbCr color space (denoted as "edited SDRYCbCr") within the boundary supported by the TPB-based reshaping operations or well-defined in these reshaping operations. The boundaries supported by the TPB-based reshaping operations may not be regular shapes, such as cubes, whose sides are simply defined as a valid or specified (normalized) range [0,1].

The (input) SDR YCbCr image may be obtained from the original HDR image by forward reshaping or ISP processing with a known (e.g., white box, ISP, etc.) HDR to SDR forward transform or mapping, and the boundaries may be determined based on the HDR to SDR forward or mapping and/or any applicable color space conversion matrix and represented with a (TPB boundary clipping) 3D-LUT. For example, using full grid sampling data covering the entire HDR domain or color space in which the original HDR image is represented, the boundary pixel or codeword values in the forward reshaping or ISP SDR color space may be determined.

Boundary Clipping 3D-LUT Construction and Clipping

HDR to SDR forward reshaping or mapping such as TPB-based forward reshaping may be a non-linear function. Although the input HDR codewords in the input HDR domain or color space may be well organized in a simple 3D cube, unlike a 3D cube, the boundaries of the mapped SDR codewords generated from applying a (non-linear or TPB) HDR to SDR mapping may be relatively irregular.

In order to perform (TPB) boundary clipping for relatively irregular boundaries, a (TPB) boundary clipping 3D-LUT may be constructed. The 3D-LUT may be searched with an SDR codeword as input (or a lookup key) and the SDR codeword may be returned as a value in response to determining that the SDR codeword is within the boundary. Otherwise, in response to determining that the SDR codeword is outside the boundary, the 3D-LUT may return a clipped SDR codeword different from the original SDR codeword, the clipped SDR codeword being within the boundary.

In some operating scenarios, boundary clipping can be implemented in two levels. In the first level, regular clipping is performed with a range defined by the minimum and maximum values (or lower and upper limits) that define the (3D) codeword range. In the second level, irregular clipping is performed with a (TPB) boundary clipping 3D-LUT.

FIG3N illustrates an example process flow for constructing a (e.g., TPB, etc.) boundary clipping 3D-LUT. The left side of the process flow as shown in FIG3N may be implemented or executed to construct a 3D boundary using the alphaShape technique. The right side of the process flow as shown in FIG3N may be implemented or executed to generate a 3D-LUT for irregular clipping using the constructed 3D boundary.

Block 3022 includes preparing a 3D uniform sampling grid or set of sample values in an input HDR domain or color space, such as an R.2020 (container) domain or color space. (eg, as given in equations (42) and (43) above, etc.) By way of illustration and not limitation, the HDR domain or color space represents the HDR YCbCr color space HLG.

The sampling values in the R.2020YCbCr color space HLG can be converted to corresponding values in the R.2020RGB color space HLG, as indicated in the above expression (44). The converted values in the R.2020RGB color space HLG can be further converted to corresponding values corresponding to the optimized value (a ^opt ) of the parameter a in the aRGB color space HLG, as indicated in the above expression (45). The converted values in the (a)RGB color space HLG can be clipped, as indicated in the above expression (46), and these converted values can be converted to corresponding clipped values in the R.2020RGB color space HLG, as indicated in the above expression (47). The clipped values in the R.2020RGB color space HLG obtained using the above expression (47) can be converted to corresponding clipped values in the R.2020YCbCr color space HLG. As indicated in the above expression (48).

Box 3024 includes the clipped or constrained (HDR YCbC HLG) values (Referred to as “Constrained Input” in FIG. 3N ) Apply TPB-based forward reshaping (Referred to as “Forward TPB” in FIG. 3N ). More specifically, the optimized forward TPB coefficients for TPB-based forward reshaping are related to the forward generation matrix (in the above expression (50)) together or multiplied, the forward generation matrix is the clipped HDR value in the R.2020YCbCr color space HLG is constructed as the input parameter of the forward TPB basis function to obtain The SDR or R.709YCbCr values after mapping or forward reshaping are as follows:

Block 3046 includes converting the representation to The R.709YCbCr values to generate SDR or R.709RGB values in the unclipped RGB domain or color space are as follows:

R.709RGB values, such as converted from R.709YCbCr values, may include values outside a valid or specified range, such as the value range [0, 1].

For each channel, the minimum and maximum values among the R.709RGB values given in expression (100) may be measured or determined as follows:

These extreme values may be used as lower and upper bounds in block 3030 to construct or prepare a uniformly sampled 3D grid or set of sample values in the unclipped RGB domain or color space.

FIG. 2P illustrates an example distribution of R.709RGB values (in an SDR RGB image) as converted from R.709YCbCr values. As shown, the distribution of R.709RGB values in the SDR RGB image has an irregular shape rather than a 3D cube. The irregular shape with a 3D boundary represents the maximum supported SDR RGB color space that can be mapped back to or inversely reshaped into a reconstructed HDR color in a reconstructed HDR domain or color space without losing information. Any SDR RGB color outside of this irregular shape or range will suffer information loss in the inverse reshaping operation.

After performing an image/video editing operation on an SDR RGB image, the resulting (3D) codeword range or distribution of the edited codewords or colors may be wider (e.g., much wider, etc.) than the irregular shape in the SDR RGB color space, resulting in many SDR RGB codewords that are not defined or supported by the inverse reshaping operation.

Additionally, it may be difficult to characterize, represent, or approximate the actual 3D boundaries of irregular shapes using analytical equations or multiple 2D planes used to cut a 3D cube in RGB color space into the irregular shapes.

As described above, a two-level clipping solution can be used to clip the edited codeword back into the maximum supported RGB color space as represented in the irregular shape. More specifically, at the first level, by Regular clipping can be applied by clipping the edited codeword within a fixed 3D range. Regular clipping can utilize a relatively simple clipping function based on if-else or min()/max() operations to reduce the (input) edited codeword range to a range consisting of A 3D cube (including a rectangle) defined or bounded. At the second level, irregular shearing is applied by shearing those codewords (among the codewords after simple shearing edit) that are still outside the irregular shape into the maximum supported SDR RGB color space outlined or specified by the irregular shape. Since the relatively irregular shearing boundaries of the maximum supported SDR RGB color space are not suitable for relatively simple shearing operations with analytical equations, simple codes or operations, the boundary shearing 3D-LUT can be used to perform irregular shearing at the second level of the two-level shearing solution.

Block 3028 of FIG. 3N includes: given a set of SDR RGB codewords In the case of , an alphaShape can be constructed using an available alphaShape construction tool such as the alphaShape() MATLAB function. As used herein, an alphaShape is a shape that encompasses a set of 2-D or 3-D points (such as the set of SDR RGB codewords in the current example). ) is a delimited area or volume.

The alphaShape object created to represent the alphaShape can be manipulated to summarize the set of SDR RGB codewords The bounding polygon of the SDR RGB codeword is tightened or relaxed } The alphaShape around the point, e.g. to fit a non-convex area. Additional points may be added or removed, e.g. to suppress holes or areas or simply suppress the alphaShape.

The function used to construct alphaShape is denoted as f ^αS (Φ, r ^α ), where r ^α represents the radius parameter.

By using this set of SDR RGB codewords Passed as the first input parameter to the function, the bounding polyhedron denoted αS ^(R709) can be constructed by the alphaShape constructor as follows:

2Q to 2T illustrate the set of SDR RGB points or codewords with different r ^α . As shown, larger values of the radius parameter r ^α produce or construct larger polyhedra, which can result in coarser quantization if nearest neighbors outside the SDR RGB points or codewords are searched and used to fill in the clipped values of these outer SDR RGB points. On the other hand, smaller r ^α produces or constructs smaller polyhedra, which results in a higher probability of forming holes and has less effective boundaries. The radius parameter r ^α can be adjusted to different values (e.g., 0.1, 0.5, etc.) for different video editing applications to maximize the effectiveness or coverage of the supported SDR colors and avoid introducing visual artifacts.

alphaShape provides a bounding polygon as a clipping boundary for the irregular shape and allows determining whether a 3D point represented by an SDR RGB codeword is inside or outside the irregular shape.

Let ^IαS (αS,x) denote a binary function for determining whether a 3D point or SDR RGB codeword (denoted as x) is inside an irregular shape. Assume that a returned binary value of "1" indicates inside the irregular shape and a returned binary value of "0" indicates outside.

Let NN ^αS (αS, x) denote an indexing function that receives a given query 3D point or SDR RGB codeword such as x as a second input parameter and returns the nearest neighbors of the query 3D point or SDR RGB codeword x in αS.

As described above, a boundary clipping 3D-LUT can be constructed in the SDR RGB domain or color space to clip any SDR RGB codewords outside the irregular shape representing the maximum supported color space through (TPB-based forward and reverse) reshaping operations.

Block 3030 includes constructing a boundary clipping 3D-LUT as a full-grid 3D-LUT having a plurality of nodes/entries. These nodes/entries in the 3D-LUT include a query SDR codeword as a lookup key and include a return SDR codeword for the query SDR codeword as a value.

For each color channel and The extreme values determined (in block 3026 of FIG. 3N ) may be used as lower or upper bounds of a 3D cube (or rectangle). The query SDR codewords represented or included in the 3D-LUT may form a full grid comprising 3D points sampled uniformly within the range of the 3D codeword defined by the extreme values or lower/upper bounds. For example, for each dimension or color channel represented by a corresponding axis among the R axis, G axis, and B axis, the query SDR codewords in that dimension or color channel may be uniformly sampled. The codeword value range between can be uniformly sampled into corresponding total number of partitions among three total number of partitions _NR , _NG and _NB corresponding to three dimensions or color channels, as follows:

where i∈[0,…, _NR -1], j∈[0,…, _NG -1], k∈[0,…, _NB -1]

Therefore, _the total number of represented query SDR codewords in the 3D- _LUT can be given as _Nu = _NRNGNB .

Block 3032 includes beginning to execute a node processing loop for each node/entry in the 3D-LUT by selecting a current node/entry among multiple nodes/entries in the 3D-LUT (eg, in a sequential or non-sequential loop/iteration order, etc.).

For simplicity, (i, j, k) in the above expression (103) can be vectorized to p. The current node/entry can be the p-th node/entry among the multiple nodes/entries in the 3D-LUT. The search key of the p-th node/entry can be represented by the p- _th represented query SDR RGB codeword represented as up on the LHS of the above expression (103). Given the p-th represented query SDR RGB codeword as the search key, let Represents the current output clipped SDR RGB codeword or value returned by the pth node/entry of the 3D-LUT.

Block 3034 includes checking or determining whether the current or _pth represented query SDR RGB codeword up of the current node/entry is within the alphaShapeαS ^(R709) generated in block 3028 (or the shape constructed using the alpha shape constructor as shown in expression (102) above). In response to determining that the current or _pth represented query SDR RGB codeword up of the current node/entry is within the alphaShapeαS ^(R709) , process flow proceeds to block 3036. Otherwise, process flow proceeds to block 3040.

The plurality of nodes/entries in the 3D-LUT comprises a subset of nodes or entries, each of which has a corresponding lookup key specified by a represented query SDR RGB codeword located within alphaShapeαS ^(R709) . Thus, the subset of nodes or entries comprises nodes or entries that are considered to be within alphaShapeαS ^(R709) .

Block 3040 includes finding the closest node/entry for the current node/entry among the subset of nodes/entries within alphaShapeαS ^(R709) . The closest node/entry has the closest query SDR RGB codeword represented as a lookup key such that the most recent represented query SDR RGB codeword is compared to all other represented query SDR RGB codewords of all other nodes/entries in the node or entry subset within alphaShapeαS ^(R708). has the minimum distance to the current or _pth represented query SDR RGB codeword up, as measured by Euclidean or non-Euclidean distance in the SDR RGB color space.

In some operating scenarios, the index function NN ^αS (αS, x) mentioned above can be used to give the nearest represented query SDR RGB codeword of the current or _pth represented query SDR RGB codeword up as follows:

The nearest query SDR RGB codeword represented represents the boundary clipped value of the current or pth query SDR RGB codeword _up . is set to the return value of the current or p-th node entry in the 3D-LUT, and the current or p-th represented query SDR RGB codeword _up is set to the lookup key of the current or p-th node entry in the 3D-LUT.

Block 3036 includes setting the current or pth represented query SDR RGB codeword _up as the return value of the current or pth node entry in the 3D-LUT (in addition to being the lookup key) and determining whether the current or pth node/entry is the last node or entry of the plurality of nodes/entries in the 3D-LUT. In response to determining that the current or pth node/entry is the last node or entry of the plurality of nodes/entries in the 3D-LUT, process flow proceeds to block 3038. Otherwise, process flow returns to block 3032.

Block 3038 includes outputting the 3D-LUT as a final boundary clipping 3D-LUT in the SDR RGB color space (denoted as ).

The following table 5 illustrates the methods used to generate the final boundary cut The sample program.

Table 5

Given the (final) boundary clipping 3D-LUT In the case of , the aforementioned two-stage solution can be used to relatively efficiently perform boundary clipping on the (e.g., edited, etc.) SDR image. As described above, regular clipping can be performed to ensure that all (e.g., edited, etc.) SDR RGB codewords in the SDR image are within the extreme values or upper/lower limits of the SDR RGB codewords in the SDR RGB color space in which the image/video editing operation has been performed on the SDR image. Regular clipping is performed and then the (if applicable, regularly clipped) SDR codeword is passed as a lookup key to the (final) boundary clipped 3D-LUT to do irregular clipping, do boundary clipping, and use the clipping 3D-LUT from the (final) boundary clipping 3D-LUT The return value is used as the output (or other irregularly cropped, if applicable) SDR codeword in the output (e.g., after cropping and editing, etc.) SDR image.

In various operational scenarios, including but not limited to those illustrated in Figures 3I to 3M, regular and/or irregular cropping operations as described herein may be applied to video capture and/or editing applications to ensure maximum support for reconstructing HDR images from SDR images and preventing/reducing visual artifacts in the reconstructed HDR images.

Example process flow

FIG. 4A illustrates an example process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding devices/modules, transcoding devices/modules, decoding devices/modules, inverse tone mapping devices/modules, tone mapping devices/modules, media devices/modules, reverse mapping generation and application systems, etc.) may perform this process flow. In block 402, a system as described herein constructs sampled HDR color space points distributed in a high dynamic range (HDR) color space. The HDR color space is parameterized by a color source scaling parameter having a candidate value selected from a plurality of candidate values. The color source scaling parameter is used to calculate the color space coordinates of at least one of a plurality of color sources that delineate the HDR color space.

In box 404, the system generates from these sampled HDR color space points in the HDR color space: (a) a reference standard dynamic range (SDR) color space point represented in a reference SDR color space, (b) an input HDR color space point represented in an input HDR color space, and (c) a reference HDR color space point represented in a reference HDR color space point.

In block 406, the system executes a reshaping operation optimization algorithm to generate an optimized forward reshaping mapping and an optimized reverse reshaping mapping chain. The reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points, and the reference HDR color space points as inputs.

In an embodiment, the optimized forward reshaping mapping is used to forward reshape the input HDR image in the input HDR color space into a forward reshaped SDR image in the forward reshaped SDR color space; and the optimized reverse reshaping mapping is used to reverse reshape these forward reshaped SDR images in the forward reshaped SDR color space into a reverse reshaped HDR image.

In an embodiment, the sampled HDR color space points are constructed in the HDR color space without using any image.

In an embodiment, a plurality of chains of optimized forward reshaping mappings and optimized reverse reshaping mappings are generated for a plurality of candidate values of the chromatic aboriginal scaling parameter by the reshaping operation optimization algorithm; each of the plurality of chains of optimized forward reshaping mappings and optimized reverse reshaping mappings includes a corresponding optimized forward reshaping mapping and a corresponding optimized reverse reshaping mapping.

In an embodiment, the sampled HDR color space points are mapped to the reference SDR color space points based at least in part on a predefined HDR to SDR mapping.

In an embodiment, multiple sets of prediction errors are calculated for the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; each set of prediction errors in the multiple sets of prediction errors is calculated for corresponding chains in the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; and the multiple sets of prediction errors are used to select specific candidate values from multiple candidate values of the chromatic aboriginal scaling parameter.

In an embodiment, a particular candidate value of the chroma scaling parameter is used to generate a particular chain of a particular optimized forward reshaping map and a particular optimized reverse reshaping map.

In an embodiment, the specifically optimized forward reshaping map is represented in a forward reshaping three-dimensional lookup table.

In an embodiment, the specifically optimized inverse reshaping mapping is represented in an inverse reshaping three-dimensional lookup table.

In an embodiment, a video encoder applies the optimized forward reshaping mapping to an input HDR image sequence to generate a forward reshaping SDR image sequence and encodes the forward reshaping SDR image sequence into a video signal.

In an embodiment, a video decoder decodes a forward reshaped SDR image sequence from a video signal and applies the optimized inverse reshaping mapping to the forward reshaped SDR image sequence to generate a inverse reshaped HDR image sequence.

In an embodiment, a display image sequence derived from the inverse reshaped HDR image sequence is rendered on an image display operative with the video decoder.

In an embodiment, the HDR color space and the input HDR color space share a common white point.

In an embodiment, the reshaping operation optimization algorithm represents a Backward Error Subtraction for Signal Adjustment (BESA) algorithm.

FIG4B illustrates an example process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device/module, transcoding device/module, decoding device/module, inverse tone mapping device/module, tone mapping device/module, media device/module, reverse mapping generation and application system, etc.) may perform this process flow. In block 422, a system as described herein constructs sampled HDR color space points distributed in an HDR color space. The HDR color space is parameterized by a color source scaling parameter having a candidate value selected from a plurality of candidate values. The color source scaling parameter is used to calculate the color space coordinates of at least one of a plurality of color sources that delineate the HDR color space.

In block 424, the system generates from the sampled HDR color space points in the HDR color space: (a) input SDR color space points represented in the input SDR color space and (b) reference HDR color space points represented in the reference HDR color space point.

In block 426, the system executes a reshaping operation optimization algorithm to generate an optimized inverse reshaping map. The reshaping operation optimization algorithm receives as input the input SDR color space points and the reference HDR color space points.

In an embodiment, the inverse reshaping mapping is used to inversely reshape the SDR image in the input SDR color space into a inversely reshaped HDR image.

In an embodiment, the sampled HDR color space points are constructed in the HDR color space without using any image.

In an embodiment, a plurality of optimized inverse reshaping mappings are generated for a plurality of candidate values of the chroma scaling parameter by the reshaping operation optimization algorithm; each optimized inverse reshaping mapping of the plurality of optimized inverse reshaping mappings comprises a corresponding optimized inverse reshaping mapping.

In an embodiment, multiple sets of prediction errors are calculated for the multiple optimized inverse reshaping mappings; each set of prediction errors in the multiple sets of prediction errors is calculated for a corresponding optimized inverse reshaping mapping in the multiple optimized inverse reshaping mappings; and the multiple sets of prediction errors are used to select a specific candidate value from a plurality of candidate values of the chromatic aboriginal scaling parameter.

In an embodiment, the programmable ISP pipeline processes the sampled HDR color space points into the input SDR color space points based at least in part on optimized values of programmable configuration parameters of the programmable ISP pipeline.

In an embodiment, optimized values of programmable configuration parameters of the programmable ISP pipeline are determined by minimizing an approximation error between ISP SDR images generated from HDR images by the programmable ISP pipeline and reference SDR images generated by applying a predefined HDR to SDR mapping to these same HDR images.

FIG4C illustrates an example process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device/module, transcoding device/module, decoding device/module, inverse tone mapping device/module, tone mapping device/module, media device/module, reverse mapping generation and application system, etc.) may perform this process flow. In block 442, a system as described herein extracts a set of SDR image feature points from a training SDR image and extracts a set of HDR image feature points from a training HDR image.

In block 444, the system matches a subset of one or more SDR image feature points in the set of SDR image feature points with a subset of one or more HDR image feature points in the set of HDR image feature points.

In block 446, the system generates a geometric transformation using the subset of the one or more SDR image feature points and the subset of the one or more HDR image feature points to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image.

In box 448, the system determines a set of pairs of SDR color labels and HDR color labels from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and the training HDR image have been spatially aligned by the geometric transformation.

In block 450, the system generates an optimized SDR-to-HDR mapping based at least in part on the set of pairs of SDR color scales and HDR color scales derived from the training SDR image and the training HDR image.

In block 452, the system applies the optimized SDR-to-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

In an embodiment, the training SDR image and the training HDR image are captured from a three-dimensional (3D) visual scene by capture devices operating in SDR and HDR capture modes, respectively.

In an embodiment, the training SDR image and the training HDR image form a pair of training SDR images and training HDR images among multiple pairs of training SDR images and training HDR images; and the optimized SDR to HDR mapping is generated at least in part based on multiple groups of paired SDR color labels and HDR color labels obtained from the multiple pairs of training SDR images and training HDR images.

In an embodiment, each SDR image feature point in the subset of the one or more SDR image feature points is matched with a corresponding HDR image feature point in the subset of the one or more HDR image feature points; and the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm.

In an embodiment, the common feature point extraction algorithm represents one of the following: binary robust invariant scalable key point algorithm; accelerated segmented test feature algorithm; KAZE algorithm; minimum eigenvalue algorithm; maximum stable extreme area algorithm; directional FAST and rotation algorithm; scale-invariant feature transformation algorithm; accelerated robust feature algorithm; and the like.

FIG4D illustrates an example process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device/module, transcoding device/module, decoding device/module, inverse tone mapping device/module, tone mapping device/module, media device/module, reverse mapping generation and application system, etc.) may perform this process flow. In block 462, a system as described herein performs a corresponding camera distortion correction operation on each training image in a pair of training SDR images and training HDR images to generate a corresponding undistorted image in a pair of undistorted training SDR images and undistorted training HDR images.

In block 464, the system generates a corresponding projective transform in a pair of SDR image projective transforms and HDR image projective transforms using the corner pattern signatures detected from each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images.

In box 466, the system applies each of the projective transforms in the pair of SDR image projective transforms and HDR image projective transforms to a corresponding undistorted image in the pair of undistorted training SDR images and undistorted training HDR images to generate a corresponding rectified image in a pair of rectified training SDR images and rectified training HDR images.

In block 468, the system extracts a set of SDR color labels from the corrected training SDR image and a set of HDR color labels from the corrected training HDR image.

In block 470, the system generates an optimized SDR-to-HDR mapping based at least in part on the set of SDR color labels and the set of HDR color labels derived from the training SDR image and the training HDR image.

In block 472, the system applies the optimized SDR-to-HDR mapping to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

In an embodiment, the training SDR image and the training HDR image are captured from a common color table image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively.

In an embodiment, the common color table image is selected from a plurality of color table images, each of the plurality of color table images comprising a different color scale distribution arranged into a two-dimensional color table.

In an embodiment, the different color scale distributions are generated with random colors randomly selected from a common statistical distribution having a specific statistical mean and variance value combination.

In an embodiment, the common color table image is rendered on a screen of a common reference image display and is captured from the screen by the first capture device and the second capture device.

In an embodiment, the respective camera distortion correction operations are based at least in part on camera specific distortion coefficients generated from a camera calibration process performed with the camera used to acquire the training images.

In an embodiment, a three-dimensional mapping table (3DMT) is obtained using the set of SDR color labels and the set of HDR color labels; and the optimized SDR to HDR mapping is generated based at least in part on the 3DMT.

In an embodiment, the optimized SDR to HDR mapping represents one of the following: a TPB-based mapping or a non-TPB-based mapping.

In an embodiment, the optimized SDR to HDR mapping is one of: a static mapping applied to all non-training SDR images represented in the video signal, or a dynamic mapping generated based at least in part on a particular SDR codeword value distribution for one of the non-training SDR images represented in the video signal.

FIG4E illustrates an example process flow according to an embodiment of the present invention. In some embodiments, one or more computing devices or components (e.g., encoding device/module, transcoding device/module, decoding device/module, inverse tone mapping device/module, tone mapping device/module, media device/module, reverse mapping generation and application system, etc.) may perform this process flow. In block 482, a system as described herein constructs sampled HDR color space points distributed in an HDR color space for representing a reconstructed HDR image.

In box 484, the system converts the sampled HDR color space points to SDR color space points in a first SDR color space in which the SDR image to be edited by the editing device is represented.

In block 486, the system determines a bounding SDR color space rectangle based on extreme SDR codeword values of the SDR color space points in the first SDR color space and determines an irregular 3D shape from the distribution of the SDR color space points.

In block 488, the system constructs sampled SDR color space points distributed in the bounding SDR color space rectangle in the first SDR color space.

The system generates a boundary clipping 3D-LUT using the sampled SDR color space points and the irregular shape in block 490. The boundary clipping 3D-LUT uses the sampled SDR color space points as lookup keys.

In block 492, the system performs a shearing operation on the edited SDR image in the first SDR color space based at least in part on the boundary-cropped 3D-LUT to generate a boundary-cropped edited SDR image in the first SDR color space.

In an embodiment, these cropping operations include first performing regular cropping on the edited SDR image using the delimiting SDR color space rectangle to generate a regularly cropped edited SDR image, and then performing irregular cropping on the regularly cropped edited SDR image using the 3D-LUT to generate the boundary cropped edited SDR image.

In an embodiment, a group of one or more SDR pixels in the SDR image to be edited is edited from one or more first brightness values to one or more second brightness values in the edited image; the one or more second brightness values are different from the one or more first brightness values.

In an embodiment, a group of one or more SDR pixels in the SDR image to be edited is edited from one or more first chromaticity values to one or more second chromaticity values in the edited image; the one or more second chromaticity values are different from the one or more first chromaticity values.

In an embodiment, image details depicted in the SDR image to be edited are removed in the edited SDR image.

In an embodiment, image details not depicted in the SDR image to be edited are added to the edited SDR image.

In an embodiment, the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; the lookup key is equal to the lookup value; the lookup key is within the irregular shape.

In an embodiment, the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; the lookup key is outside the irregular shape, and the lookup value is inside the irregular shape.

In an embodiment, the lookup value is determined based on an index function that takes the irregular shape and the lookup key as input and returns the nearest neighbor of the lookup key as output.

In an embodiment, a computing device such as a display device, a mobile device, a set-top box, a multimedia device, etc. is configured to perform any of the aforementioned methods. In an embodiment, an apparatus includes a processor and is configured to perform any of the aforementioned methods. In an embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause any of the aforementioned methods to be performed.

In an embodiment, a computing device includes one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, causes performance of any of the foregoing methods.

Note that although separate embodiments are discussed herein, any combination of embodiments and/or portions of embodiments discussed herein may be combined to form further embodiments.

Example Computer System Implementation

Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and/or an apparatus including one or more of such systems, devices, or components. The computer and/or IC may execute, control, or implement instructions related to adaptive perceptual quantization of images with enhanced dynamic range, such as those described herein. The computer and/or IC may calculate any of the various parameters or values related to the adaptive perceptual quantization process described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present disclosure. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement methods related to adaptive perceptual quantization of HDR images as described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention can also be provided in the form of a program product. The program product may include any non-transient medium carrying a set of computer-readable signals, the set of computer-readable signals including instructions that, when executed by a data processor, cause the data processor to perform the method of an embodiment of the present invention. The program product according to an embodiment of the present invention may take any of a variety of forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signal on the program product may be optionally compressed or encrypted.

Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents (e.g., functionally equivalent) to the components, including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

According to one embodiment, the technology described herein is implemented by one or more special computing devices. Special computing devices can be hard-wired, for performing these technologies, or can include digital electronic devices that are permanently programmed to perform these technologies, such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), or can include one or more general-purpose hardware processors that are programmed to perform these technologies according to program instructions in firmware, memory, other storage devices, or combinations. This special computing device can also combine customized hard-wired logic, ASIC or FPGA with custom programming to realize these technologies. Special computing devices can be desktop computer systems, portable computer systems, handheld devices, networking devices, or any other equipment that incorporates hard wiring and/or program logic to implement technology.

For example, Figure 5 is a block diagram illustrating a computer system 500 on which an embodiment of the present invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for transmitting information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, cause computer system 500 to become a special-purpose machine customized to perform the operations specified in the instructions.

Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

The computer system 500 may be coupled to a display 512, such as a liquid crystal display, via the bus 502 for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 504 and for controlling cursor movement on the display 512. Typically, this input device has two degrees of freedom in two axes, a first axis (e.g., an x-axis) and a second axis (e.g., a y-axis), allowing the device to specify a position in a plane.

The computer system 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and/or program logic that, in combination with the computer system, enables the computer system 500 to become or be programmed as a special purpose machine. According to one embodiment, the computer system 500 performs the techniques described herein in response to the processor 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions may be read into the main memory 506 from another storage medium, such as the storage device 510. Execution of the sequences of instructions contained in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hardwired circuits may be used in place of or in combination with software instructions.

The term "storage medium" as used herein refers to any non-transitory medium that stores data and/or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and/or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, diskettes, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROMs and EPROMs, flash EPROMs, NVRAMs, any other memory chips or storage boxes.

Storage media are distinct from but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may be initially carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 may receive the data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may optionally be stored on the storage device 510 before or after execution by the processor 504.

Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides bidirectional data communication coupled to network link 520, which is connected to local network 522. For example, communication interface 518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection to a telephone line of a corresponding type. As another example, communication interface 518 can be a local area network (LAN) card for providing a data communication connection with a compatible LAN. A wireless link can also be implemented. In any such implementation, communication interface 518 sends and receives electrical signals, electromagnetic signals, or optical signals carrying digital data streams representing various types of information.

The network link 520 typically provides data communication to other data devices through one or more networks. For example, the network link 520 can provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. The ISP 526 in turn provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 528. Both the local network 522 and the Internet 528 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518 (which carry the digital data to and from the computer system 500) are example forms of transmission media.

Computer system 500 can send messages and receive data, including program code, through the network, network link 520 and communication interface 518. In the Internet example, server 530 can transmit the requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

The received code may be executed by processor 504 as it is received, and/or stored in storage device 510 or other non-volatile storage for later execution.

Equivalents, extensions, alternatives and miscellaneous

In the foregoing description, embodiments of the present invention have been described with reference to many specific details, which may vary depending on the implementation. Therefore, the only and exclusive indication of the claimed embodiments of the present invention and the claimed embodiments of the present invention considered by the applicant is the set of claims issued in a specific form according to the present application, wherein such claim issuance includes any subsequent corrections. Any definition explicitly set forth herein for the terms contained in such claims should govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes not explicitly cited in the claims should not limit the scope of such claims in any way. Therefore, this specification and the drawings should be viewed in an illustrative rather than a restrictive sense.

Example embodiments of the enumeration

The present invention may be embodied in any form described herein, including but not limited to the following enumerated example embodiments (EEE) which describe the structure, features, and functionality of some portions of embodiments of the present invention.

EEE1. A method comprising:

constructing sampled high dynamic range (HDR) color space points distributed in a HDR color space, wherein the HDR color space is parameterized by a chromatic ablation parameter having a candidate value selected from a plurality of candidate values, wherein the chromatic ablation parameter is used to calculate color space coordinates of at least one of a plurality of chromatic ablation delineating the HDR color space;

generating from the sampled HDR color space points in the HDR color space: (a) a reference standard dynamic range (SDR) color space point expressed in a reference SDR color space, (b) an input HDR color space point expressed in an input HDR color space, and (c) a reference HDR color space point expressed in a reference HDR color space point;

executing a reshaping operation optimization algorithm to generate an optimized forward reshaping mapping and an optimized chain of reverse reshaping mappings, wherein the reshaping operation optimization algorithm uses as input the reference SDR color space point, the input HDR color space point, and the reference HDR color space point;

wherein the optimized forward reshaping mapping is used to forward reshape the input HDR image in the input HDR color space into a forward reshaped SDR image in the forward reshaped SDR color space; wherein the optimized reverse reshaping mapping is used to reverse reshape the forward reshaped SDR image in the forward reshaped SDR color space into a reverse reshaped HDR image.

EEE2. A method as described in EEE1, wherein the sampled HDR color space points are constructed in the HDR color space without using any image.

EEE3. A method as described in EEE1 or EEE2, wherein a plurality of optimized forward reshaping mappings and optimized reverse reshaping mapping chains are generated for a plurality of candidate values of the color source scaling parameter through the reshaping operation optimization algorithm; wherein each of the plurality of optimized forward reshaping mappings and optimized reverse reshaping mapping chains includes a corresponding optimized forward reshaping mapping and a corresponding optimized reverse reshaping mapping.

EEE4. A method as described in any one of EEE1 to EEE3, wherein the sampled HDR color space points are mapped to the reference SDR color space points based at least in part on a predefined HDR to SDR mapping.

EEE5. A method as described in any one of EEE1 to EEE4, wherein multiple groups of prediction errors are calculated for the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; wherein each group of prediction errors in the multiple groups of prediction errors is calculated for a corresponding chain in the multiple chains of optimized forward reshaping mappings and optimized reverse reshaping mappings; wherein the multiple groups of prediction errors are used to select a specific candidate value from multiple candidate values of the chromatic algebra scaling parameter.

EEE6. A method as described in EEE5, wherein a specific candidate value of the color source scaling parameter is used to generate a specific chain of a specific optimized forward reshaping mapping and a specific optimized reverse reshaping mapping.

EEE7. A method as described in EEE6, wherein the specifically optimized forward reshaping mapping is represented in a forward reshaping three-dimensional lookup table.

EEE8. A method as described in EEE6 or EEE7, wherein the specifically optimized inverse reshaping mapping is represented in an inverse reshaping three-dimensional lookup table.

EEE9. A method as described in any one of EEE6 to EEE8, wherein the video encoder applies the optimized forward reshaping mapping to an input HDR image sequence to generate a forward reshaped SDR image sequence and encodes the forward reshaped SDR image sequence into a video signal.

EEE10. A method as described in any one of EEE6 to EEE9, wherein the video decoder decodes a forward reshaped SDR image sequence from a video signal and applies the optimized reverse reshaping mapping to the forward reshaped SDR image sequence to generate a reverse reshaped HDR image sequence.

EEE11. A method as described in EEE10, wherein a display image sequence obtained from the inversely reshaped HDR image sequence is rendered on an image display operating with the video decoder.

EEE12. A method as described in any one of EEE1 to EEE10, wherein the HDR color space and the input HDR color space share a common white point.

EEE13. A method as described in any one of EEE1 to EEE10, wherein the reshaping operation optimization algorithm represents a Backward Error Subtraction with Neutral Color Preservation for Signal Adjustment (BESA) algorithm.

EEE14. A method comprising:

constructing sampled high dynamic range (HDR) color space points distributed in a HDR color space, wherein the HDR color space is parameterized by a chromatic ablation parameter having a candidate value selected from a plurality of candidate values, wherein the chromatic ablation parameter is used to calculate color space coordinates of at least one of a plurality of chromatic ablation delineating the HDR color space;

generating from the sampled HDR color space points in the HDR color space: (a) an input standard dynamic range (SDR) color space point represented in an input SDR color space and (b) a reference HDR color space point represented in a reference HDR color space point;

executing a reshaping operation optimization algorithm to generate an optimized inverse reshaping mapping, wherein the reshaping operation optimization algorithm receives as input the input SDR color space point and the reference HDR color space point;

The inverse reshaping mapping is used to inversely reshape the SDR image in the input SDR color space into a inversely reshaped HDR image.

EEE15. A method as described in EEE14, wherein the sampled HDR color space points are constructed in the HDR color space without using any image.

EEE16. A method as described in EEE14 or EEE15, wherein multiple optimized inverse reshaping mappings are generated for multiple candidate values of the color source scaling parameter through the reshaping operation optimization algorithm; wherein each of the multiple optimized inverse reshaping mappings includes a corresponding optimized inverse reshaping mapping.

EEE17. A method as described in any one of EEE14 to EEE16, wherein multiple groups of prediction errors are calculated for the multiple optimized inverse reshaping mappings; wherein each group of prediction errors in the multiple groups of prediction errors is calculated for a corresponding optimized inverse reshaping mapping in the multiple optimized inverse reshaping mappings; wherein the multiple groups of prediction errors are used to select a specific candidate value from multiple candidate values of the color source scaling parameter.

EEE18. A method as described in any one of EEE14 to EEE17, wherein a programmable image signal processor (ISP) pipeline processes the sampled HDR color space points into the input SDR color space points at least in part based on optimized values of programmable configuration parameters of the programmable ISP pipeline.

EEE19. A method as described in any one of EEE14 to EEE18, wherein the optimized values of the programmable configuration parameters of the programmable ISP pipeline are determined by minimizing the approximate error between an ISP SDR image generated from an HDR image by the programmable ISP pipeline and a reference SDR image generated by applying a predefined HDR to SDR mapping to the same HDR image.

EEE20. A method comprising:

Extracting a set of SDR image feature points from the training standard dynamic range (SDR) images and extracting a set of HDR image feature points from the training high dynamic range (HDR) images;

matching a subset of one or more SDR image feature points in the set of SDR image feature points with a subset of one or more HDR image feature points in the set of HDR image feature points;

generating a geometric transformation using the subset of the one or more SDR image feature points and the subset of the one or more HDR image feature points to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image;

determining a set of pairs of SDR color labels and HDR color labels from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and the training HDR image have been spatially aligned by the geometric transformation;

generating an optimized SDR to HDR mapping based at least in part on the set of pairs of SDR color scales and HDR color scales derived from the training SDR image and the training HDR image;

The optimized SDR-to-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

EEE21. A method as described in EEE20, wherein the training SDR image and the training HDR image are captured from a three-dimensional (3D) visual scene by a capture device operating in SDR and HDR capture modes, respectively.

EEE22. A method as described in EEE20 or EEE21, wherein the training SDR image and the training HDR image form a pair of training SDR images and training HDR images among multiple pairs of training SDR images and training HDR images; wherein the optimized SDR to HDR mapping is generated at least in part based on multiple groups of paired SDR color labels and HDR color labels obtained from the multiple pairs of training SDR images and training HDR images.

EEE23. A method as described in any one of EEE20 to EEE22, wherein each SDR image feature point in the subset of the one or more SDR image feature points is matched with a corresponding HDR image feature point in the subset of the one or more HDR image feature points; wherein the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm.

EEE24. A method as described in EEE23, wherein the common feature point extraction algorithm represents one of the following items: binary robust invariant scalable key point algorithm; accelerated segmented test feature algorithm; KAZE algorithm; minimum eigenvalue algorithm; maximum stable extreme area algorithm; directional FAST and rotation algorithm; scale-invariant feature transformation algorithm; or accelerated robust feature algorithm.

EEE25. A method comprising:

performing a corresponding camera distortion correction operation on each training image in a pair of training standard dynamic range (SDR) images and training high dynamic range (HDR) images to generate a corresponding undistorted image in a pair of undistorted training SDR images and undistorted training HDR images;

generating a corresponding projection transform in a pair of SDR image projection transforms and HDR image projection transforms using corner pattern markers detected from each undistorted image in the pair of undistorted training SDR images and undistorted training HDR images;

Applying each of the pair of SDR image projective transforms and HDR image projective transforms to a corresponding undistorted image in the pair of undistorted training SDR images and undistorted training HDR images to generate a corresponding rectified image in a pair of rectified training SDR images and rectified training HDR images;

Extracting a set of SDR color labels from the corrected training SDR image and a set of HDR color labels from the corrected training HDR image;

generating an optimized SDR to HDR mapping based at least in part on the set of SDR color scales and the set of HDR color scales derived from the training SDR image and the training HDR image;

The optimized SDR-to-HDR mapping is applied to one or more non-training SDR images to generate one or more corresponding non-training HDR images.

EEE26. The method of EEE25, wherein the training SDR image and the training HDR image are captured from a common color table image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively.

EEE27. A method as described in EEE26, wherein the common color table image is selected from a plurality of color table images, each of the plurality of color table images comprising a different color scale distribution arranged into a two-dimensional color table.

EEE28. A method as described in EEE27, wherein the different color scale distributions are generated using random colors randomly selected from a common statistical distribution having a specific statistical mean and variance value combination.

EEE29. The method as described in any one of EEE25 to EEE28, wherein the common color table image is rendered on a screen of a common reference image display and is captured from the screen by the first capture device and the second capture device.

EEE30. A method as described in any one of EEE25 to EEE29, wherein the corresponding camera distortion correction operation is at least partially based on camera-specific distortion coefficients generated from a camera calibration process performed using a camera used to acquire the training image.

EEE31. A method as described in any one of EEE25 to EEE30, wherein the set of SDR color standards and the set of HDR color standards are used to obtain a three-dimensional mapping table (3DMT); wherein the optimized SDR to HDR mapping is generated at least in part based on the 3DMT.

EEE32. A method as described in any one of EEE25 to EEE31, wherein the optimized SDR to HDR mapping represents one of the following items: a tensor product B-spline (TPB) based mapping or a non-TPB based mapping.

EEE33. A method as described in any one of EEE25 to EEE32, wherein the optimized SDR to HDR mapping is one of: a static mapping applied to all non-training SDR images represented in the video signal, or a dynamic mapping generated at least in part based on a particular SDR codeword value distribution of one of the non-training SDR images represented in the video signal.

EEE34. A method comprising:

Constructing sampled high dynamic range (HDR) color space points distributed in a high dynamic range (HDR) color space for representing a reconstructed HDR image;

converting the sampled HDR color space points into SDR color space points in a first standard dynamic range (SDR) color space, the SDR image to be edited by the editing device being represented in the first SDR color space;

determining a bounding SDR color space rectangle based on extreme SDR codeword values of the SDR color space points in the first SDR color space and determining an irregular three-dimensional (3D) shape from a distribution of the SDR color space points;

constructing sampled SDR color space points distributed in the delimited SDR color space rectangle in the first SDR color space;

generating a boundary clipping 3D lookup table (3D-LUT) using the sampled SDR color space point and the irregular shape, wherein the boundary clipping 3D-LUT uses the sampled SDR color space point as a lookup key;

A shearing operation is performed on the edited SDR image in the first SDR color space based at least in part on the boundary-cropped 3D-LUT to generate a boundary-cropped edited SDR image in the first SDR color space.

EEE35. As described in the method of EEE34, the cropping operation includes first using the delimiting SDR color space rectangle to perform regular cropping on the edited SDR image to generate a regularly cropped edited SDR image, and then using the 3D-LUT to perform irregular cropping on the regularly cropped edited SDR image to generate the boundary cropped edited SDR image.

EEE36. A method as described in EEE34 or EEE35, wherein a group of one or more SDR pixels in the SDR image to be edited are edited from one or more first brightness values to one or more second brightness values in the edited image; wherein the one or more second brightness values are different from the one or more first brightness values.

EEE37. A method as described in any one of EEE34 to EEE36, wherein a group of one or more SDR pixels in the SDR image to be edited are edited from one or more first chromaticity values to one or more second chromaticity values in the edited image; wherein the one or more second chromaticity values are different from the one or more first chromaticity values.

EEE38. A method as described in any one of EEE34 to EEE37, wherein image details depicted in the SDR image to be edited are removed from the edited SDR image.

EEE39. A method as described in any one of EEE34 to EEE38, wherein image details not depicted in the SDR image to be edited are added to the edited SDR image.

EEE40. A method as described in any one of EEE34 to EEE40, wherein the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; wherein the lookup key is equal to the lookup value; wherein the lookup key is within the irregular shape.

EEE41. A method as described in any one of EEE34 to EEE40, wherein the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; wherein the lookup key is outside the irregular shape and the lookup value is inside the irregular shape.

EEE42. A method as described in EEE41, wherein the lookup value is determined based on an index function, which takes the irregular shape and the lookup key as input and returns the nearest neighbor of the lookup key as output.

EEE43. An apparatus comprising a processor and configured to perform any of the methods described in EEE 1 to 42.

EEE44. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for performing a method using one or more processors according to any of the methods described in EEE 1 to 42. EEE45.

EEE45. A computer system configured to perform any of the methods described in EEE 1 to 42. EEE45.

Claims (15)

1. A method, comprising:

Extracting a set of Standard Dynamic Range (SDR) image feature points from a training SDR image and extracting a set of High Dynamic Range (HDR) image feature points from a training HDR image;

Matching a subset of one or more SDR image feature points of the set of SDR image feature points with a subset of one or more HDR image feature points of the set of HDR image feature points;

Generating a geometric transformation using the subset of the one or more SDR image feature points and the subset of the one or more HDR image feature points to spatially align a set of SDR pixels in the training SDR image with a set of HDR pixels in the training HDR image;

determining a set of paired SDR color patches and HDR color patches from the set of SDR pixels in the training SDR image and the set of HDR pixels in the training HDR image after the training SDR image and the training HDR image have been spatially aligned by the geometric transformation;

Generating an optimized SDR-to-HDR mapping based at least in part on the set of paired SDR color patches and HDR color patches derived from the training SDR image and the training HDR image;

The optimized SDR-to-HDR mapping is applied to one or more non-trained SDR images to generate one or more corresponding non-trained HDR images.

2. The method of claim 1, wherein the training SDR image and the training HDR image are captured from a three-dimensional (3D) visual scene by a capture device operating in SDR and HDR capture modes, respectively.

3. The method of claim 1 or 2, wherein the training SDR image and the training HDR image form a pair of training SDR image and training HDR image of a plurality of pairs of training SDR image and training HDR image; wherein the optimized SDR-to-HDR mapping is generated based at least in part on multiple sets of paired SDR color patches and HDR color patches derived from the multiple pairs of training SDR images and training HDR images.

4. The method of any of claims 1 to 3, wherein each SDR image feature point of the subset of one or more SDR image feature points is matched to a respective HDR image feature point of the subset of one or more HDR image feature points; wherein the SDR image feature points and the HDR image feature points are extracted from the training SDR image and the HDR image, respectively, using a common feature point extraction algorithm.

5. A method, comprising:

performing a respective camera distortion correction operation on each training image of a pair of training Standard Dynamic Range (SDR) images and training High Dynamic Range (HDR) images to generate a respective undistorted image of a pair of undistorted training SDR images and undistorted training HDR images;

Generating a pair of SDR image projective transforms and a corresponding projective transform of an HDR image projective transform using the detected angular pattern markers from each of the pair of undistorted training SDR images and the undistorted training HDR image;

Applying each projective transformation of the pair of SDR image projective transformations and HDR image projective transformations to a respective undistorted image of the pair of undistorted training SDR images and undistorted training HDR images to generate a respective corrected image of a pair of corrected training SDR images and corrected training HDR images;

extracting a set of SDR color patches from the corrected training SDR image and extracting a set of HDR color patches from the corrected training HDR image;

Generating an optimized SDR-to-HDR mapping based at least in part on the set of SDR color patches and the set of HDR color patches derived from the training SDR image and the training HDR image;

The optimized SDR-to-HDR mapping is applied to one or more non-trained SDR images to generate one or more corresponding non-trained HDR images.

6. The method of claim 5, wherein the training SDR image and the training HDR image are captured from a common color table image by a first capture device operating in an SDR capture mode and a second capture device operating in an HDR capture mode, respectively.

7. The method of any of claims 5 or 6, wherein the common color table image is rendered on a screen of a common reference image display and captured from the screen by the first and second capture devices.

8. The method of any of claims 5 to 7, wherein the respective camera distortion correction operations are based at least in part on camera specific distortion coefficients generated from a camera calibration process performed with a camera used to acquire the training images.

9. The method of any of claims 5 to 8, wherein the set of SDR color patches and the set of HDR color patches are used to derive a three-dimensional mapping table (3 DMT); wherein the optimized SDR-to-HDR mapping is generated based at least in part on the 3 DMT.

10. The method of any of claims 5 to 9, wherein the optimized SDR-to-HDR mapping represents one of: tensor product B spline (TPB) based mapping or non-TPB based mapping.

11. A method, comprising:

constructing sampled High Dynamic Range (HDR) color space points distributed in an HDR color space for representing a reconstructed HDR image;

converting the sampled HDR color space point to a first Standard Dynamic Range (SDR) color space point in an SDR color space in which an SDR image to be edited by an editing device is represented;

Determining a bounding SDR color space rectangle based on extreme SDR codeword values of the SDR color space points in the first SDR color space and determining an irregular three-dimensional (3D) shape from a distribution of the SDR color space points;

Constructing sampling SDR color space points distributed in the bounding SDR color space rectangle in the first SDR color space;

Generating a boundary clipping 3D lookup table (3D-LUT) using the sampled SDR color space points and the irregular shape, wherein the boundary clipping 3D-LUT uses the sampled SDR color space points as lookup keys;

A clipping operation is performed on the edited SDR image in the first SDR color space based at least in part on the boundary clipping 3D-LUT to generate a clipped edited SDR image of a boundary in the first SDR color space.

12. The method of claim 11, wherein the cropping operation comprises first performing a regular cropping on the edited SDR image using the bounding SDR color space rectangle to generate a regular cropped edited SDR image, and then performing an irregular cropping on the regular cropped edited SDR image using the 3D-LUT to generate the boundary cropped edited SDR image.

13. The method of any of claims 11 or 12, wherein the 3D-LUT comprises one or more nodes, each of the nodes comprising a lookup key and a lookup value; wherein the search key is outside the irregular shape and the search value is inside the irregular shape.

14. An apparatus comprising a processor and configured to perform any of the methods of claims 1-13.

15. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions for performing any one of the methods of claims 1-13 with one or more processors.