TW202539237A - Method and apparatus of adaptive loop filter with additional modes and taps related to cccm and fixed filters in video coding - Google Patents

Abstract

A method and apparatus for in-loop filtering of reconstructed video are disclosed. According to the method, reconstructed pixels are received, wherein the reconstructed pixels comprise a current block and the current block comprises a first-colour block and a second-colour block. A target fixed filter set is selected from multiple fixed filter sets according to filter-selection indication at a CTB (Coding Tree Block) level for filtering the current block using a target filter type from a filter-type group comprising chroma ALF (Adaptive Loop Filter), CCALF (Cross-Component ALF), or both. The target fixed filter set is applied to the current block to generate a filtered current block. The filtered current block is provided.

Description

Methods and apparatus for adaptive loop filters with additional modes and taps associated with cross-component chromaticity models and fixed filters in video encoding and decoding.

This invention relates to a video encoding/decoding system using an Adaptive Loop Filter (ALF). Specifically, this invention discloses a method and apparatus for improving performance by selecting a fixed set of filters for a chroma ALF and/or a cross-component ALF (CCALF) based on instructions at the Coding Tree Block (CTB) level.

Versatile Video Coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET), which comprises the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). This standard has been published as an ISO standard: ISO/IEC 23090-3:2021, Information technology – Coded representation of immersive media – Part 3: Versatile Video Coding, published in February 2021. VVC is based on its predecessor, High Efficiency Video Coding (HEVC), and improves encoding and decoding efficiency by adding more encoding and decoding tools, and can handle various types of video sources, including 3D video signals.

Figure 1A illustrates an exemplary adaptive inter-frame/intra-frame video encoding system that includes loop processing. For intra-frame prediction, the prediction data is derived from previously encoded video data in the current image. For inter-frame prediction 112, motion estimation (ME) is performed at the encoder, and motion compensation (MC) is performed based on the result of ME to provide prediction data derived from other images and motion data. Switch 114 selects either intra-frame prediction 110 or inter-frame prediction 112 and provides the selected prediction data to adder 116 to form the prediction error, also known as the residual. The prediction error is then processed by a transform (T) 118 and then quantized (Q) 120. The residuals after transform and quantization are then encoded by an entropy encoder 122 and included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged with additional information, such as motion and encoding/decoding modes associated with intra-frame and inter-frame predictions, and parameters associated with the loop filter applied to the underlying image region. Additional information associated with intra-frame prediction 110, inter-frame prediction 112, and the loop filter 130, as shown in Figure 1A, is provided to the entropy encoder 122. When using inter-frame prediction mode, the reference picture or multiple reference pictures must also be reconstructed at the encoder. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) 124 and inverse transformation (IT) 126 to recover the residuals. The residuals are then added to the prediction data 136 at reconstruction (REC) 128 to reconstruct the video data. The reconstructed video data can be stored in the Reference Picture Buffer (Ref. Pic. Buffer) 134 and used for prediction of other frames.

As shown in Figure 1A, the input video data undergoes a series of processing steps in the encoding system. The reconstructed video data from REC 128 may be subject to various forms of degradation due to these processing steps. Therefore, an in-loop filter module (ILPF) 130 is typically used to reconstruct the video data, which is then stored in the reference image buffer 134 to improve video quality. For example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF) can be used. Loop filter information may need to be incorporated into the bitstream so that the decoder can correctly recover the required information. Therefore, loop filter information is also provided to entropy encoder 122 for inclusion in the bit stream. In Figure 1A, loop filter 130 is applied to reconstruct the video, and then the reconstructed sample is stored in reference image buffer 134. The system in Figure 1A is intended to illustrate an exemplary structure of a typical video encoder. It may correspond to a High Efficiency Video Codec (HEVC) system, VP8, VP9, H.264, or VVC.

As shown in Figure 1B, the decoder can use the same or partially the same functional modules as the encoder, except for transform 118 and quantization 120, since the decoder only needs inverse quantization 124 and inverse transform 126. The decoder uses an entropy decoder (ED) 140 to decode the video bitstream into quantized transform coefficients and the required encoding/decoding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). The intra-frame prediction 150 at the decoder end does not need to perform a mode search. Instead, the decoder only needs to generate intra-frame predictions based on the intra-frame prediction information received from the entropy decoder 140. Furthermore, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) based on the inter-frame prediction information received from the entropy decoder 140, without needing to perform motion estimation.

According to VVC, the input image is divided into non-overlapping square regions called Coding Tree Units (CTUs), similar to HEVC. Each CTU can be further divided into one or more smaller Coding Units (CUs). The resulting CU regions can be square or rectangular. Additionally, VVC divides CTUs into Prediction Units (PUs) as units for applying prediction processes (such as inter-frame prediction, intra-frame prediction, etc.).

This invention discloses a method and apparatus for improving the performance of an adaptive loop filter (ALF).

A method and apparatus for loop filtering in video reconstruction are disclosed. According to the method, reconstructed pixels are received, wherein the reconstructed pixels comprise a current block, and the current block comprises a first color block and a second color block. A target set of fixed filters is selected from multiple sets of fixed filters according to a filter selection instruction at the Coding Tree Block (CTB) level, for filtering the current block using a target filter type from a filter type group, which includes an Adaptive Loop Filter (ALF), a Cross-Component ALF (CCALF), or both. Apply the target-fixed filter set to the current block to generate a filtered current block. Provide the filtered current block.

In one embodiment, at the CTB level, a sample generated from a cross-component model and/or a sample generated from a cross-component model with fixed filters is selected for the luminance ALF, the chromaticity ALF, the CCALF, or a combination thereof.

In one embodiment, multiple Adaptation Parameter Sets (APSs) are selected at the image level, tile level, slice level, or a combination thereof, and a target APS is selected from the multiple APSs at the CTB level for the step of selecting the target fixed filter set.

In one embodiment, a flag is signaled or parsed at the image level, tile level, slice level, or a combination thereof to control the use of fixed filters at that CTB level, and the use of fixed filters is for that filter type group, which further includes luminance ALF. In one embodiment, the flag is used to control whether the selection of a fixed filter set is enabled at the CTB level. In another embodiment, the flag is used to indicate that only one fixed filter set is selected. In yet another embodiment, the flag is used to enable the selection of the Adaptation Parameter Set (APS) ID and the APS filter set at the CTB level. In another embodiment, the flag is used to indicate that only one APS (Adaptive Parameter Set) ID and APS filter set are selected.

In one embodiment, a cross-component model is selected as the target filter type. In another embodiment, the sample generated by the cross-component model is directly used as the current block after filtering.

The components of this invention, as generally described herein and shown in the figures, can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the invention's system and method is merely a representative illustration of selected embodiments of the invention and is not intended to limit the scope of the claimed invention.

The terms “an embodiment,” “an embodiment,” or similar expressions used in this specification indicate that a particular feature, structure, or characteristic associated with that embodiment may be included in at least one embodiment of the invention. Therefore, when the phrases “in an embodiment” or “in an embodiment” appear in different places in this specification, they do not necessarily refer to the same embodiment.

Furthermore, the described features, structures, or characteristics can be combined in one or more embodiments in any suitable manner. However, those skilled in the art will recognize that the invention can be practiced without including one or more specific details, or other methods, components, etc. In some cases, well-known structures or operations are not shown or described in detail to avoid obscuring certain aspects of the invention. The illustrated embodiments of the invention will be best understood by referring to the accompanying drawings, in which the same parts are labeled with the same numbers throughout the figures. The following description is by way of example only, illustrating only certain selected embodiments of apparatus and methods consistent with the invention claimed herein.

Adaptive Loop Filter (ALF) in VVC

In VVC, the Adaptive Loop Filter (ALF) is applied using block-based filter adaptation technology. For the luminance component, a filter is selected from 25 filters for each 4×4 block based on the direction and activity of the local gradient.

1. Filter shape

Two rhombus filter shapes were used (as shown in Figure 2). A 7×7 rhombus shape 220 was used for the luminance component, while a 5×5 rhombus shape 210 was used for the chrominance component.

2. Block Classification

For the luminance component, each The block is classified into one of 25 categories. The classification index C is based on its directionality. And the quantitative value of activity Derivatives, as shown below:

In order to calculate and First, the gradients in the horizontal, vertical, and two diagonal directions are calculated using a one-dimensional Laplacian operator:

Where index and It refers to The coordinates of the top left sample within the block. Indicates coordinates The reconstructed sample at the location.

To reduce the complexity of block classification, one-dimensional Laplacian calculations using subsampling are applied to the vertical direction (Figure 3A) and the horizontal direction (Figure 3B). As shown in Figures 3C-D, the same subsampling locations are used for gradient calculations in all directions (Figure 3C). And in the 3D diagram ).

then, The maximum and minimum values of the horizontal and vertical gradients are set as follows:

,

The maximum and minimum values of the gradients in the two diagonal directions are set as follows:

, .

In order to derive the directionality value These values are compared with each other and with two threshold values. and Comparison:

Step 1. If and If all are true, then Set to .

Step 2. If If yes, proceed to step 3; otherwise, proceed to step 4.

Step 3. If Then Set to Otherwise Set to .

Step 4. If Then Set to Otherwise Set to .

Activity Value The calculation is as follows:

Further quantization to the range of 0 to 4 (inclusive), the quantized value is represented as .

The chromaticity components in the image are not categorized.

3. Geometric transformations of filter coefficients and clipping values

Before filtering each 4×4 luminance block, the filter coefficient is adjusted based on the gradient value calculated for that block. and the corresponding filter clipping value Apply geometric transformations, such as rotation or diagonal and vertical flipping. This is equivalent to applying these transformations to the samples in the filter support region. The aim is to make the different blocks applying ALF more similar through alignment directionality.

Three geometric transformations are introduced, including diagonal rotation, vertical rotation, and rotation:

Diagonal flip:

Vertical flip: , ,

Rotation: , ,

in It's the size of the filter. These are coefficient coordinates that make the position Located in the upper left corner, position Located in the bottom right corner. Based on the gradient value calculated for this block, the transformation is applied to the filter coefficient f(k, l) and the clipping value. The relationship between the transformation and the four gradients in the four directions is summarized in the table below.

Table 1. Mapping of gradients and transformations calculated for a block. Gradient values Transformation Transpose indexes g _d2 < g _d1 and g _h < g _v No transformation 0 g _d2 < g _d1 and g _v < g _h Diagonal 1 g _d1 < g _d2 and g _h < g _v Vertical flip 2 g _d1 < g _d2 and g _v < g _h Rotation 3

4. Filtering process

At the decoder end, when ALF is enabled in CTB, each sample in CU All will be filtered, resulting in the generation of sample values. As shown below:

in This indicates the filter coefficients for decoding. It is a clipping function. This represents the clipping parameters for decoding. The variables k and l range from -L/2 to L/2, where L represents the length of the filter. Clipping function. Corresponding to a function The pruning operation introduces nonlinearity, making ALF more efficient by reducing the influence of neighboring sample values that differ too much from the current sample value.

5. Cross-component adaptive loop filter

CC-ALF uses luminance sample values to refine each chromaticity component by applying an adaptive linear filter to the luminance channel and then using the output of that filter operation for chromaticity refinement. Figure 4A provides a system hierarchy diagram of the CC-ALF process relative to the SAO, luminance ALF, and chromaticity ALF processes. As shown in Figure 4A, each color component (i.e., Y, Cb, and Cr) is processed by its corresponding SAO (i.e., SAO Luma 410, SAO Cb 412, and SAO Cr 414). Following the SAO, ALF Luma 420 is applied to the luminance processed by the SAO, while ALF Chroma 430 is applied to the Cb and Cr processed by the SAO. However, there are cross-component terms from luminance to chrominance components (i.e., CC-ALF Cb 422 and CC-ALF Cr 424). The outputs of the cross-component ALF are added to the outputs of the ALF Chroma 430 via adders 432 and 434, respectively.

In CC-ALF, filtering is accomplished by applying linear, diamond-shaped filters (such as filters 440 and 442 in Figure 4B) to the luminance channel. In Figure 4B, blank circles represent luminance samples, and dot-filled circles represent chrominance samples. One filter is used for each chrominance channel, as shown below:

in This is the position of the chroma component i being refined. It is based on The brightness position, This is the filter support area in the brightness component. This indicates the filter coefficient.

As shown in Figure 4B, the luminance filter support region is the region co-located with the current chrominance sample, taking into account the spatial scaling factor between the luminance and chrominance planes. In Figure 4B, circles represent luminance samples, while dot-filled circles represent chrominance samples being refined.

In the VVC reference software, CC-ALF filter coefficients are calculated by minimizing the mean square error of each chroma channel relative to the original chroma content. To achieve this, the VVC Test Model (VTM) algorithm uses a coefficient derivation process similar to that of chroma ALF. Specifically, a correlation matrix is derived, and the coefficients are calculated using a Cholesky decomposition solver, attempting to minimize the mean square error metric. Up to eight CC-ALF filters can be designed and transmitted per image during filter design. The resulting filters are then indicated for two chroma channels based on CTU.

Additional features of CC-ALF include:

The design uses a 3x4 diamond shape with 8 points.

Seven filter factors are transmitted in APS.

Each transmission coefficient has a 6-bit dynamic range and is limited to a power value of 2.

The eighth filter coefficient is derived in the decoder, making the sum of the filter coefficients equal to 0.

APS can be referenced in the slice header.

The CC-ALF filter selection is controlled for each chromaticity component at the CTU level.

The boundary padding for horizontal virtual boundaries uses the same memory access mode as the luminance ALF.

As an additional feature, the reference encoder can be configured to enable some basic subjective adjustments via a configuration file. When enabled, the VTM reduces the application of CC-ALF in high-QP encoded areas, which are either close to mid-gray or contain a large number of high-luminance frequencies. Algorithmically, this is achieved by disabling the application of CC-ALF in CTUs that meet any of the following conditions:

The slice QP value minus 1 is less than or equal to the base QP value.

The number of chromaticity samples with a local contrast greater than (1 << (bitDepth – 2)) – 1 exceeds the CTU height, where local contrast is the difference between the maximum and minimum luminance sample values within the filter's supported area.

More than a quarter of the chroma samples fall within the range of (1 << (bitDepth – 1)) – 16 and (1 << (bitDepth – 1)) + 16.

The motivation behind this feature is to provide assurance that CC-ALF will not amplify earlier-introduced artifacts in the decoding path (primarily because VTM does not currently explicitly optimize chroma subjective quality). It is anticipated that alternative encoder implementations may not use this feature or will incorporate alternative strategies suited to their encoding characteristics.

6. Filter parameter signaling

ALF filter parameters are signaled in the Adaptation Parameter Set (APS). Within an APS, up to 25 sets of luma filter coefficients and cutoff value indices, and up to eight sets of chroma filter coefficients and cutoff value indices can be signaled. To reduce bit overhead, filter coefficients for different categories of luma components can be merged. In the slice header, the signaling is used to index the APS for the current slice.

The clipping value index from APS decoding allows the use of a clipping value table to determine clipping values for the luma and chroma components. These clipping values depend on the depth of the internal components. More precisely, the clipping values are obtained using the following formula:

AlfClip

Where B equals the internal depth, (α is a predefined constant value equal to 2.35, N equals 4, which is the number of clipping values allowed in VVC. Then AlfClip is rounded to the nearest millisecond value of 2.

In the slice header, up to 7 APS indices can be signaled to specify the luma filter set used for the current slice. The filtering process can be further controlled at the CTB level. Always signal a flag to indicate whether ALF is applied to the luma CTB. The luma CTB can select a filter set between 16 fixed filter sets and filter sets in the APS. Signal a filter set index for the luma CTB to indicate which filter set is applied. The 16 fixed filter sets are predefined and hard-coded in the encoder and decoder.

For chroma components, an APS index is signaled in the slice header to indicate the set of chroma filters used for the current slice. At the CTB level, if there are multiple chroma filter sets in the APS, a filter index is signaled for each chroma CTB.

The filter coefficients are quantized to a standard of 128. To limit multiplication complexity, bitstream consistency is applied so that coefficient values at non-center locations are in the range of -2^ ⁷ to 2 ^{^7} - 1, inclus. The coefficients at the center location are not signaled in the bitstream and are treated as equal to 128.

Adaptive loop filters in ECM

In ECM8 (Muhammed Coban et al., “Algorithmic description of Augmented Compression Model 8 (ECM 8)”, Joint Video Expert Group (JVET) of the International Telecommunication Union ITU-T SG 16 WP 3 and the International Organization for Standardization ISO/IEC JTC 1/SC 29, 29th Meeting, by teleconference, 11-20 January 2023, document: JVET-AC2025), some changes from VVC ALF were disclosed. A brief overview follows.

1. Simplified removal of ALF

ALF gradient subsampling and ALF virtual boundary processing have been removed. The classification block size has been reduced from 4x4 to 2x2. The signaling luminance and chrominance filter sizes have been increased to 9x9.

2. Fixed Filter ALF

To filter the brightness samples, three different classifiers ( _C0 , _C1 , and _C2 ) and three different filter sets ( _F0 , _F1 , and _F2 ) were used. Sets _F0 and _F1 contain fixed filters whose coefficients are trained with respect to classifiers _C0 and _C1 . The coefficients of the filters in _F2 are signaled. For a given sample, the class is assigned using classifier Ci _{[equationId74].} Determine which filter to use from set F _i .

3. Filtering

First, two 13x13 diamond-shaped fixed filters, _F0 and _F1, are applied to derive two intermediate samples. and Then, _F2 will be applied to... , The filtered sample is derived from neighboring samples, as shown below:

in It is the neighboring sample and the current sample The difference in cutting between them, yes The cropping difference between the current sample and the current sample. Filter coefficient. It's a signaling signal.

4. Classification

Based on directionality and activity Assign a category to each 2x2 block. :

,

in Indicates directionality The sum of the gradients is calculated using a 1-D Laplacian operator, similar to VVC. The sum of gradients for each sample within a 4×4 window covered by the target 2x2 block is used for classifier _C0 , while the sum of gradients for samples within a 12×12 window is used for classifiers _C1 and _C2 . The sums of the horizontal, vertical, and diagonal gradients are expressed as follows: , , and Directionality It was determined by comparing the following:

Use a set of thresholds. Directionality It is derived from the thresholds 2 and 4.5 in VVC. For and First, calculate the horizontal/vertical edge strength. Diagonal edge strength Use threshold value .if Then the edge strength If it is 0; otherwise, Is to make The largest integer that holds true. If Then the edge strength If it is 0; otherwise, Is to make The largest integer that holds true. When When the horizontal/vertical edges are dominant, the derivation can be done using Table 2A. Otherwise, the diagonal edge dominates, and the derivation can be performed using Table 2B. .

Table 2A. and arrive mapping 0 1 2 3 4 5 6 0 0 0 0 0 0 0 0 1 1 2 0 0 0 0 0 2 3 4 5 0 0 0 0 3 6 7 8 9 0 0 0 4 10 11 12 13 14 0 0 5 15 16 17 18 19 20 0 6 twenty one twenty two twenty three twenty four 25 26 27

Table 2B. and arrive mapping 0 1 2 3 4 5 6 0 28 0 0 0 0 0 0 1 29 30 0 0 0 0 0 2 31 32 33 0 0 0 0 3 34 35 36 37 0 0 0 4 38 39 40 41 42 0 0 5 43 44 45 46 47 48 0 6 49 50 51 52 53 54 55

In order to obtain The sum of the vertical and horizontal gradients Mapping to 0 The range, of which For It is 4, for and It is 15.

In ALF_APS, a maximum of 4 sets of luminance filter groups can be trusted, with each group containing up to 25 filters.

Convolutional Cross-Component Model (CCCM)

The convolutional 7-tap filter consists of a 5-tap plus sign-shaped spatial component, a nonlinear term, and a bias term. The input to the filter's spatial 5-tap component includes the center (C) luminance sample co-located with the chromaticity sample to be predicted, as well as its above/north (N), below/south (S), left/west (W), and right/east (E) neighbors, as shown in Figure 5.

The nonlinear term (denoted as P) is represented as the square of the center brightness sample C, and is compressed into the range of sample values for the content:

P = ( C*C + midVal ) >> bitDepth.

In other words, for 10-bit content, the calculation method is as follows:

P = (C*C + 512) >> 10

The offset (denoted as B) represents the scalar offset between the input and output (similar to the offset in CCLM) and is set to the intermediate chroma value (512 for 10-bit content).

The filter output is calculated by convolving the filter coefficient _ci with the input value and then cropping it to the range of the effective chromaticity samples.

predChromaVal = c ₀ C + c ₁ N + c ₂ S + c ₃ E + c ₄ W + c ₅ P + c ₆ B

1. Calculation of filter coefficients

The filter coefficient _ci is calculated by minimizing the mean square error (MSE) between the predicted and reconstructed chromaticity samples within the reference area. Figure 6 illustrates the reference area, which consists of six rows of chromaticity samples above and to the left of the PU (prediction unit). The reference area extends one PU width to the right and one PU height downward, beyond the PU boundary. This area is adjusted to include only usable samples. The area expansion shown in blue is to support the "side samples" of the "plus sign spatial filter" and to fill in unusable areas.

MSE minimization is performed by calculating the autocorrelation matrix of the luminance input and the cross-correlation vector between the luminance input and the chrominance output. The autocorrelation matrix is decomposed using LDL, and the final filter coefficients are calculated by back substitution. This process largely follows the calculation of ALF filter coefficients in ECM, but LDL decomposition is chosen instead of Cholesky decomposition to avoid the use of square root operations.

The autocorrelation matrix is calculated using reconstructed values from luma and chromaticity samples. These samples are full-range (e.g., between 0 and 1023 for 10-bit content), resulting in relatively large values in the autocorrelation matrix. This necessitates high-bit depth operations during model parameter calculations. It is recommended to remove the fixed offsets of the luma and chromaticity samples from each PU in each model. This reduces the magnitude of the values used in model creation and allows for a reduction in the precision required for fixed-point operations. Therefore, 16-bit decimal precision is recommended instead of the 22-bit precision used in the original CCCM implementation.

Reference sample values outside the top left corner of the PU are used as offsets (offsetLuma, offsetCb, and offsetCr) to simplify operations. The sample values used in model creation and final prediction (i.e., the luminance and chromaticity in the reference region, and the luminance in the current PU) are subtracted from these fixed values by:

C' = C – offsetLuma

N' = N – offsetLuma

S' = S – offsetLuma

E' = E – offsetLuma

W' = W – offsetLuma

P' = nonLinear(C')

B = midValue = 1 ＜＜ (bitDepth - 1)

Chromaticity values are predicted using the following formulas, where offsetChroma is equal to offsetCr and offsetCb for the Cr and Cb components, respectively:

predChromaVal = c ₀ C' + c ₁ N' + c ₂ S' + c ₃ E' + c ₄ W' + c ₅ P' + c ₆ B + offsetChroma.

To avoid any additional sample-level operations, the luminance offset is removed during luminance reference sample interpolation. For example, this can be done by replacing the rounding used in luminance reference sample interpolation with an updated offset that includes rounding and offsetLuma. Chroma offsets can be removed by subtracting the chroma offset directly from the reference chroma sample. Alternatively, the effect of the chroma offset can be removed from the cross-component vectors to achieve the same result. To add the chroma offset back to the output of the convolution prediction operation, the chroma offset is added to the bias term of the convolution model.

The CCCM model parameter calculation process requires division operations. Division operations are not always considered implementation-friendly. Division operations are replaced by multiplication (with scaling factors) and shift operations, where the scaling factor and the number of shifts are calculated based on the denominator, similar to the method used in CCLM parameter calculations.

Suggested methods

Filter taps associated with cross-component models and fixed filters

In one embodiment, to perform a chromatic ALF filtering process with signaling coefficients using the results of a fixed filter and chromaticity samples generated by applying a cross-component model to the luminance samples, the following steps can be applied (i.e., applying a cross-component model to the results of the fixed filter):

- Classification using a fixed filter classifier

- Filtering is performed using a fixed set of filter factors, where the fixed set of filter factors does not signal in the bit stream.

- Derive the luminance-to-chrominance cross-component model coefficients using the luminance fixed filter results and chrominance fixed filter results from the nearest reference region and/or the current block.

- By applying a cross-component model from luminance to chrominance to the luminance-fixed filter result of the current block, the chrominance sample of the current block is derived.

- Use signaling coefficients to perform chroma filtering by combining chroma samples and chroma samples from cross-component models.

In the above embodiments, the order of steps can be changed (e.g., applying fixed filtering to samples from cross-component models):

- Export luminance-to-chrominance cross-component model coefficients using reconstructed luminance and chrominance samples from neighboring reference regions and/or the current block.

- Export the chromaticity sample of the current block by applying a cross-component model from luminance to chrominance to the luminance sample of the current block.

- Classify chromaticity samples from across component models using a fixed filter classifier.

- Chromaticity samples from cross-component models are filtered using a fixed set of filter coefficients, where the fixed set of filter coefficients is not signaled in the bit stream.

- Use signaling coefficients to perform chroma filtering by combining chroma samples with fixed filtered chroma samples from cross-component models.

In the above embodiments, the step "classification using a fixed filter classifier" can be one of the following or a combination thereof:

- Classification of chromaticity samples using a fixed filter classifier

- Classification of luminance samples based on a fixed filter classifier

The fixed filter classifier is the same as the brightness fixed filter classifier.

Fixed filter classifiers are a subset of brightness fixed filter classifiers. A "subset of brightness fixed filter classifiers" means that, compared to brightness fixed filter classifiers, they have fewer categories, less directionality, and/or less mobility.

Fixed filter classifiers are different from brightness fixed filter classifiers.

In the above embodiment, in the step of "using a fixed filter coefficient set for filtering, wherein the fixed filter coefficient set is not signaled in the bit stream", the fixed filter coefficient set can be the following or a combination of the following:

The set of fixed filter coefficients is the same as the set of fixed brightness filter coefficients.

The set of fixed filter coefficients is a subset of the set of fixed brightness filter coefficients.

The set of fixed filter coefficients is different from the set of fixed brightness filter coefficients.

When using a fixed set of filter coefficients to filter chromaticity samples, the filter selection is determined by one or a combination of the following:

Classification results of chromaticity fixed filter.

The corresponding brightness fixed filter classification results.

In the above embodiments, a chroma-to-luminance cross-component model can also be applied. This means that by using luminance samples and applying the cross-component model to chroma samples for the luminance adaptive loop filter (ALF) or cross-component ALF (CCALF) filtering process, and using signaling coefficients, the following steps can be performed:

Classification is performed using a fixed filter classifier.

Filtering is performed using a fixed set of filter coefficients, where the fixed set of filter coefficients is not signaled in the bit stream.

The coefficients of the cross-component model from chromaticity to luminance are derived by using the results of the chromaticity fixed filter and the luminance fixed filter for the nearest reference region and/or the current block.

The luminance sample of the current block is derived by applying the results of the chromaticity fixed filter to the cross-component model of the chromaticity to luminance of the current block.

Using signaling coefficients, a luminance ALF or CCALF filtering process is performed using luminance samples and luminance samples from cross-component models.

In the above embodiments, the reconstructed luminance and chromaticity samples of the nearest reference region used to derive the cross-component model coefficients can be samples before or after the application of the ALF filtering process.

In the above embodiments, the reconstructed luminance or chromaticity sample of the current block used to derive the cross-component model sample can be a sample before or after the application of the ALF filtering process.

In the above embodiments, the step of "deriving the coefficients of the cross-component model from luminance to chrominance using the results of the luminance fixed filter and the chrominance fixed filter of the nearest reference region and/or the current block" can be the following or a combination of the following:

The coefficients of the cross-component model of luminance to chrominance are derived using the results of the luminance fixed filter of the nearest reference region and/or the current block and the reconstructed chrominance sample.

The coefficients of the cross-component model for luminance to chrominance are derived using reconstructed luminance samples and chrominance fixed filter results from neighboring reference regions and/or the current block.

In the above embodiment, the step of "deriving the chromaticity sample of the current block by applying the results of the luminance-fixed filter to the cross-component model of luminance to chromaticity of the current block" can be the following or a combination of the following:

The chromaticity sample of the current block is derived by applying a cross-component model from luminance to chrominance to the reconstructed luminance sample of the current block.

ALF Additional Mode

In ECM ALF, the syntactic design of slices and CTB levels is shown below. Several new slice and CTB level syntactic designs are illustrated in the following sections:

Brightness ALF:

Slice level: On/Off, number of APS or APS IDs (up to 7).

CTB Levels: Off, Fixed Filter Set (choose 1 of 2), or APS ID and APS Filter Set.

Chromaticity ALF:

Slicing level: On/Off (Cb and Cr are separate), 1 APS ID (shared by Cb and Cr).

CTB level: Off, APS filter set (maximum 1) (Cb and Cr separated).

CCALF:

Slice level: On/Off (Cb and Cr separated), 2 APS IDs (Cb and Cr separated).

CTB level: Off, APS filter set (maximum 1) (Cb and Cr separated).

In one embodiment, a fixed filter set can be selected for chromatic ALF and/or CCALF at the CTB level.

In one embodiment, multiple APSs can be selected at the image/tile/slice level, and one APS can be selected from these APSs at the CTB level for chroma ALF and/or CCALF.

In one embodiment, samples from a cross-component model, and/or samples from a cross-component model with fixed filters, can be selected at the CTB level for luminance ALF, chromaticity ALF, and/or CCALF.

In one embodiment, a flag can be signaled at the image/tile/slice level for the luma ALF, chroma ALF, and/or CCALF to control whether a fixed filter set can be selected at the CTB level. Therefore, if the flag indicates that a fixed filter set can be selected at the CTB level, then both the fixed filter set and the APS filter set can be selected at the CTB level. Otherwise, only the APS filter set can be selected at the CTB level.

In one embodiment, for the luma ALF, chroma ALF, and/or CCALF, a flag can be signaled at the image/tile/slice level to indicate that the fixed filter set can only be selected at the CTB level. Therefore, if the flag indicates that the fixed filter set can only be selected at the CTB level, then the fixed filter set can only be selected at the CTB level. Otherwise, both the fixed filter set and the APS filter set can be selected at the CTB level.

In one embodiment, for the luma ALF, chroma ALF, and/or CCALF, a flag can be signaled at the image/tile/slice level to control whether the APS ID and APS filter set can be selected at the CTB level. Therefore, if the flag indicates that the APS ID and APS filter set can be selected at the CTB level, then both the fixed filter set and the APS filter set can be selected at the CTB level. Otherwise, only the fixed filter set can be selected at the CTB level.

In one embodiment, for a luma adaptive loop filter (luma ALF), a chroma adaptive loop filter (chroma ALF), and/or a cross-component adaptive loop filter (Cross-Component ALF, or CCALF), a flag can be signaled at the image level, tile level, or slice level to ensure that the Adaptation Parameter Set (APS) ID and APS filter set can only be selected at the Coding Tree Block (CTB) level. Therefore, if the flag indicates that the APS ID and APS filter set can only be selected at the CTB level, then the APS ID and APS filter set can only be selected at the CTB level. Otherwise, both the fixed filter set and the APS filter set can be selected at the CTB level.

In one embodiment, for the luma ALF, chroma ALF, and/or CCALF, a set of APS filters can be used at the image level, tile level, or slice level for signaling from different APSs, so that these APS filter sets can only be selected at the CTB level.

Example: There are two APSs, APS0 and APS1. For APS0, there are three filter sets. For APS1, there are two filter sets, APS0 = {0, 1, 2} and APS1 = {0, 1}. The following shows an example of two slice selections:

Example 1: Slice selection selects only filter sets 0 and 2 in APS0, and filter set 0 in APS1. Then, at the CTB level, only these filter sets can be used (i.e., APS0’ = {0, 2}, APS1’ = {0}).

Example 2: Slice selection selects only filter sets 0 and 2 in APS0, and filter set 0 in APS1. A list containing all filter sets is formed. Then, at the CTB level, only these filter sets can be used (i.e., APS’ = {APS0-0, APS1-0, APS0-2}).

In one embodiment, the cross-component model described in the preceding sections can be selected at the CTB level for chroma ALF and/or CCALF. That is, samples generated from the cross-component model can be directly used as the output of ALF.

The proposed method can be implemented in an encoder and/or decoder. For example, the proposed method can be implemented in the loop filter module of an encoder, and/or in the loop filter module of a decoder.

Any of the methods described above can be implemented in an encoder and/or decoder. Furthermore, any method of the aforementioned unified classification process can also be implemented in an encoder and/or decoder. For example, any proposed method can be implemented in the loop filter module of the encoder or decoder (e.g., ILPF 130 in Figures 1A and 1B). Alternatively, any proposed method can be implemented as a circuit coupled to an internal module of the encoder/decoder, and/or a motion compensation module, or a merge candidate derivation module of the decoder. The simplified ALF method can also be implemented using executable software or firmware code stored on media (such as hard drives or flash memory) for use by a central processing unit (CPU) or a programmable device (such as a digital signal processor (DSP) or a field programmable gate array (FPGA)).

Figure 7 illustrates a flowchart of an exemplary video encoding/decoding system that selects a fixed set of filters for chroma ALF and/or CCALF according to instructions at the CTB level, disclosed according to an embodiment of the invention. The steps shown in the flowchart can be executed as code on the encoder side of one or more processors (e.g., one or more CPUs). The steps shown in the flowchart can also be hardware-based, such as one or more electronic devices or processors arranged to execute the steps in the flowchart. According to the method, a reconstructed pixel is received in step 710, wherein the reconstructed pixel includes a current block, and the current block includes a first color block and a second color block. In step 720, a target fixed filter set is selected from multiple fixed filter sets according to the filter selection instruction at the CTB (Code Block) level. This target fixed filter set is used to filter the current block using a target filter type from a filter type group, which includes Chromatic ALF (Adaptive Loop Filter), CCALF (Cross-Component ALF), or both. In step 730, the target fixed filter set is applied to the current block to generate the filtered current block. In step 740, the filtered current block is provided.

The flowchart shown is intended to illustrate an example of video encoding/decoding according to the present invention. Those skilled in the art can implement the present invention by modifying each step, rearranging steps, splitting steps, or merging steps without departing from the spirit of the present invention. Specific syntax and semantics are used in this disclosure to illustrate examples of implementing embodiments of the present invention. Those skilled in the art can implement the present invention by using equivalent syntax and semantic substitutions without departing from the spirit of the present invention. The above description is intended to enable those skilled in the art to practice the present invention according to specific applications and their requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, this invention is not intended to be limited to the specific embodiments shown and described, but should be given its broadest scope in accordance with the principles and novel features disclosed herein. Various specific details have been shown in the foregoing detailed description to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced without departing from its spirit.

Embodiments of the present invention, as described above, can be implemented in various hardware, software code, or a combination of both. For example, embodiments of the present invention may be one or more circuits integrated into a video compression chip, or program code integrated into video compression software, to perform the processing described herein. Embodiments of the present invention may also be program code executed on a digital signal processor (DSP) to perform the processing described herein. The present invention may also relate to multiple functions executed by a computer processor, digital signal processor, microprocessor, or field programmable gate array (FPGA). These processors can be configured by executing machine-readable software or firmware code to define the specific methods embodied in this invention. This software or firmware code can be developed using different programming languages and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles, and languages used to perform the tasks of this invention, as well as other ways of configuring the code, do not depart from the spirit and scope of this invention.

This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The examples described are, in all respects, illustrative only and not limiting. Therefore, the scope of this invention is indicated by the appended claims, and not by the foregoing description. All changes within the meaning and equivalent scope of the claims are included within their scope.

110: Intra-frame prediction 112: Inter-frame prediction 114: Switch 116: Adder 118: T 120: Q 122: Entropy Encoder 124: IQ 126: IT 128: REC 130: Loop filter 134: Ref. Pic. Buffer 136: Prediction data 140: Entropy Decoder 150: Intra-frame prediction 152: MC 210-220: Diamond filter shape 410, 412, 414: Corresponding SAO 420: ALF Luma 422, 424: Cross-component item for luminance to chrominance components 430: ALF Chroma 432, 434: Adder 440, 442: Filter 710-740: Steps

Figure 1A illustrates an exemplary adaptive Inter/Intra video encoding/decoding system incorporating loop processing. Figure 1B shows the corresponding decoder for the encoder in Figure 1A. Figure 2 shows the ALF filter shapes for the chroma (left) and luma (right) components. Figures 3A-3D show the subsampling Laplacian calculations for _gv (3A), _gh (3B), _gd1 (3C), and _gd2 (3D). Figure 4A shows the position of the CC-ALF relative to the other loop filters. Figure 4B shows a diamond-shaped filter used for chroma samples. Figure 5 shows an example of the spatial portion of a convolution filter. Figure 6 shows a reference area used to derive filter coefficients, with padding employed. Figure 7 illustrates a flowchart of an exemplary video encoding/decoding system that selects a fixed set of filters for chroma ALF and/or CCALF based on instructions at the CTB level, disclosed according to an embodiment of the present invention.

710-740: Steps

Claims (12)

A method for processing color images or videos, the method comprising: receiving reconstructed pixels, wherein the reconstructed pixels include a current block, and the current block includes a first color block and a second color block; selecting a target set of fixed filters from a plurality of fixed filter sets according to a filter selection instruction at a Coding Tree Block (CTB) level, for filtering the current block using a target filter type from a filter type group, the filter type group including an Adaptive Loop Filter (ALF) and a Cross-Component ALF. (ALF, CCALF) or both; apply the target fixed filter set to the current block to generate a filtered current block; and provide the filtered current block. The method as described in claim 1, wherein at the CTB level, a sample generated from a cross-component model and/or a sample generated from a cross-component model with fixed filters are selected for the luminance ALF, the chrominance ALF, the CCALF, or a combination thereof. The method of claim 1, wherein multiple APSs are selected at the image level, tile level, slice level, or a combination thereof, and wherein a target APS is selected from the multiple APSs at the CTB level for the step of selecting the target fixed filter set. The method of claim 1, wherein a flag is signaled or parsed at the image level, tile level, slice level or a combination thereof for controlling the use of a fixed filter at the CTB level, and the use of the fixed filter is for the filter type group, which further includes a luminance ALF. The method described in claim 4, wherein the flag is used to control whether the selection of a fixed filter set is enabled at the CTB level. The method described in claim 4, wherein the flag is used to indicate whether only one or more fixed filter sets are allowed to be selected at the CTB level. The method described in claim 4, wherein the flag is used to enable the selection of the Adaptation Parameter Set (APS) ID and the APS filter set at the CTB level. The method described in claim 4, wherein the flag is used to indicate whether only one or more APS (Adaptive Parameter Set) IDs and APS filter sets are allowed to be selected at the CTB level. The method of claim 1, wherein a set of adaptation parameter set (APS) filters is signaled or parsed at the image level, tile level, slice level, or a combination thereof, and the APS filter set comes from different APSs, and the APS filter set is only allowed to be selected at the CTB level. The method described in claim 1, wherein the cross-component model is selected as the target filter type. The method described in claim 1, wherein the sample generated by the cross-component model is directly used as the current block after filtering. An apparatus for color images or videos, the apparatus comprising one or more electronic devices or processors configured to: receive reconstructed pixels, wherein the reconstructed pixels include a current block, and the current block includes a first color block and a second color block; select a target set of fixed filters from a plurality of fixed filter sets according to filter selection instructions at a Coding Tree Block (CTB) level, for filtering the current block using a target filter type from a filter type group, the filter type group including an Adaptive Loop Filter (ALF), a Cross-Component ALF, and a Cross-Component ALF. (ALF, CCALF) or both; apply the target fixed filter set to the current block to generate a filtered current block; and provide the filtered current block.