[go: up one dir, main page]

WO2023011005A1 - 图像处理方法及装置、电子设备和存储介质 - Google Patents

图像处理方法及装置、电子设备和存储介质 Download PDF

Info

Publication number
WO2023011005A1
WO2023011005A1 PCT/CN2022/098600 CN2022098600W WO2023011005A1 WO 2023011005 A1 WO2023011005 A1 WO 2023011005A1 CN 2022098600 W CN2022098600 W CN 2022098600W WO 2023011005 A1 WO2023011005 A1 WO 2023011005A1
Authority
WO
WIPO (PCT)
Prior art keywords
matrix
block
zero
row
frequency coefficient
Prior art date
Application number
PCT/CN2022/098600
Other languages
English (en)
French (fr)
Inventor
李晓波
叶天晓
Original Assignee
上海哔哩哔哩科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海哔哩哔哩科技有限公司 filed Critical 上海哔哩哔哩科技有限公司
Priority to US18/294,053 priority Critical patent/US20250088666A1/en
Publication of WO2023011005A1 publication Critical patent/WO2023011005A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/007—Transform coding, e.g. discrete cosine transform
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/625—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using discrete cosine transform [DCT]

Definitions

  • the present disclosure relates to the technical field of image and video processing, and in particular to an image processing method, an image processing device, electronic equipment, a computer-readable storage medium, and a computer program product.
  • Image and video data have strong spatial correlation, and there is a large amount of redundant information in the spatial domain.
  • Transform coding technology can transform image or video data from the spatial domain to the frequency domain, greatly reducing redundant information, so as to realize the compression of image or video data, and facilitate the storage and transmission of image or video data.
  • each video frame can be predicted, transformed (Transform), quantized, inverse quantized, inverse transformed (Inverse Transform), reconstructed, entropy encoded, etc. at the encoding end to obtain the encoding of each video frame; Then entropy decoding, inverse quantization, inverse transformation and other processing are performed on the encoded data at the decoding end to restore each video frame.
  • Both the video encoding process and the video decoding process include inverse transform processing. Inverse transformation processing requires a large amount of calculation, low calculation efficiency, and requires a large amount of memory space, which greatly limits the efficiency of video encoding and decoding.
  • the present disclosure provides an image processing method, an image processing device, electronic equipment, a computer-readable storage medium, and a computer program product.
  • an image processing method including: acquiring a frequency coefficient matrix of a target image, the frequency coefficient matrix is obtained by performing frequency domain transformation on the target image, and the frequency coefficient
  • the matrix includes at least one non-zero frequency coefficient; based on the position of the at least one non-zero frequency coefficient, determine a non-zero block in the frequency coefficient matrix; determine a transformation sub-matrix and a transposition sub-matrix corresponding to the non-zero block; and determining an inverse transform result of the frequency coefficient matrix based on the transform sub-matrix, the non-zero blocks, and the transposed sub-matrix.
  • an image processing device including: an acquisition unit configured to acquire a frequency coefficient matrix of a target image, the frequency coefficient matrix is obtained by performing frequency domain transformation on the target image , and the frequency coefficient matrix includes at least one non-zero frequency coefficient; the non-zero block determination unit is configured to determine a non-zero block in the frequency coefficient matrix based on the position of the at least one non-zero frequency coefficient; matrix a determination unit configured to determine a transform sub-matrix and a transpose sub-matrix corresponding to the non-zero block; and an inverse transform unit configured to determine based on the transform sub-matrix, the non-zero block, and the transpose sub-matrix , to determine the inverse transformation result of the frequency coefficient matrix.
  • an electronic device including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, and the computer program is executed by The at least one processor implements the method according to the above when executed.
  • a non-transitory computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.
  • a computer program product including a computer program, wherein the computer program implements the method according to the above when executed by a processor.
  • the non-zero blocks in the frequency coefficient matrix of the target image by determining the non-zero blocks in the frequency coefficient matrix of the target image, only the non-zero blocks are calculated to obtain the inverse transformation result of the frequency coefficient matrix, without the need for other Frequency coefficients (that is, frequency coefficients other than zero blocks) are calculated, which greatly reduces the calculation amount of inverse transformation and improves the calculation efficiency of inverse transformation, thereby improving the efficiency of image and video encoding and decoding.
  • FIG. 1A shows a schematic diagram of an image encoding and decoding process according to an embodiment of the present disclosure
  • FIG. 1B shows a schematic structural diagram of a video encoding end according to an embodiment of the present disclosure
  • FIG. 1C shows a schematic structural diagram of a video decoding end according to an embodiment of the present disclosure
  • FIG. 2 shows a flowchart of an image processing method according to an embodiment of the present disclosure
  • Fig. 3 shows a flow chart of calculating every adjacent multi-row element of an intermediate matrix according to an embodiment of the present disclosure
  • FIG. 4 shows a schematic diagram of a calculation process of elements in the first two rows of an intermediate matrix according to an embodiment of the present disclosure
  • FIG. 5 shows a structural block diagram of an image processing device according to an embodiment of the present disclosure.
  • FIG. 6 shows a structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure.
  • first, second, etc. to describe various elements is not intended to limit the positional relationship, temporal relationship or importance relationship of these elements, and such terms are only used for Distinguishes one element from another.
  • first element and the second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on contextual description.
  • the encoding process of an image is roughly as follows: the image is divided into multiple image blocks, and the image block is used as a transformation unit (Transform Unit, TU) for transformation and encoding.
  • the image block can be, for example, a square image with a size of 4*4, 8*8, 16*16, or 32*32, or a rectangular image.
  • each image block is transformed separately, and the image block data in the spatial domain is converted to the frequency domain to eliminate redundant information in the image block and obtain the corresponding Matrix of transform coefficients in the frequency domain.
  • the method used for the transformation may be, for example, discrete cosine transform (Discrete Cosine Transform, DCT), discrete sine transform (Discrete Sine Transform, DST), etc. Quantize the transformation coefficient matrix obtained through transformation, and perform variable-length coding (for example, Huffman coding, etc.) on the quantized result to obtain a coding result.
  • DCT discrete Cosine Transform
  • DST discrete sine transform
  • variable-length coding for example, Huffman coding, etc.
  • the decoding process of an image is roughly as follows: as shown in FIG. 1A , variable-length decoding, inverse quantization, and inverse transformation are performed on the encoding result of each image block to obtain a decoded image block.
  • the inverse transformation is the inverse process of the transformation, and the method used in the inverse transformation can be, for example, the inverse discrete cosine transform (Inverse Discrete Cosine Transform, IDCT), the inverse discrete sine transform (Inverse Discrete Sine Transform, IDST), etc.
  • the video encoding process is roughly as follows: the original video frame is divided into multiple image blocks, and the image blocks are used as transform units (TUs) for transform encoding.
  • the image block can be, for example, a square image with a size of 4*4, 8*8, 16*16, or 32*32, or a rectangular image.
  • a YUV video frame is composed of a luma channel (Y), a first chrominance channel (U) and a second chrominance channel (V).
  • a video frame may include multiple image blocks of different channels, for example, a YUV video frame may include a luma image block, a first chrominance image block, and a second chrominance image block.
  • the encoding end may include an encoding intra-frame prediction module, an encoding inter-frame prediction module, a transformation module, a quantization module, an entropy encoding module, an inverse quantization module, an inverse transformation module, a reconstruction module, a filtering module, and a reference image buffering module , which are respectively used to perform prediction, transformation, quantization, entropy coding, inverse quantization, inverse transformation, reconstruction, filtering and other processing on the original video frame, and finally output the encoded stream of the video.
  • the original video frame is used as one input of the adder
  • the prediction data output by the encoded intra prediction module or the encoded inter prediction module is another input of the adder
  • the adder outputs the prediction residual difference matrix.
  • the prediction residual matrix is transformed and quantized to obtain the residual coefficient matrix.
  • the residual coefficient matrix is processed by inverse quantization and inverse transformation to restore the prediction residual matrix.
  • the recovered prediction residual matrix can be used to reconstruct video frames.
  • the method used for the transformation process may be DCT, DST, etc., for example.
  • the inverse transform is the inverse process of the transform, and correspondingly, the method adopted for the inverse transform processing may be, for example, IDCT, IDST, and the like.
  • the video decoding process is roughly as follows: as shown in FIG. 1C , first entropy decoding is performed on the received coded stream to obtain the residual coefficient matrix, inter-frame prediction related information, intra-frame prediction related information, etc. After the residual coefficient matrix is dequantized and inversely transformed, the prediction residual matrix is recovered. The reconstruction module adds back the prediction residual to the corresponding intra-predicted or inter-predicted data. A switch can be used to select to use intra-frame prediction data or inter-frame prediction data to obtain the reconstructed frame.
  • the decoding end may include a decoding intra prediction module, a decoding inter prediction module, an entropy decoding module, an inverse quantization module, an inverse transformation module, a reconstruction module, a filtering module, a reference image buffer module, and a video playback buffer module.
  • the inverse transform process can be applied to processes such as image decoding (for inverse transformation of the transform coefficient matrix), video encoding and decoding (for inverse transformation of the residual coefficient matrix).
  • the inverse transformation process is computationally intensive.
  • the size of TU is 8*8 (that is, the size of the residual coefficient matrix is 8*8)
  • the inverse transformation method is IDCT as an example
  • the inverse transformation of the residual coefficient matrix X is performed as follows The calculation shown in the formula:
  • Y is the result matrix obtained from the inverse transformation process
  • H 8 is the 8*8 DCT transformation matrix
  • X is the residual coefficient matrix to be inversely transformed
  • H 8 T is the transposition matrix of H 8 .
  • the elements in the DCT transformation matrix H 8 are calculated through the cosine function, and its value is usually a floating point number.
  • the transformation matrix H 8 can be expressed as the product of a common factor of a floating point number and an integer matrix, and correspondingly, the transposition matrix H 8 T of the transformation matrix H 8 can also be expressed as the same common factor and The product of an integer matrix H T .
  • the common factor of the 8*8 transformation matrix H 8 and its transpose matrix H 8 T can be 0.0055
  • the corresponding integer matrix H corresponding to the transformation matrix H 8 is as follows:
  • the integer matrix H T corresponding to the transposed matrix H 8 T is as follows:
  • the transformation matrix is referred to by an integer matrix corresponding to the transformation matrix
  • the transposition matrix is referred to by an integer matrix corresponding to the transposition matrix of the transformation matrix.
  • the residual coefficient matrix X is as follows:
  • a butterfly operation may be used to accelerate the calculation process of the inverse transformation.
  • the first column of matrix B can be calculated by common terms:
  • the frequency coefficient matrix X has only 4 non-zero frequency coefficients located in the upper left corner area, and the values of other frequency coefficients are all zero. If the frequency coefficient matrix X is inversely transformed according to the calculation method of directly performing matrix multiplication or butterfly operation, many calculations will be invalid, because the result of the addition, subtraction, multiplication and division of zero is still zero, resulting in a large amount of storage and waste of computing resources.
  • the present disclosure provides an improved image processing method.
  • the method obtains the frequency coefficient matrix of the target image, determines the non-zero blocks in the frequency coefficient matrix based on the position of at least one non-zero frequency coefficient included in the frequency coefficient matrix, and converts the transformation matrix and the transformation matrix respectively based on the non-zero blocks. Then, the inverse transformation result of the frequency coefficient matrix is determined based on the transformation submatrix, the non-zero block and the transposition submatrix.
  • the image processing method of the embodiment of the present disclosure determines the non-zero blocks in the frequency coefficient matrix of the target image, and only calculates the non-zero blocks to obtain the inverse transformation result of the frequency coefficient matrix, without requiring other frequency coefficients in the frequency coefficient matrix ( That is, the frequency coefficients other than zero blocks) are calculated, which greatly reduces the calculation amount of inverse transformation and improves the calculation efficiency of inverse transformation, thereby improving the efficiency of image and video encoding and decoding.
  • FIG. 2 shows a flowchart of an image processing method 200 according to an embodiment of the present disclosure.
  • the method 200 can be applied to image decoding or video coding and decoding scenarios, and can improve the calculation efficiency of inverse transformation, thereby improving the efficiency of image decoding and video coding and decoding.
  • method 200 may include:
  • Step S210 acquiring the frequency coefficient matrix of the target image, the frequency coefficient matrix is obtained by performing frequency domain transformation on the target image, and the frequency coefficient matrix includes at least one non-zero frequency coefficient;
  • Step S220 based on the position of at least one non-zero frequency coefficient, determine a non-zero block in the frequency coefficient matrix
  • Step S230 determining the transformation sub-matrix and transpose sub-matrix corresponding to the non-zero block.
  • Step S240 Determine the inverse transform result of the frequency coefficient matrix based on the transformed sub-matrix, the non-zero block and the transposed sub-matrix.
  • the non-zero blocks in the frequency coefficient matrix of the target image by determining the non-zero blocks in the frequency coefficient matrix of the target image, only the non-zero blocks are calculated to obtain the inverse transformation result of the frequency coefficient matrix, and there is no need to perform calculations on other frequency coefficients in the frequency coefficient matrix (i.e. Frequency coefficients other than zero blocks) are calculated, which greatly reduces the calculation amount of inverse transformation and improves the calculation efficiency of inverse transformation, thereby improving the efficiency of image and video encoding and decoding.
  • the target image may be, but not limited to, a single image or an image block (ie, TU) in a video frame.
  • the target image may be an image block in an image to be decoded, or an image block in a video frame to be encoded or an image block in a video frame to be decoded.
  • the target image may be, for example, an image block of a luminance channel of a video frame, or an image block of another channel.
  • the target image may also be an image block of the first chroma channel or an image block of the second chroma channel.
  • the frequency coefficient matrix of the target image is obtained by transforming the target image in frequency domain.
  • the frequency coefficient matrix can be, for example, the transformation coefficient matrix obtained by performing frequency-domain transformation on the image block of the image, or the frequency-domain transformation, quantization, and inverse quantization of the prediction residual of the image block of the video frame.
  • the residual coefficient matrix of can be, for example, the transformation coefficient matrix obtained by performing frequency-domain transformation on the image block of the image, or the frequency-domain transformation, quantization, and inverse quantization of the prediction residual of the image block of the video frame.
  • the residual coefficient matrix of Methods for performing frequency domain transformation on the target image include but not limited to DCT, DST, and the like.
  • the target image can be a square image with the same width and height as 4*4, 8*8, 16*16, 32*32, etc.
  • the frequency coefficient matrix obtained by performing frequency domain transformation on the target image can be 4*4 , 8*8, 16*16, 32*32 and other matrices (that is, square matrices) with the same number of rows and columns.
  • the target image can also be 4*8, 8*4, 4*16, 16*4, 8*16, 16*8, 16*32, 32*16, 8*32, 32*8, etc.
  • the frequency coefficient matrix obtained by performing frequency domain transformation on the target image can be 4*8, 8*4, 4*16, 16*4, 8*16, 16*8, 16*32 , 32*16, 8*32, 32*8 and other matrices with unequal numbers of rows and columns.
  • step S220 may include: determining the maximum value of the row coordinate and the maximum value of the column coordinate corresponding to the at least one non-zero frequency coefficient; and determining the non-zero block according to the maximum value of the row coordinate and the maximum value of the column coordinate.
  • the non-zero frequency coefficients in the frequency coefficient matrix are usually located in the upper left corner of the matrix, it is possible to obtain Determine the location area where the non-zero frequency coefficients in the non-zero frequency coefficient matrix are located, so as to determine the non-zero blocks in the frequency coefficient matrix.
  • the row coordinate of the frequency coefficient may refer to the row where the frequency coefficient is located in the frequency coefficient matrix
  • the column coordinate of the frequency coefficient may refer to the column where the frequency coefficient is located in the frequency coefficient matrix
  • the non-zero block includes one or more frequency coefficients in the frequency coefficient matrix whose corresponding row coordinates are less than or equal to the maximum value of the row coordinates and whose corresponding column coordinates are less than or equal to the maximum value of the column coordinates. Since the non-zero frequency coefficients in the frequency coefficient matrix are usually located in the upper left corner of the matrix, the maximum value of the row coordinates and the maximum value of the column coordinates correspond to the non-zero frequency coefficients in the lower right corner of the frequency coefficient matrix.
  • the non-zero blocks include the frequency coefficients whose row coordinates in the frequency coefficient matrix are less than or equal to the maximum value of the row coordinates and whose column coordinates are less than or equal to the maximum value of the column coordinates, it can be guaranteed that the non-zero blocks include all non-zero frequency coefficients in the frequency coefficient matrix, such that Lossless (that is, no loss of information) inverse transformation can be performed on the frequency coefficient matrix in a subsequent step.
  • Lossless that is, no loss of information
  • a non-zero block may be composed of one or more frequency coefficients whose corresponding row coordinates are less than or equal to the maximum value of the row coordinates and whose corresponding column coordinates are less than or equal to the maximum value of the column coordinates in the frequency coefficient matrix. That is, a non-zero block is a block located at the upper left corner of the frequency coefficient matrix and whose size is the maximum value of the row coordinate*the maximum value of the column coordinate.
  • the non-zero block may be determined according to the following steps: determine the maximum row boundary value of the non-zero block based on the maximum value of the row coordinates; determine the maximum column boundary value of the non-zero block based on the maximum value of the column coordinates, wherein, The maximum row boundary value is the smallest power of 2 that is greater than or equal to the maximum value of the row coordinate, and the maximum column boundary value is the smallest power of 2 that is greater than or equal to the maximum value of the column coordinate.
  • a non-zero block consists of all frequency coefficients whose row coordinates are less than or equal to the maximum row boundary value and whose column coordinates are less than or equal to the maximum column boundary value in the frequency coefficient matrix.
  • Both the maximum row boundary value and the maximum column boundary value are powers of 2, by expanding the maximum row coordinate value to the maximum row boundary value, expanding the column coordinate maximum value to the maximum column boundary value, and according to the maximum row boundary value, maximum column Boundary value is used to determine the non-zero block, which can make the number of rows and columns of the determined non-zero block be a power of 2, so as to facilitate the optimization of the instruction set and improve the utilization efficiency of the memory space in the subsequent inverse transformation calculation process. Accelerate the calculation process.
  • the smallest power of 2 that is greater than or equal to the maximum value of row coordinates 3 is 4, that is, the maximum row boundary value is 4; the smallest power of 2 that is greater than or equal to the maximum value of column coordinates 4 is 4, that is, the maximum column boundary value is 4.
  • the non-zero block consists of all frequency coefficients whose row coordinates in the frequency coefficient matrix are less than or equal to the maximum row boundary 4 and whose column coordinates are less than or equal to the maximum column boundary value 4, that is, the non-zero block is 4 in the upper left corner of the frequency coefficient matrix X *4 areas. That is, the non-zero blocks C are as follows:
  • step S230 may include: determining a preset transformation matrix and a transpose matrix of the transformation matrix based on the frequency coefficient matrix; The transposed matrix is clipped to obtain the transposed submatrix.
  • the transformation matrix and its transpose matrix may be determined by the size of the frequency coefficient matrix. Specifically, the size of the transformation matrix may be determined according to the size of the frequency coefficient matrix.
  • the size of the transformation matrix H is the same as the size of the frequency coefficient matrix, that is
  • the transformation matrix is a square matrix with the number of rows and columns equal to the number of rows (or columns) of the frequency coefficient matrix.
  • the corresponding transformation matrix H and its transpose matrix H T are also an 8*8 square matrix.
  • the transformation matrix H is a square matrix whose number of rows and columns are equal to the number of columns of the frequency coefficient matrix
  • matrix H T is a square matrix whose number of rows and columns are equal to the number of rows of the frequency coefficient matrix.
  • the frequency coefficient matrix is an 8*4 matrix
  • the corresponding transformation matrix H is a 4*4 square matrix
  • H T is an 8*8 square matrix.
  • each element in the transformation matrix can be calculated according to a preset formula, so as to obtain the transformation matrix. Further, the transformation matrix is transposed, that is, the transposition matrix of the transformation matrix is obtained.
  • the size of the frequency coefficient matrix is 8*8, and correspondingly, the size of the transformation matrix is also 8*8.
  • IDCT the inverse transformation as an example, when the transformation matrix is a square matrix (that is, the number of rows and columns of the transformation matrix is the same), the element D i of the i-th row and j-th column in the transformation matrix can be calculated according to the following formula ,j :
  • N is the number of rows (or columns) of the transformation matrix.
  • the elements calculated according to the above formula are floating point numbers D i,j .
  • the elements D i,j in the transformation matrix can be scaled to convert them into integers, so as to facilitate computer processing.
  • the scaling factor of D i,j may be determined according to different application scenarios. For example, for different video coding standards, different scaling factors can be set. For the HEVC (High Efficiency Video Coding, high-efficiency video coding) video coding standard, the elements in the transformation matrix can be enlarged by 128 times; for the AV1 video coding standard, the elements in the transformation matrix can be enlarged by 8192 times.
  • HEVC High Efficiency Video Coding, high-efficiency video coding
  • the transformation matrix may be clipped to obtain a transformation sub-matrix based on the size of the non-zero blocks, and the transpose matrix of the transformation matrix may be clipped to obtain a transpose sub-matrix.
  • the size of the transformed sub-matrix and transposed sub-matrix obtained by clipping is adapted to the size of the non-zero block, and matrix multiplication operation can be performed with the non-zero block.
  • the transpose matrix H T is clipped to obtain the 8*4 transposed sub-matrix H T ':
  • the change sub-matrix and the transpose sub-matrix can also be directly determined based on the size of the non-zero block.
  • the specific method is similar to determining the change matrix and the transpose matrix based on the frequency coefficient matrix, and will not be described in detail here.
  • step S240 can be executed to determine the inverse transform result of the frequency coefficient matrix.
  • step S240 may be executed in response to at least one of the following conditions: the maximum row boundary value of the non-zero block is less than the total number of rows of the frequency coefficient matrix; the maximum column boundary value of the non-zero block is less than the total number of columns of the frequency coefficient matrix .
  • the size of the non-zero block determined according to the maximum row boundary value and the maximum column boundary value is smaller than the frequency
  • the size of the coefficient matrix so determining the inverse transformation result of the frequency coefficient matrix based on the non-zero blocks can reduce the calculation amount of the inverse transformation and improve the calculation efficiency.
  • step S240 can be performed to inversely transform the frequency coefficient matrix, or step S240 can not be performed, and the traditional method (such as directly performing matrix multiplication or butterfly operation on the frequency coefficient matrix) is directly used to convert the frequency coefficient matrix The matrix is inversely transformed.
  • step S240 may include: performing a matrix multiplication operation on the transposed sub-matrix and the non-zero block based on a SIMD instruction to obtain an intermediate matrix; and performing a matrix multiplication operation on the intermediate matrix and the transformed sub-matrix based on a SIMD instruction to obtain a frequency coefficient The result of the inverse transformation of the matrix.
  • SIMD Single Instruction Multiple Data
  • Common SIMD instruction sets include MMX, SSE, and AVX instruction sets of the X86 architecture, NEON instruction sets of the ARM Cortex architecture, X-Burst instruction sets of the MIPS architecture, and so on.
  • Performing matrix multiplication operations on transposed sub-matrixes and non-zero blocks based on SIMD instructions, and performing matrix multiplication operations on intermediate matrices and transformation sub-matrixes can improve the computational efficiency of inverse transformation, thereby improving the efficiency of video encoding and decoding.
  • the following steps S351-S356 can be used to perform matrix multiplication operations on transposed sub-matrixes and non-zero blocks based on SIMD instructions to obtain every adjacent multiple rows of the intermediate matrix (for example, every adjacent two rows, every adjacent row elements of four lines, etc.):
  • Step S351 dividing the adjacent rows corresponding to the adjacent rows of the intermediate matrix in the transposed sub-matrix into at least one first block;
  • Step S352 based on the above-mentioned at least one first block, divide the non-zero block into at least one second block, wherein the number of columns of the first block is the same as the number of rows of the second block;
  • Step S353 after copying the elements of each row in each first block multiple times, rearrange them row by row to obtain the matrix of the first row corresponding to each first block, wherein the elements of each row in each first block
  • the number of copies is the same as the number of columns per second block
  • Step S354 rearranging the elements in each second block by column to obtain a second row matrix corresponding to each second block;
  • Step S355 for each first block, based on the SIMD instruction, multiply the first row matrix of the first block by the corresponding element of the second row matrix of each second block, and multiply the first summing the results obtained by multiplying adjacent elements corresponding to each row in the first block of the row matrix to obtain a third row matrix;
  • Step S356 adding elements corresponding to positions of one or more third row matrices corresponding to one or more first blocks located in the same row, to obtain corresponding rows of the intermediate matrix.
  • steps S351 and S352 there are multiple ways to divide adjacent rows in the transposed sub-matrix into at least one first block, and divide non-zero blocks into at least one second block.
  • the non-zero blocks may be divided by rows, and correspondingly, at least one second block obtained by the division is distributed along the row direction, and each second block includes at least one complete row of the non-zero blocks.
  • the size of the first block can be determined according to the size of the second block, for example, the number of rows of the first block corresponds to the above-mentioned multiple adjacent rows of the intermediate matrix, and the number of columns of the first block is the same as the number of rows of the second block.
  • the non-zero blocks may be divided by rows and columns, and correspondingly, at least one second block obtained by the division is distributed in a matrix, and each second block includes a part of the rows and a part of the columns in the non-zero block, That is, each second block does not include a full row or a full column in the nonzero block.
  • the size of the first block can be determined according to the size of the second block, for example, the number of rows of the first block corresponds to the above-mentioned multiple adjacent rows of the intermediate matrix, and the number of columns of the first block is the same as the number of rows of the second block.
  • steps S351 and S352 it may be determined in response to determining that the number of rows of a non-zero block is greater than or equal to 2 (for example, the number of rows is 2, 4, 8, 16, 32, etc.) and the number of columns is greater than or equal to 4 (For example, the number of columns is 4, 8, 16, 32, etc.), divide the adjacent rows of the transposed sub-matrix into at least one 2*2 first block, and divide the non-zero block into at least one 2*4 second block.
  • 2 for example, the number of rows is 2, 4, 8, 16, 32, etc.
  • 4 the number of columns is 4, 8, 16, 32, etc.
  • steps S351 and S352 in response to determining that the number of rows of a non-zero block is greater than or equal to 2 (for example, the number of rows is 2, 4, 8, 16, 32, etc.) and the number of columns is equal to 2, transpose
  • the adjacent multiple rows of the sub-matrix are divided into at least one 4*2 first block, and the non-zero blocks are divided into at least one 2*2 second block.
  • the following still takes the transposed sub-matrix H T ' and the non-zero block C (the elements in the transposed sub-matrix H T ' and the non-zero block C are both 2-byte integer values) as an example, and the SIMD instruction is AVX Instructions are taken as an example to illustrate the process of calculating elements in every two rows of the intermediate matrix according to steps S351-S356 based on SIMD instructions.
  • steps S351 and S352 since the number of rows and columns of the non-zero block C are both 4, the condition that the number of rows is greater than or equal to 2 and the number of columns is greater than or equal to 4 is satisfied, so the first two rows of the transposed sub-matrix H T ' Divide into two first blocks 402, 404 of 2*2, and divide the non-zero block C into two second blocks 406, 408 of 2*4.
  • step S353 since the number of columns in the second block is 4, the elements in each row in the first block 402 are copied 4 times and then rearranged by row (the arrangement direction is shown by the arrow in Figure 4), to obtain the first row matrix410. Similarly, the elements in each row in the first block 404 are copied 4 times and rearranged row by row to obtain the first row matrix 412 .
  • the elements in the second block 406 are rearranged in columns, that is, rearranged in the direction indicated by the arrows in FIG. 4 .
  • AVX instructions can use 256-bit registers, that is, 16 2-byte data can be calculated in parallel (or 8 4-byte data can be calculated in parallel), and the rearranged elements can be copied twice to get the second line matrix414.
  • rearrange the elements in the second block 408 by column that is, rearrange according to the direction shown by the arrow in FIG. 4 , and copy the rearranged elements twice to obtain the second row matrix 416 .
  • step S355 in the matrix multiplication operation, the first block 402 corresponds to the second block 406 , and the first block 404 corresponds to the second block 408 .
  • the first row matrix 410 corresponding to the first two rows of the intermediate matrix is multiplied with the corresponding element of the second row matrix 414 of the second block 406, and then the first row
  • the results obtained by multiplying the two adjacent elements corresponding to each row in the first block 402 of the matrix 410 are summed (the number of adjacent elements to be added is the same as the column number of the first block, that is, each phase two adjacent elements) to obtain the corresponding matrix 418 in the third row.
  • the first row matrix 412 corresponding to the first two rows of the intermediate matrix is multiplied with the corresponding element in the position of the second row matrix 416 of the second block 408, and then the first row
  • the matrix 412 corresponding to each row in the first block 404 is obtained by multiplying two adjacent elements of the matrix 412 to obtain a matrix 420 corresponding to the third row.
  • step S356 based on the AVX instruction, the elements corresponding to the positions of the respective third row matrices 418 and 420 of the first blocks 402 and 404 in the same row are added to obtain the first two rows of elements corresponding to the middle matrix (i.e. 1-2 rows of elements) row matrix 422.
  • the first 4 elements in the row matrix 422 are the first row of the middle matrix
  • the last 4 elements are the second row of the middle matrix.
  • the elements of the 3rd-4th row, the 5th-6th row, and the 7th-8th row of the intermediate matrix can be calculated. Then, the elements in rows 1-2, elements in rows 3-4, elements in rows 5-6, and elements in rows 7-8 are spliced to obtain an 8*4 intermediate matrix.
  • steps S351-S356 can also be used to perform matrix multiplication on the intermediate matrix and the transformation sub-matrix based on SIMD instructions (replacing the transposed sub-matrix in steps S351-S356 with the intermediate matrix, replacing the non-zero blocks can be converted into a sub-matrix), and the elements of each adjacent multiple rows of the inverse transformation result of the frequency coefficient matrix are obtained. Then the elements of each adjacent row are spliced to obtain the inverse transformation result of the frequency coefficient matrix.
  • an image processing device may include: an acquiring unit 510 configured to acquire a frequency coefficient matrix of a target image, wherein the frequency coefficient matrix is obtained by performing frequency domain transformation on the target image, and The frequency coefficient matrix includes at least one non-zero frequency coefficient; the non-zero block determination unit 520 is configured to determine a non-zero block in the frequency coefficient matrix based on the position of the at least one non-zero frequency coefficient; the matrix determination unit 530, configured to determine a transform submatrix and a transpose submatrix corresponding to the non-zero block; and an inverse transform unit 540, configured to , to determine the inverse transformation result of the frequency coefficient matrix.
  • an acquiring unit 510 configured to acquire a frequency coefficient matrix of a target image, wherein the frequency coefficient matrix is obtained by performing frequency domain transformation on the target image, and The frequency coefficient matrix includes at least one non-zero frequency coefficient
  • the non-zero block determination unit 520 is configured to determine a non-zero block in the frequency coefficient matrix based on the
  • the non-zero blocks in the frequency coefficient matrix of the target image by determining the non-zero blocks in the frequency coefficient matrix of the target image, only the non-zero blocks are calculated to obtain the inverse transformation result of the frequency coefficient matrix, and there is no need to perform calculations on other frequency coefficients in the frequency coefficient matrix (i.e. Frequency coefficients other than zero blocks) are calculated, which greatly reduces the calculation amount of inverse transformation and improves the calculation efficiency of inverse transformation, thereby improving the efficiency of image and video encoding and decoding.
  • each unit of the apparatus 500 shown in FIG. 5 may correspond to each step in the method 200 described with reference to FIG. 3 .
  • the operations, features and advantages described above with respect to the method 200 are also applicable to the apparatus 500 and the units included therein. For the sake of brevity, some operations, features and advantages are not described in detail here.
  • various techniques may be described herein in the general context of software hardware elements or program modules.
  • the various units described above with respect to FIG. 5 may be implemented in hardware or in hardware combined with software and/or firmware.
  • these units may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer-readable storage medium.
  • these units may be implemented as hardware logic/circuitry.
  • one or more of the acquisition unit 510, the non-zero block determination unit 520, the matrix determination unit 530, and the inverse transformation unit 540 may be implemented together in a System on Chip (SoC).
  • SoC System on Chip
  • the SoC may include an integrated circuit chip (which includes a processor (for example, a central processing unit (Central Processing Unit, CPU), a microcontroller, a microprocessor, a digital signal processor (Digital Signal Processor, DSP), etc.), a memory, a or multiple communication interfaces, and/or one or more components in other circuits), and may optionally execute received program code and/or include embedded firmware to perform functions.
  • a processor for example, a central processing unit (Central Processing Unit, CPU), a microcontroller, a microprocessor, a digital signal processor (Digital Signal Processor, DSP), etc.
  • DSP Digital Signal Processor
  • an electronic device including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, and the computer program is executed by The at least one processor implements the method according to the above when executed.
  • a non-transitory computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.
  • a computer program product including a computer program, wherein the computer program implements the method according to the above when executed by a processor.
  • FIG. 6 a structural block diagram of an electronic device 600 that can serve as a server or a client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure.
  • the electronic devices may be different types of computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers.
  • Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices.
  • the components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.
  • an electronic device 600 may include at least one processor 601 capable of communicating with each other through a system bus 603, a working memory 602, an input unit 604, a display unit 605, a speaker 606, a storage unit 607, a communication unit 608, and other output Unit 609.
  • the processor 601 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores.
  • Processor 6101 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and/or any device that manipulates signals based on operational instructions.
  • the processor 601 may be configured to acquire and execute computer-readable instructions stored in the working memory 602, the storage unit 607 or other computer-readable media, such as program codes of the operating system 602a, program codes of the application program 602b, and the like.
  • the working memory 602 and the storage unit 607 are examples of computer-readable storage media for storing instructions, which are executed by the processor 601 to implement the various functions described above.
  • the working memory 602 may include both volatile and nonvolatile memory (eg, RAM, ROM, etc.). Additionally, storage unit 607 may include hard drives, solid state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD, DVD), storage arrays, network attached storage, storage areas net and so on.
  • Both the working memory 602 and the storage unit 607 may be collectively referred to as a memory or a computer-readable storage medium herein, and may be a non-transitory medium capable of storing computer-readable and processor-executable program instructions as computer program codes.
  • the program codes may be executed by the processor 601 as a specific machine configured to implement the operations and functions described in the examples herein.
  • the input unit 606 can be any type of equipment capable of inputting information to the electronic device 600, the input unit 606 can receive input digital or character information, and generate key signal input related to user settings and/or function control of the electronic device, and This may include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone, and/or remote control.
  • the output unit may be any type of device capable of presenting information, and may include, but is not limited to, a display unit 605, a speaker 606, and other output units 609, which may include, but are not limited to, video/audio output terminals, vibrators, and/or or a printer.
  • the communication unit 608 allows the electronic device 600 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks, and may include but not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and/or a chip Groups, such as Bluetooth TM devices, 1302.6 devices, WiFi devices, WiMax devices, cellular communication devices, and/or the like.
  • the application program 602b in the working register 602 can be loaded to execute various methods and processes described above, such as steps S210-step S240 in FIG. 3 .
  • the method 200 described above may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 607 .
  • part or all of the computer program can be loaded and/or installed on the electronic device 600 via the storage unit 607 and/or the communication unit 608 .
  • the processor 601 may be configured to execute the method 200 in any other suitable manner (for example, by means of firmware).
  • Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof.
  • FPGAs field programmable gate arrays
  • ASICs application specific integrated circuits
  • ASSPs application specific standard products
  • SOC system of systems
  • CPLD load programmable logic device
  • computer hardware firmware, software, and/or combinations thereof.
  • programmable processor can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.
  • Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented.
  • the program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.
  • a machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • a machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM compact disk read only memory
  • magnetic storage or any suitable combination of the foregoing.
  • the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • a keyboard and pointing device eg, a mouse or a trackball
  • Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.
  • the systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system.
  • the components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.
  • a computer system may include clients and servers.
  • Clients and servers are generally remote from each other and typically interact through a communication network.
  • the relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other.
  • steps may be reordered, added or deleted using the various forms of flow shown above.
  • each step described in the present disclosure may be executed in parallel, sequentially or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Discrete Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本公开提供了一种图像处理方法及装置、设备和介质，涉及图像及视频处理技术领域。实现方案为：获取目标图像的频率系数矩阵，所述频率系数矩阵是通过对所述目标图像进行频域变换来得到的，并且所述频率系数矩阵包括至少一个非零频率系数；基于所述至少一个非零频率系数的位置，确定所述频率系数矩阵中的非零块；确定所述非零块相应的变换子矩阵和转置子矩阵；以及基于所述变换子矩阵、所述非零块和所述转置子矩阵，确定所述频率系数矩阵的反变换结果。

Description

图像处理方法及装置、电子设备和存储介质
相关申请的交叉引用
本公开要求于2021年08月04日提交的中国专利申请202110891796.4的优先权，其全部内容通过引用整体结合在本公开中。
技术领域
本公开涉及图像及视频处理技术领域，具体涉及一种图像处理方法、图像处理装置、电子设备、计算机可读存储介质和计算机程序产品。
背景技术
图像、视频数据具有较强的空间相关性，在空间域存在大量的冗余信息。变换编码技术可以将图像或视频数据从空间域变换到频域，大大减少冗余信息，以实现对图像或视频数据的压缩，便于图像或视频数据的存储和传输。
例如，在视频压缩场景中，可以在编码端对各视频帧进行预测、变换(Transform)、量化、反量化、反变换(Inverse Transform)、重建、熵编码等处理，得到各视频帧的编码；然后在解码端对编码数据进行熵解码、反量化、反变换等处理，恢复出各视频帧。视频编码过程和视频解码过程均包括反变换处理。反变换处理的计算量大，计算效率低，需要占用大量的内存空间，大大限制了视频编解码的效率。
在此部分中描述的方法不一定是之前已经设想到或采用的方法。除非另有指明，否则不应假定此部分中描述的任何方法仅因其包括在此部分中就被认为是现有技术。类似地，除非另有指明，否则此部分中提及的问题不应认为在任何现有技术中已被公认。
发明内容
本公开提供了一种图像处理方法、图像处理装置、电子设备、计算机可读存储介质和计算机程序产品。
根据本公开的一方面，提供了一种图像处理方法，包括：获取目标图像的频率系数矩阵，所述频率系数矩阵是通过对所述目标图像进行频域变换来得到的，并且所述频率系数矩阵包括至少一个非零频率系数；基于所述至少一个非零频率系数的位置，确定所述频率系数矩阵中的非零块；确定所述非零块相应的变换子矩阵和转置子矩阵；以及基 于所述变换子矩阵、所述非零块和所述转置子矩阵，确定所述频率系数矩阵的反变换结果。
根据本公开的另一方面，还提供一种图像处理装置，包括：获取单元，被配置为获取目标图像的频率系数矩阵，所述频率系数矩阵是通过对所述目标图像进行频域变换来得到的，并且所述频率系数矩阵包括至少一个非零频率系数；非零块确定单元，被配置为基于所述至少一个非零频率系数的位置，确定所述频率系数矩阵中的非零块；矩阵确定单元，被配置为确定所述非零块相应的变换子矩阵和转置子矩阵；以及反变换单元，被配置为基于所述变换子矩阵、所述非零块和所述转置子矩阵，确定所述频率系数矩阵的反变换结果。
根据本公开的另一方面，还提供一种电子设备，包括：至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中所述存储器存储有计算机程序，所述计算机程序在被所述至少一个处理器执行时实现根据上述的方法。
根据本公开的另一方面，还提供一种存储有计算机程序的非瞬时计算机可读存储介质，其中，所述计算机程序在被处理器执行时实现根据上述的方法。
根据本公开的另一方面，还提供一种计算机程序产品，包括计算机程序，其中，所述计算机程序在被处理器执行时实现根据上述的方法。
根据本公开的一个或多个实施例，通过确定目标图像的频率系数矩阵中的非零块，仅对非零块进行计算来得到频率系数矩阵的反变换结果，无需对频率系数矩阵中的其他频率系数(即除非零块以外的频率系数)进行计算，大大减少了反变换的计算量，提高了反变换的计算效率，从而能够提高图像、视频编解码的效率。
应当理解，本部分所描述的内容并非旨在标识本公开的实施例的关键或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的说明书而变得容易理解。
附图说明
附图示例性地示出了实施例并且构成说明书的一部分，与说明书的文字描述一起用于讲解实施例的示例性实施方式。所示出的实施例仅出于例示的目的，并不限制权利要求的范围。在所有附图中，相同的附图标记指代类似但不一定相同的要素。
图1A示出了根据本公开的实施例的图像编码和解码过程的示意图；
图1B示出了根据本公开的实施例的视频编码端的结构示意图；
图1C示出了根据本公开的实施例的视频解码端的结构示意图；
图2示出了根据本公开的实施例的图像处理方法的流程图；
图3示出了根据本公开实施例的计算中间矩阵的每相邻多行元素的流程图；
图4示出了根据本公开的实施例的中间矩阵的前两行元素的计算过程的示意图；
图5示出了根据本公开的实施例的图像处理装置的结构框图；以及
图6示出了能够用于实现本公开的实施例的示例性电子设备的结构框图。
具体实施方式
以下结合附图对本公开的示范性实施例做出说明，其中包括本公开实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本公开的范围。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。
在本公开中，除非另有说明，否则使用术语“第一”、“第二”等来描述各种要素不意图限定这些要素的位置关系、时序关系或重要性关系，这种术语只是用于将一个元件与另一元件区分开。在一些示例中，第一要素和第二要素可以指向该要素的同一实例，而在某些情况下，基于上下文的描述，它们也可以指代不同实例。
在本公开中对各种所述示例的描述中所使用的术语只是为了描述特定示例的目的，而并非旨在进行限制。除非上下文另外明确地表明，如果不特意限定要素的数量，则该要素可以是一个也可以是多个。此外，本公开中所使用的术语“和/或”涵盖所列出的项目中的任何一个以及全部可能的组合方式。
下面将结合附图详细描述本公开的实施例。
为了便于理解，首先对图像、视频的变换编码过程进行简单介绍。
图像的编码过程大致如下：将图像切分为多个图像块，以图像块为变换单元(Transform Unit，TU)进行变换和编码。图像块例如可以是4*4、8*8、16*16、32*32大小的正方形图像，也可以是矩形图像。如图1A所示，在图像编码过程中，分别对每个图像块进行变换，将空间域的图像块数据转换到频域，以消除图像块中的冗余信息，得到每个图像块对应的频域的变换系数矩阵。变换所采用的方法例如可以是离散余弦变换(Discrete Cosine Transform，DCT)、离散正弦变换(Discrete Sine Transform，DST)等。对变换得到的变换系数矩阵进行量化，并对量化后的结果进行变长编码(例如Huffman编码等)，得到编码结果。
图像的解码过程大致如下：如图1A所示，对各图像块的编码结果进行变长解码、反量化、反变换，得到解码后的图像块。反变换是变换的逆过程，反变换所采用的方法例如可以是反离散余弦变换(Inverse Discrete Cosine Transform，IDCT)、反离散正弦变换(Inverse Discrete Sine Transform，IDST)等。
视频编码过程大致如下：原始的视频帧被切分成多个图像块，以图像块为变换单元(TU)进行变换编码。图像块例如可以是4*4、8*8、16*16、32*32大小的正方形图像，也可以是矩形图像。对于任一视频帧，可以由多种不同的通道构成，例如，YUV视频帧由亮度通道(Y)、第一色度通道(U)和第二色度通道(V)构成。相应地，视频帧可以包括不同通道的多个图像块，例如，YUV视频帧可以包括亮度图像块、第一色度图像块和第二色度图像块。
如图1B所示，编码端可以包括编码帧内预测模块、编码帧间预测模块、变换模块、量化模块、熵编码模块、反量化模块、反变换模块、重建模块、滤波模块、参考图像缓冲模块，分别用于对原始的视频帧进行预测、变换、量化、熵编码、反量化、反变换、重建、滤波等处理，最终输出视频的编码流。
如图1B所示，在视频编码过程中，原始视频帧作为加法器的一个输入，编码帧内预测模块或编码帧间预测模块输出的预测数据为加法器的另一个输入，加法器输出预测残差矩阵。预测残差矩阵经过变换、量化处理，得到残差系数矩阵。残差系数矩阵经过反量化、反变换处理，恢复出预测残差矩阵。恢复出的预测残差矩阵可用于重构视频帧。变换处理所采用的方法例如可以是DCT、DST等。反变换是变换的逆过程，相应地，反变换处理所采用的方法例如可以是IDCT、IDST等。
视频解码过程大致如下：如图1C所示，首先对接收到的编码流进行熵解码，得到残差系数矩阵、帧间预测相关信息、帧内预测相关信息等。残差系数矩阵经过反量化、反变换处理之后，恢复出预测残差矩阵。重建模块将预测残差加回至相应的帧内预测数据或帧间预测数据。可以通过开关选择使用帧内预测数据或帧间预测数据，以得到重建帧。相应地，解码端可以包括解码帧内预测模块、解码帧间预测模块、熵解码模块、反量化模块、反变换模块、重建模块、滤波模块、参考图像缓存模块、视频播放缓存模块。
由以上介绍可见，反变换处理可以被应用于图像解码(用于对变换系数矩阵进行反变换)、视频编解码(用于对残差系数矩阵进行反变换)等过程中。反变换处理的计算量很大。
以应用场景为视频编解码、TU的尺寸为8*8(即残差系数矩阵的尺寸为8*8)、反变换方法为IDCT为例，对残差系数矩阵X进行反变换，即进行如下式所示的计算：
Y＝H 8 TXH 8
其中，Y为反变换处理得到的结果矩阵，H 8为8*8的DCT变换矩阵，X为待进行反变换的残差系数矩阵，H 8 T为H 8的转置矩阵。
需要说明的是，DCT变换矩阵H 8中的元素是通过余弦函数计算得出，其值通常是一个浮点数。为了便于存储和计算，可以将变换矩阵H 8表示成一个浮点数的公共因子与一个整数矩阵的乘积，相应地，变换矩阵H 8的转置矩阵H 8 T也可以表示成同一个公共因子与一个整数矩阵H T的乘积。例如，8*8的变换矩阵H 8及其转置矩阵H 8 T的公共因子可以是0.0055，相应的变换矩阵H 8对应的整数矩阵H如下所示：
Figure PCTCN2022098600-appb-000001
转置矩阵H 8 T对应的整数矩阵H T如下所示：
Figure PCTCN2022098600-appb-000002
相应地，结果矩阵Y可以表示为：Y＝0.0055*0.0055*H TXH。
为了便于描述，在下文中，用变换矩阵对应的整数矩阵来指代该变换矩阵，用变换矩阵的转置矩阵对应的整数矩阵来指代该转置矩阵。
残差系数矩阵X如下所示：
Figure PCTCN2022098600-appb-000003
如果直接按照上式进行矩阵乘法运算，需要先将H T与X做矩阵乘法，然后再将乘法结果与H做矩阵乘法，计算量非常大。
相关技术中，可以采用蝶形运算来加速反变换的计算过程。采用蝶形运算来计算Y＝H TXH的过程如下：
第一步：计算B＝H TX
提取公共项：
O0＝89×A10+75×A30+50×A50+18×A70
O1＝75×A10-18×A30-89×A50-50×A70
O2＝50×A10-89×A30+18×A50+75×A70
O3＝18×A10-50×A30+75×A50-89×A70
EE0＝64×A00+64×A40
EE1＝64×A00-64×A40
EO0＝83×A20+36×A60
EO1＝36×A20-83×A60
E0＝EE0+EO0
E1＝EE1+EO0
E2＝EE1-EO1
E3＝EE0-EO0
矩阵B的第一列可以通过公共项计算得到：
B00＝E0+O0
B10＝E1+O1
B20＝E2+O2
B30＝E3+O3
B40＝E3-O3
B50＝E2-O2
B60＝E1-O1
B70＝E0-O0
按照类似的方法，可以计算出矩阵B中的第二至第八列。
第二步：按照与第一步类似的方式，计算Y＝BH。
发明人经研究发现，在对TU进行变换(在变换后还可以进行量化和反量化)所得到的频率系数矩阵中(例如上文所述的变换系数矩阵和残差系数矩阵)，通常只有左上角的部分具有非零频率系数，其他大部分频率系数的值均为零。
以8比特图像(即图像的每个像素的每个通道的像素值用一个字节表示)为例，一个8*8的亮度通道(即Y通道)的TU经过变换、量化、反量化后，得到如下所示的频率系数矩阵X：
Figure PCTCN2022098600-appb-000004
可见，频率系数矩阵X仅有位于左上角区域的4个非零频率系数，其他频率系数的值均为零。如果按照直接进行矩阵乘法或蝶形运算的计算方式来对频率系数矩阵X进行反变换，将会有很多计算是无效的，因为对零的加减乘除运算的结果仍然是零，造成大量的存储和计算资源浪费。
基于此，本公开提供一种改进的图像处理方法。该方法获取目标图像的频率系数矩阵，基于频率系数矩阵所包括的至少一个非零频率系数的位置，确定频率系数矩阵中的非零块，基于非零块来分别对变换矩阵和变换矩阵的转置矩阵进行裁剪，得到变换子矩阵和转置子矩阵，然后基于变换子矩阵、非零块和转置子矩阵来确定频率系数矩阵的反变换结果。本公开实施例的图像处理方法通过确定目标图像的频率系数矩阵中的非零块，仅对非零块进行计算来得到频率系数矩阵的反变换结果，无需对频率系数矩阵中的其他频率系数(即除非零块以外的频率系数)进行计算，大大减少了反变换的计算量，提高了反变换的计算效率，从而能够提高图像、视频编解码的效率。
下面将结合附图具体描述本公开实施例中的图像处理方法。
图2示出了根据本公开的实施例的图像处理方法200的流程图。方法200例如可以应用于图像解码或视频编解码的场景中，能够提高反变换的计算效率，从而提高图像解码以及视频编解码的效率。
如图2所示，方法200可以包括：
步骤S210、获取目标图像的频率系数矩阵，频率系数矩阵是通过对目标图像进行频域变换来得到的，并且频率系数矩阵包括至少一个非零频率系数；
步骤S220、基于上述至少一个非零频率系数的位置，确定频率系数矩阵中的非零块；
步骤S230、确定所述非零块相应的变换子矩阵和转置子矩阵；以及
步骤S240、基于变换子矩阵、非零块和转置子矩阵，确定频率系数矩阵的反变换结果。
根据本公开的实施例，通过确定目标图像的频率系数矩阵中的非零块，仅对非零块进行计算来得到频率系数矩阵的反变换结果，无需对频率系数矩阵中的其他频率系数(即除非零块以外的频率系数)进行计算，大大减少了反变换的计算量，提高了反变换的计算效率，从而能够提高图像、视频编解码的效率。
目标图像可以但不限于为单个图像或视频帧中的一个图像块(即TU)。例如，目标图像可以是待解码的图像中的图像块，或者是待编码的视频帧中的图像块或待解码的视频帧中的图像块。进一步地，目标图像例如可以为视频帧的亮度通道的图像块，也可以是其他通道的图像块。例如，针对YUV视频帧，目标图像也可以为第一色度通道的图像块或第二色度通道的图像块。
目标图像的频率系数矩阵是通过对目标图像进行频域变换来得到的。如上文所述，频率系数矩阵例如可以是对图像的图像块进行频域变换所得到的变换系数矩阵，或者是对视频帧的图像块的预测残差进行频域变换、量化、反量化所得到的残差系数矩阵。对目标图像进行频域变换的方法包括但不限于DCT、DST等。
目标图像例如可以是4*4、8*8、16*16、32*32等宽度与高度相等的正方形图像，相应地，对目标图像进行频域变换所得到的频率系数矩阵可以是4*4、8*8、16*16、32*32等行数与列数相等的矩阵(即方阵)。目标图像例如也可以是4*8，8*4，4*16，16*4，8*16，16*8，16*32，32*16，8*32，32*8等宽度与高度不相等的矩形图像，相应地，对目标图像进行频域变换所得到的频率系数矩阵可以是4*8，8*4，4*16，16*4，8*16，16*8，16*32，32*16，8*32，32*8等行数与列数不相等的矩阵。
根据一些实施例，步骤S220可以包括：确定上述至少一个非零频率系数所对应的行坐标最大值和列坐标最大值；以及根据行坐标最大值和列坐标最大值来确定非零块。如前所述，由于频率系数矩阵中的非零频率系数通常位于矩阵的左上角区域，因此可以通过频率系数矩阵中的至少一个非零频率系数对应的行坐标最大值以及纵坐标最大值，来确定非零频率系数矩阵中的非零频率系数所在的位置区域，从而确定出频率系数矩阵中的非零块。
本公开的实施例中，频率系数的行坐标可以是指该频率系数在频率系数矩阵中所处的行，频率系数的列坐标可以是指该频率系数在频率系数矩阵中所处的列。
根据一些实施例，非零块包括频率系数矩阵中所对应的行坐标小于等于行坐标最大值且所对应的列坐标小于等于列坐标最大值的一个或多个频率系数。由于频率系数矩阵中的非零频率系数通常位于矩阵的左上角区域，行坐标最大值和列坐标最大值对应于频率系数矩阵中的最右下角的非零频率系数。通过使非零块包括频率系数矩阵中的行坐标小于等于行坐标最大值且列坐标小于等于列坐标最大值的频率系数，可以保证非零块包括频率系数矩阵中的所有非零频率系数，使得在后续步骤中能够对频率系数矩阵进行无损(即不丢失信息)的反变换。
根据一些实施例，可以由频率系数矩阵中所对应的行坐标小于等于行坐标最大值且所对应的列坐标小于等于列坐标最大值的一个或多个频率系数来组成非零块。即，非零块为位于频率系数矩阵左上角的尺寸为行坐标最大值*列坐标最大值的块。
根据另一些实施例，可以按照以下步骤来确定非零块：基于行坐标最大值，确定非零块的最大行边界值；基于列坐标最大值，确定非零块的最大列边界值，其中，最大行边界值为大于等于行坐标最大值的最小的2的幂数，最大列边界值为大于等于列坐标最大值的最小的2的幂数。相应地，非零块由频率系数矩阵中的行坐标小于等于最大行边界值且列坐标小于等于最大列边界值的所有频率系数组成。最大行边界值、最大列边界值均为2的幂数，通过将行坐标最大值扩展为最大行边界值、将列坐标最大值扩展为最大列边界值，并根据最大行边界值、最大列边界值来确定非零块，可以使确定出的非零块的行数、列数均为2的幂数，从而便于实现指令集优化，提高后续反变换计算过程中的内存空间的利用效率，加速计算过程。
仍以上文的频率系数矩阵X为例，该频率系数矩阵X包括4个非零频率系数，位置坐标分别为(2,1)、(2,4)、(3,1)、(3,2)。因此，非零频率系数的行坐标最大值为max(2,2,3,3)＝3，列坐标最大值为max(1,4,1,2)＝4。大于或等于行坐标最大值3的最 小的2的幂数为4，即最大行边界值为4；大于或等于列坐标最大值4的最小的2的幂数为4，即最大列边界值为4。相应地，非零块由频率系数矩阵中的行坐标小于等于最大行边界4且列坐标小于等于最大列边界值4的所有频率系数组成，即非零块为频率系数矩阵X的左上角的4*4的区域。即，非零块C如下：
Figure PCTCN2022098600-appb-000005
根据一些实施例，步骤S230可以包括：基于频率系数矩阵，确定预设的变换矩阵和该变换矩阵的转置矩阵；以及基于非零块，对变换矩阵进行裁剪以得到变换子矩阵，并且对转置矩阵进行裁剪以得到转置子矩阵。
根据一些实施例，变换矩阵及其转置矩阵可以通过频率系数矩阵的尺寸来确定。具体地，可以根据频率系数矩阵的尺寸来确定变换矩阵的尺寸。
当频率系数矩阵为4*4、8*8、16*16、32*32等行数与列数相等的矩阵(即方阵)时，变换矩阵H的尺寸与频率系数矩阵的尺寸相同，即变换矩阵为行数和列数均等于频率系数矩阵的行数(或列数)的方阵。例如，当频率系数矩阵为8*8的方阵时，相应的变换矩阵H及其转置矩阵H T也是8*8的方阵。
当频率系数矩阵为4*8，8*4，4*16，16*4，8*16，16*8，16*32，32*16，8*32，32*8等行数与列数不相等的矩阵时，变换矩阵H为行数、列数均等于频率系数矩阵的列数的方阵，矩阵H T为行数、列数均等于频率系数矩阵的行数的方阵。例如，当频率系数矩阵为8*4的矩阵时，相应的变换矩阵H为4*4的方阵，H T为8*8的方阵。
基于所确定的变换矩阵的尺寸，可以按照预设的公式来计算得出变换矩阵中的每一个元素，从而得出变换矩阵。进一步地，对变换矩阵进行转置，即得到变换矩阵的转置矩阵。
例如，频率系数矩阵的尺寸为8*8，相应地，变换矩阵的尺寸也是8*8。以反变换为IDCT为例，当变换矩阵为方阵(即变换矩阵的行数与列数相同)时，可以按照以下公式来计算得出变换矩阵中的第i行第j列的元素D i,j：
Figure PCTCN2022098600-appb-000006
其中，N为变换矩阵的行数(或列数)。可以理解的，按照上述公式计算出的元素为浮点数D i,j。根据一些实施例，可以对变换矩阵中的元素D i,j进行缩放，以将其转化成整数，从而方便计算机处理。具体地，D i,j的缩放倍数可以根据不同的应用场景来确定。例 如，针对不同的视频编码标准，可以设置不同的缩放倍数。针对HEVC(High Efficiency Video Coding，高效视频编码)视频编码标准，可以将变换矩阵中的元素扩大128倍；针对AV1视频编码标准，可以将变换矩阵中的元素扩大8192倍。
根据一些实施例，可以基于非零块的尺寸，来对变换矩阵进行裁剪以得到变换子矩阵，并且对变换矩阵的转置矩阵进行裁剪以得到转置子矩阵。裁剪所得到的变换子矩阵和转置子矩阵的尺寸与非零块的尺寸相适应，能够与非零块进行矩阵乘法运算。
仍以上文的8*8的变换矩阵H及其转置矩阵H T为例，基于4*4非零块C，对转置矩阵H T进行裁剪，得到8*4的转置子矩阵H T’：
Figure PCTCN2022098600-appb-000007
对8*8的变换矩阵H进行裁剪，得到4*8的变换子矩阵H’：
Figure PCTCN2022098600-appb-000008
可以理解的是，也可以基于非零块的尺寸直接确定上述变化子矩阵和转置子矩阵，具体的方法与基于频率系数矩阵确定变化矩阵和转置矩阵类似，在此不再详述。
在确定非零块相应的变化子矩阵和转置子矩阵之后，即可执行步骤S240，以确定频率系数矩阵的反变换结果。
根据一些实施例，步骤S240可以为响应于以下至少一个条件来执行：非零块的最大行边界值小于频率系数矩阵的总行数；非零块的最大列边界值小于频率系数矩阵的总列数。
当最大行边界值小于频率系数矩阵的总行数，和/或最大列边界值小于频率系数矩阵的总列数时，根据最大行边界值、最大列边界值确定出的非零块的尺寸小于频率系数矩阵的尺寸，因此基于非零块来确定频率系数矩阵的反变换结果，能够减少反变换的计算量，提高计算效率。
当最大行边界值大于等于频率系数矩阵的总行数且最大列边界值大于等于频率系数矩阵的总列数时，根据最大行边界值、最大列边界值确定得出的非零块的尺寸将大于等于频率系数矩阵的尺寸，因此无法达到减少计算量、提高计算效率的技术效果。在这种情况下，可以执行步骤S240来对频率系数矩阵进行反变换，也可以不执行步骤S240，而直接采用传统方法(例如直接对频率系数矩阵进行矩阵乘法或蝶形运算)来对频率系数矩阵进行反变换。
根据一些实施例，步骤S240可以包括：基于SIMD指令对转置子矩阵和非零块进行矩阵乘法运算，得到中间矩阵；以及基于SIMD指令对中间矩阵和变换子矩阵进行矩阵乘法运算，得到频率系数矩阵的反变换结果。
SIMD(Single Instruction Multiple Data，单指令多数据流)能够复制多个操作数，并把它们打包在大型寄存器中进行并行计算。常见的SIMD指令集包括X86架构的MMX、SSE、AVX指令集，ARM Cortex架构的NEON指令集，MIPS架构的X-Burst指令集，等等。基于SIMD指令来对转置子矩阵和非零块进行矩阵乘法运算，以及对中间矩阵和变换子矩阵进行矩阵乘法运算，能够提高反变换的计算效率，从而提高视频编解码的效率。
根据一些实施例，可以按照以下步骤S351-S356来基于SIMD指令对转置子矩阵和非零块进行矩阵乘法运算，得到中间矩阵的每相邻多行(例如每相邻两行、每相邻四行等)的元素：
步骤S351、将转置子矩阵中的与中间矩阵的该相邻多行相对应的相邻多行划分为至少一个第一块；
步骤S352、基于上述至少一个第一块，将非零块划分为至少一个第二块，其中，第一块的列数和第二块的行数相同；
步骤S353、将每个第一块中的每一行元素复制多次后，按行重排，得到每个第一块对应的第一行矩阵，其中，每个第一块中的每一行元素的复制次数与每个第二块的列数相同；
步骤S354、将每个第二块中的元素按列重排，得到每个第二块对应的第二行矩阵；
步骤S355、对于每个第一块，基于SIMD指令，将该第一块的第一行矩阵分别和相应的每个第二块的第二行矩阵的位置对应的元素相乘，并且将第一行矩阵的与该第一块中每一行对应的多个相邻元素经过相乘所得到的结果进行求和，得到第三行矩阵；以及
步骤S356、将位于同一行的一个或多个第一块各自相应的一个或多个第三行矩阵的位置对应的元素相加，得到中间矩阵的相应行。
针对步骤S351、S352，将转置子矩阵中的相邻多行划分为至少一个第一块，将非零块划分为至少一个第二块有多种方式。
根据一些实施例，可以按行来划分非零块，相应地，划分所得到的至少一个第二块沿行方向分布，每个第二块包括非零块中的至少一个完整行。第一块的尺寸可以根据第二块的尺寸来确定，例如，第一块的行数与中间矩阵的上述相邻多行对应，并且第一块的列数与第二块的行数相同。
根据另一些实施例，可以按行和列来划分非零块，相应地，划分所得到的至少一个第二块呈矩阵分布，每个第二块包括非零块中的部分行和部分列，即每个第二块不包括非零块中的完整行或完整列。第一块的尺寸可以根据第二块的尺寸来确定，例如，第一块的行数与中间矩阵的上述相邻多行对应，并且第一块的列数与第二块的行数相同。
更具体地，根据一些实施例，针对步骤S351、S352，可以响应于确定非零块的行数大于等于2(例如行数为2,4,8,16,32等)且列数大于等于4(例如列数为4,8,16,32等)，将转置子矩阵的相邻多行划分为至少一个2*2的第一块，并且将非零块划分为至少一个2*4的第二块。
根据另一些实施例，针对步骤S351、S352，可以响应于确定非零块的行数大于等于2(例如行数为2,4,8,16,32等)且列数等于2，将转置子矩阵的相邻多行划分为至少一个4*2的第一块，并且将非零块划分为至少一个2*2的第二块。
以下仍以转置子矩阵H T’和非零块C(转置子矩阵H T’和非零块C中的元素均为2字节的整型数值)为例，并且以SIMD指令为AVX指令为例，来说明基于SIMD指令按照步骤S351-S356来计算出中间矩阵的每两行元素的过程。
首先，按照步骤S351-S356来计算中间矩阵的前两行元素：
针对步骤S351、S352，由于非零块C的行数和列数均为4，满足行数大于等于2且列数大于等于4的条件，因此可以将转置子矩阵H T’的前两行划分为两个2*2的第一块402、404，以及将非零块C划分为两个2*4的第二块406、408。
针对步骤S353，由于第二块的列数为4，因此将第一块402中的每一行元素复制4次后按行重排(排列方向如图4中的箭头所示)，得到第一行矩阵410。类似地，将第一块404中的每一行元素复制4次后按行重排，得到第一行矩阵412。
针对步骤S354，将第二块406中的元素按列重排，即按照图4中箭头所示的方向重排。考虑到AVX指令可以使用256比特的寄存器，即可以并行计算16个2字节的数据(或者并行计算8个4字节的数据等)，将重排后的元素复制2次，得到第二行矩阵414。类似地，将第二块408中的元素按列重排，即按照图4中箭头所示的方向重排，并将重排后的元素复制2次，得到第二行矩阵416。
针对步骤S355，在矩阵乘法运算中，第一块402对应于第二块406，第一块404对应于第二块408。
对于第一块402，基于AVX指令，将中间矩阵的前两行对应的第一行矩阵410和相应的第二块406的第二行矩阵414的位置对应的元素相乘，然后将第一行矩阵410的与第一块402中每一行对应的两个相邻元素经过相乘所得到的结果进行求和(进行相加的相邻元素的数量与第一块的列数相同，即将每相邻的两个元素相加)，得到对应的第三行矩阵418。
对于第一块404，基于AXV指令，将中间矩阵的前两行对应的第一行矩阵412和相应的第二块408的第二行矩阵416的位置对应的元素相乘，然后将第一行矩阵412的与第一块404中每一行对应的两个相邻元素经过相乘所得到的结果，得到对应的第三行矩阵420。
针对步骤S356，基于AVX指令，将位于同一行的第一块402、404各自相应的第三行矩阵418、420的位置对应的元素相加，得到对应于中间矩阵的前两行元素(即第1-2行元素)的行矩阵422。具体地，行矩阵422中的前4个元素为中间矩阵的第1行，后4个元素为中间矩阵的第2行。
按照类似的方法，可以计算出中间矩阵的第3-4行元素、第5-6行元素、第7-8行元素。然后将第1-2行元素、第3-4行元素、第5-6行元素、第7-8行元素进行拼接，得到8*4的中间矩阵。
可以理解，也可以按照步骤S351-S356的方法，来基于SIMD指令对中间矩阵和变换子矩阵进行矩阵乘法运算(将步骤S351-S356中的转置子矩阵替换为中间矩阵，将非零块替换成变换子矩阵即可)，得到频率系数矩阵的反变换结果的每相邻多行的元素。然后将各相邻多行的元素进行拼接，得到频率系数矩阵的反变换结果。
根据本公开的另一方面，还提供一种图像处理装置。如图6所示，该装置600可以包括：获取单元510，被配置为获取目标图像的频率系数矩阵，其中，所述频率系数矩阵是通过对所述目标图像进行频域变换来得到的，并且所述频率系数矩阵包括至少一个非 零频率系数；非零块确定单元520，被配置为基于所述至少一个非零频率系数的位置，确定所述频率系数矩阵中的非零块；矩阵确定单元530，被配置为确定所述非零块相应的变换子矩阵和转置子矩阵；以及反变换单元540，被配置为基于所述变换子矩阵、所述非零块和所述转置子矩阵，确定所述频率系数矩阵的反变换结果。
根据本公开的实施例，通过确定目标图像的频率系数矩阵中的非零块，仅对非零块进行计算来得到频率系数矩阵的反变换结果，无需对频率系数矩阵中的其他频率系数(即除非零块以外的频率系数)进行计算，大大减少了反变换的计算量，提高了反变换的计算效率，从而能够提高图像、视频编解码的效率。
应当理解，图5中所示装置500的各个单元可以与参考图3描述的方法200中的各个步骤相对应。由此，上面针对方法200描述的操作、特征和优点同样适用于装置500及其所包括的单元。为了简洁起见，某些操作、特征和优点在此不再赘述。
还应当理解，本文可以在软件硬件元件或程序模块的一般上下文中描述各种技术。上面关于图5描述的各个单元可以在硬件中或在结合软件和/或固件的硬件中实现。例如，这些单元可以被实现为计算机程序代码/指令，该计算机程序代码/指令被配置为在一个或多个处理器中执行并存储在计算机可读存储介质中。可替换地，这些单元可以被实现为硬件逻辑/电路。例如，在一些实施例中，获取单元510、非零块确定单元520、矩阵确定单元530以及反变换单元540中的一个或多个可以一起被实现在片上系统(System on Chip,SoC)中。SoC可以包括集成电路芯片(其包括处理器(例如，中央处理单元(Central Processing Unit,CPU)、微控制器、微处理器、数字信号处理器(Digital Signal Processor,DSP)等)、存储器、一个或多个通信接口、和/或其他电路中的一个或多个部件)，并且可以可选地执行所接收的程序代码和/或包括嵌入式固件以执行功能。
根据本公开的另一方面，还提供一种电子设备，包括：至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中所述存储器存储有计算机程序，所述计算机程序在被所述至少一个处理器执行时实现根据上述的方法。
根据本公开的另一方面，还提供一种存储有计算机程序的非瞬时计算机可读存储介质，其中，所述计算机程序在被处理器执行时实现根据上述的方法。
根据本公开的另一方面，还提供一种计算机程序产品，包括计算机程序，其中，所述计算机程序在被处理器执行时实现根据上述的方法。
参见图6，现将描述可以作为本公开的服务器或客户端的电子设备600的结构框图，其是可以应用于本公开的各方面的硬件设备的示例。电子设备可以是不同类型的计算机 设备，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本公开的实现。
如图6所示，电子设备600可以包括能够通过系统总线603彼此通信的至少一个处理器601、工作存储器602、输入单元604、显示单元605、扬声器606、存储单元607、通信单元608以及其它输出单元609。
处理器601可以是单个处理单元或多个处理单元，所有处理单元可以包括单个或多个计算单元或者多个核心。处理器6101可以被实施成一个或更多微处理器、微型计算机、微控制器、数字信号处理器、中央处理单元、状态机、逻辑电路和/或基于操作指令来操纵信号的任何设备。处理器601可以被配置成获取并且执行存储在工作存储器602、存储单元607或者其他计算机可读介质中的计算机可读指令，诸如操作系统602a的程序代码、应用程序602b的程序代码等。
工作存储器602和存储单元607是用于存储指令的计算机可读存储介质的示例，指令由处理器601执行来实施前面所描述的各种功能。工作存储器602可以包括易失性存储器和非易失性存储器二者(例如RAM、ROM等等)。此外，存储单元607可以包括硬盘驱动器、固态驱动器、可移除介质、包括外部和可移除驱动器、存储器卡、闪存、软盘、光盘(例如CD、DVD)、存储阵列、网络附属存储、存储区域网等等。工作存储器602和存储单元607在本文中都可以被统称为存储器或计算机可读存储介质，并且可以是能够把计算机可读、处理器可执行程序指令存储为计算机程序代码的非暂态介质，计算机程序代码可以由处理器601作为被配置成实施在本文的示例中所描述的操作和功能的特定机器来执行。
输入单元606可以是能向电子设备600输入信息的任何类型的设备，输入单元606可以接收输入的数字或字符信息，以及产生与电子设备的用户设置和/或功能控制有关的键信号输入，并且可以包括但不限于鼠标、键盘、触摸屏、轨迹板、轨迹球、操作杆、麦克风和/或遥控器。输出单元可以是能呈现信息的任何类型的设备，并且可以包括但不限于显示单元605、扬声器606以及其它输出单元609，其它输出单元609可以但不限于包括视频/音频输出终端、振动器和/或打印机。通信单元608允许电子设备600通过诸如因特网的计算机网络和/或各种电信网络与其他设备交换信息/数据，并且可以包括但不限于 调制解调器、网卡、红外通信设备、无线通信收发机和/或芯片组，例如蓝牙 TM设备、1302.6设备、WiFi设备、WiMax设备、蜂窝通信设备和/或类似物。
工作寄存器602中的应用程序602b可以被加载执行上文所描述的各个方法和处理，例如图3中的步骤S210-步骤S240。例如，在一些实施例中，上文描述的方法200可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元607。在一些实施例中，计算机程序的部分或者全部可以经由存储单元607和/或通信单元608而被载入和/或安装到电子设备600上。当计算机程序被加载并由处理器601执行时，可以执行上文描述的方法200的一个或多个步骤。备选地，在其他实施例中，处理器601可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行方法200。
本文中以上描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、芯片上系统的系统(SOC)、负载可编程逻辑设备(CPLD)、计算机硬件、固件、软件、和/或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。
用于实施本公开的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和/或框图中所规定的功能/操作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)和互联网。
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。
应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本公开中记载的各步骤可以并行地执行、也可以顺序地或以不同的次序执行，只要能够实现本公开公开的技术方案所期望的结果，本文在此不进行限制。
虽然已经参照附图描述了本公开的实施例或示例，但应理解，上述的方法、系统和设备仅仅是示例性的实施例或示例，本发明的范围并不由这些实施例或示例限制，而是仅由授权后的权利要求书及其等同范围来限定。实施例或示例中的各种要素可以被省略或者可由其等同要素替代。此外，可以通过不同于本公开中描述的次序来执行各步骤。进一步地，可以以各种方式组合实施例或示例中的各种要素。重要的是随着技术的演进，在此描述的很多要素可以由本公开之后出现的等同要素进行替换。

Claims (17)

  1. 一种图像处理方法，包括：
    获取目标图像的频率系数矩阵，其中，所述频率系数矩阵是通过对所述目标图像进行频域变换来得到的，并且所述频率系数矩阵包括至少一个非零频率系数；
    基于所述至少一个非零频率系数的位置，确定所述频率系数矩阵中的非零块；
    确定所述非零块相应的变换子矩阵和转置子矩阵；以及
    基于所述变换子矩阵、所述非零块和所述转置子矩阵，确定所述频率系数矩阵的反变换结果。
  2. 如权利要求1所述的方法，其中，基于所述至少一个非零频率系数的位置，确定所述频率系数矩阵中的非零块包括：
    确定所述至少一个非零频率系数所对应的行坐标最大值和列坐标最大值；以及
    根据所述行坐标最大值和所述列坐标最大值来确定所述非零块。
  3. 如权利要求2所述的方法，其中，所述非零块包括所述频率系数矩阵中所对应的行坐标小于等于所述行坐标最大值且所对应的列坐标小于等于所述列坐标最大值的一个或多个频率系数。
  4. 如权利要求2或3所述的方法，其中，根据所述行坐标最大值和所述列坐标最大值来确定所述非零块包括：
    基于所述行坐标最大值，确定所述非零块的最大行边界值；
    基于所述列坐标最大值，确定所述非零块的最大列边界值，
    其中，所述最大行边界值为大于等于所述行坐标最大值的最小的2的幂数，所述最大列边界值为大于等于所述列坐标最大值的最小的2的幂数。
  5. 如权利要求4所述的方法，其中，所述非零块由所述频率系数矩阵中的行坐标小于等于所述最大行边界值且列坐标小于等于所述最大列边界值的所有频率系数组成。
  6. 如权利要求4所述的方法，其中，基于所述变换子矩阵、所述非零块和所述转置子矩阵，确定所述频率系数矩阵的反变换结果的步骤为响应于确定以下至少一个条件来执行：
    所述最大行边界值小于所述频率系数矩阵的总行数；
    所述最大列边界值小于所述频率系数矩阵的总列数。
  7. 如权利要求1-6中任一项所述的方法，其中，基于所述变换子矩阵、所述非零块和所述转置子矩阵，确定所述频率系数矩阵的反变换结果包括：
    基于SIMD指令对所述转置子矩阵和所述非零块进行矩阵乘法运算，得到中间矩阵；以及
    基于SIMD指令对所述中间矩阵和所述变换子矩阵进行矩阵乘法运算，得到所述频率系数矩阵的反变换结果。
  8. 如权利要求7所述的方法，其中，通过所述矩阵乘法运算来计算所述中间矩阵的每相邻多行的元素包括：
    将所述转置子矩阵中的与所述中间矩阵的该相邻多行相对应的相邻多行划分为至少一个第一块；
    基于所述至少一个第一块，将所述非零块划分为至少一个第二块，其中，所述第一块的列数和所述第二块的行数相同；
    将每个第一块中的每一行元素复制多次后，按行重排，得到每个第一块对应的第一行矩阵，其中，每个第一块中的每一行元素的复制次数与每个第二块的列数相同；
    将每个第二块中的元素按列重排，得到每个第二块对应的第二行矩阵；
    对于每个第一块，基于SIMD指令，将该第一块的第一行矩阵分别和相应的每个第二块的第二行矩阵的位置对应的元素相乘，并且将所述第一行矩阵的与该第一块中每一行对应的多个相邻元素经过相乘所得到的结果进行求和，得到第三行矩阵；以及
    将位于同一行的一个或多个第一块各自相应的一个或多个第三行矩阵的位置对应的元素相加，得到所述中间矩阵的相应行。
  9. 如权利要求8所述的方法，其中，所述至少一个第二块沿行方向分布。
  10. 如权利要求8所述的方法，其中，所述至少一个第二块呈矩阵分布。
  11. 如权利要求9或10所述的方法，其中，将所述转置子矩阵中的与所述中间矩阵的该相邻多行相对应的相邻多行划分为至少一个第一块，将所述非零块划分为至少一个第二块包括：
    响应于确定所述非零块的行数大于等于2且列数大于等于4，将所述转置子矩阵的相邻多行划分为至少一个2*2的第一块，并且将所述非零块划分为至少一个2*4的第二块。
  12. 如权利要求9或10所述的方法，其中，将所述转置子矩阵中的与所述中间矩阵的该相邻多行相对应的相邻多行划分为至少一个第一块，将所述非零块划分为至少一个第二块包括：
    响应于确定所述非零块的行数大于等于2且列数等于2，将所述转置子矩阵的相邻多行划分为至少一个4*2的第一块，并且将所述非零块划分为至少一个2*2的第二块。
  13. 如权利要求1-12中任一项所述的方法，其中，所述方法应用于图像解码，所述目标图像为待解码的图像中的图像块；或者
    所述方法应用于视频编解码，所述目标图像为待编码的视频帧中的图像块或待解码的视频帧中的图像块。
  14. 一种图像处理装置，包括：
    获取单元，被配置为获取目标图像的频率系数矩阵，其中，所述频率系数矩阵是通过对所述目标图像进行频域变换来得到的，并且所述频率系数矩阵包括至少一个非零频率系数；
    非零块确定单元，被配置为基于所述至少一个非零频率系数的位置，确定所述频率系数矩阵中的非零块；
    矩阵确定单元，被配置为确定所述非零块相应的变换子矩阵和转置子矩阵；以及
    反变换单元，被配置为基于所述变换子矩阵、所述非零块和所述转置子矩阵，确定所述频率系数矩阵的反变换结果。
  15. 一种电子设备，包括：
    至少一个处理器；以及
    与所述至少一个处理器通信连接的存储器；其中
    所述存储器存储有计算机程序，所述计算机程序在被所述至少一个处理器执行时实现根据权利要求1-13中任一项所述的方法。
  16. 一种存储有计算机程序的非瞬时计算机可读存储介质，其中，所述计算机程序在被处理器执行时实现根据权利要求1-13中任一项所述的方法。
  17. 一种计算机程序产品，包括计算机程序，其中，所述计算机程序在被处理器执行时实现根据权利要求1-13中任一项所述的方法。
PCT/CN2022/098600 2021-08-04 2022-06-14 图像处理方法及装置、电子设备和存储介质 WO2023011005A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/294,053 US20250088666A1 (en) 2021-08-04 2022-06-14 Image processing method and apparatus, electronic device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110891796.4A CN113592966B (zh) 2021-08-04 2021-08-04 图像处理方法及装置、电子设备和存储介质
CN202110891796.4 2021-08-04

Publications (1)

Publication Number Publication Date
WO2023011005A1 true WO2023011005A1 (zh) 2023-02-09

Family

ID=78255030

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/098600 WO2023011005A1 (zh) 2021-08-04 2022-06-14 图像处理方法及装置、电子设备和存储介质

Country Status (3)

Country Link
US (1) US20250088666A1 (zh)
CN (1) CN113592966B (zh)
WO (1) WO2023011005A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113592966B (zh) * 2021-08-04 2024-04-30 上海哔哩哔哩科技有限公司 图像处理方法及装置、电子设备和存储介质
CN118227944A (zh) * 2024-03-21 2024-06-21 上海壁仞科技股份有限公司 数据处理方法、装置、电子设备和计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030081851A1 (en) * 2001-10-29 2003-05-01 Sadeh Yaron M. Method and apparatus for performing spatial-to- frequency domain transform
CN108574849A (zh) * 2017-11-21 2018-09-25 北京金山云网络技术有限公司 Dct逆变换方法、逆变换装置、电子设备及存储介质
CN111741302A (zh) * 2020-08-07 2020-10-02 腾讯科技（深圳）有限公司 数据处理方法、装置、计算机可读介质及电子设备
CN111953999A (zh) * 2020-07-31 2020-11-17 北京奇艺世纪科技有限公司 一种反变换方法及装置
CN113592966A (zh) * 2021-08-04 2021-11-02 上海哔哩哔哩科技有限公司 图像处理方法及装置、电子设备和存储介质

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20110045949A (ko) * 2009-10-28 2011-05-04 삼성전자주식회사 회전 변환을 이용한 영상 부호화, 복호화 방법 및 장치
CN103745443B (zh) * 2014-01-10 2017-06-13 北京优纳科技有限公司 提高图像质量的方法和设备
FR3038196A1 (fr) * 2015-06-29 2016-12-30 B<>Com Procede de codage d'une image numerique, procede de decodage, dispositifs et programmes d'ordinateurs associes
WO2018014301A1 (zh) * 2016-07-21 2018-01-25 华为技术有限公司 视频编码方法及装置
EP3806475B1 (en) * 2018-07-06 2023-05-10 LG Electronics, Inc. Transform-based image coding method and device
CN113207002B (zh) * 2021-02-21 2022-07-26 腾讯科技（深圳）有限公司 视频编解码方法、装置、计算机可读介质及电子设备
CN113068037B (zh) * 2021-03-17 2022-12-06 上海哔哩哔哩科技有限公司 用于样点自适应补偿的方法及装置、设备和介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030081851A1 (en) * 2001-10-29 2003-05-01 Sadeh Yaron M. Method and apparatus for performing spatial-to- frequency domain transform
CN108574849A (zh) * 2017-11-21 2018-09-25 北京金山云网络技术有限公司 Dct逆变换方法、逆变换装置、电子设备及存储介质
CN111953999A (zh) * 2020-07-31 2020-11-17 北京奇艺世纪科技有限公司 一种反变换方法及装置
CN111741302A (zh) * 2020-08-07 2020-10-02 腾讯科技（深圳）有限公司 数据处理方法、装置、计算机可读介质及电子设备
CN113592966A (zh) * 2021-08-04 2021-11-02 上海哔哩哔哩科技有限公司 图像处理方法及装置、电子设备和存储介质

Also Published As

Publication number Publication date
US20250088666A1 (en) 2025-03-13
CN113592966A (zh) 2021-11-02
CN113592966B (zh) 2024-04-30

Similar Documents

Publication Publication Date Title
RU2413983C2 (ru) Обратимое преобразование для сжатия двумерных данных с потерями и без потерь
CA2653692C (en) Reduction of errors during computation of inverse discrete cosine transform
JP4425561B2 (ja) イメージおよびビデオ符号化のための２−ｄ変換
JP5015285B2 (ja) 映像符号化のための低複雑性単一化変換
JP4924904B2 (ja) データフレームのシーケンスの効率的な符号化／復号
WO2023011005A1 (zh) 图像处理方法及装置、电子设备和存储介质
CN113873252B (zh) 图像处理方法及装置、电子设备和存储介质
JP7275354B2 (ja) 符号化のための方法、装置及びコンピュータプログラム
WO2022193916A1 (zh) 用于样点自适应补偿的方法及装置、设备和介质
CN113068036B (zh) 用于样点自适应补偿的方法及装置、设备和介质
RU2419855C2 (ru) Снижение ошибок в ходе вычисления обратного дискретного косинусного преобразования
CN102036075A (zh) 一种图像及数字视频编码及解码方法
US20050281331A1 (en) Method and system for dynamically configurable DCT/IDCT module in a wireless handset
JP4704333B2 (ja) 画像符号化装置および画像復号化装置、ならびにそれらで用いられる集積回路
JP2017041786A (ja) 直交変換処理装置、逆直交変換処理装置、符号化装置、復号装置、及びコンピュータプログラム
JP2017005503A (ja) 映像符号化装置、映像符号化方法及び映像符号化プログラム
CN105472395B (zh) 一种基于离散Krawtchouk正交多项式的图像无损压缩方法
CN116916034B (zh) 基于safd的图像处理方法、装置、设备及存储介质
WO2023124602A1 (zh) 数据变换与恢复方法、装置、系统、电子设备及存储介质
WO2023136811A1 (en) Fixed-point optimizer for linear operations
CN116095310A (zh) 图像编解码方法、装置、电子设备及存储介质
Li et al. A highly efficient reconfigurable architecture of inverse transform for multiple video standards

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22851721

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18294053

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22851721

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 22851721

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 24.09.2024)