KR20230137171A - Image encoding/decoding method and apparatus based on object detection information, and recording medium storing bitstream - Google Patents

Abstract

An image encoding/decoding method, device, and recording medium for storing a bitstream generated by the image encoding method are provided. An image decoding method according to the present disclosure includes obtaining object recognition information for an input image, and restoring the input image based on the object recognition information, wherein the object recognition information is included in the input image. When it includes first object recognition information for a first object and second object recognition information for a second object in the input image, the second object recognition information will be derived as difference information with the first object recognition information. You can.

Description

Image encoding/decoding method based on object recognition information, device, and recording medium storing bitstream {IMAGE ENCODING/DECODING METHOD AND APPARATUS BASED ON OBJECT DETECTION INFORMATION, AND RECORDING MEDIUM STORING BITSTREAM}

The present invention relates to an image encoding/decoding method, a device, and a recording medium storing a bitstream. More specifically, it relates to an image encoding/decoding method based on object recognition information, a device, and a recording medium storing a bitstream.

Recently, demand for high-resolution, high-quality video, such as HD (High Definition) video and UHD (Ultra High Definition) video, is increasing in various fields. Meanwhile, as object recognition technology using machine learning is rapidly developing, it is expected that image compression efficiency can be dramatically improved by using data obtained through object recognition algorithms for image encoding/decoding. Accordingly, various research and development efforts are being actively conducted to combine object recognition technology with image compression technology.

The purpose of the present disclosure is to provide a video encoding/decoding method and device with improved encoding/decoding efficiency.

Additionally, the present disclosure aims to provide an image encoding/decoding method and device based on object recognition information.

Additionally, the present disclosure aims to provide an image encoding/decoding method and device having a compression unit reconstructed based on object recognition information.

Additionally, the present disclosure aims to provide an image encoding/decoding method and device with an adaptive bit rate based on object recognition information.

Additionally, the present disclosure aims to provide an image encoding/decoding method and device that performs prediction only on the area containing a recognized object in the image.

Additionally, the present disclosure aims to provide an image encoding/decoding method and device for encoding/decoding object recognition information together with image information.

Additionally, the present disclosure aims to provide an image encoding/decoding method and device that encodes/decodes object recognition information separately from image information.

Additionally, the present disclosure aims to provide a computer-readable recording medium that stores a bitstream generated by the video encoding method or device according to the present disclosure.

This patent was carried out as a project supported by the Ministry of Trade, Industry and Energy (Project name: Development of VVC (Vesatile Video Codec) decoder IP for 5G-linked ultra-realistic media, Project identification number: 1415172976, Project number: 20010342).

An image decoding method according to an aspect of the present disclosure includes obtaining object recognition information for an input image, and restoring the input image based on the object recognition information, wherein the object recognition information is included in the input image. When it includes first object recognition information for a first object in an image and second object recognition information for a second object in the input image, the second object recognition information is difference information with the first object recognition information. can be obtained.

An image encoding method according to another aspect of the present disclosure includes obtaining object recognition information in an input image based on an object recognition algorithm, and encoding the input image based on the object recognition information, wherein the object When the recognition information includes first object recognition information for the first object in the input image and second object recognition information for the second object in the input image, the second object recognition information is the first object recognition information. It can be encoded as difference information with .

A computer-readable recording medium according to another aspect of the present disclosure can store a bitstream generated by the video encoding method or video encoding device of the present disclosure.

The features briefly summarized above with respect to the present disclosure are merely exemplary aspects of the detailed description of the present disclosure described below, and do not limit the scope of the present disclosure.

According to the present disclosure, an image encoding/decoding method and device with improved encoding/decoding efficiency can be provided.

Additionally, according to the present disclosure, an image encoding/decoding method and device based on object recognition information can be provided.

Additionally, according to the present disclosure, an image encoding/decoding method and device having a compression unit reconstructed based on object recognition information can be provided.

Additionally, according to the present disclosure, an image encoding/decoding method and device having an adaptive bit rate based on object recognition information can be provided.

Additionally, according to the present disclosure, an image encoding/decoding method and device that performs prediction only for a region containing a recognized object in an image can be provided.

Additionally, according to the present disclosure, an image encoding/decoding method and device for encoding/decoding object recognition information together with image information can be provided.

Additionally, according to the present disclosure, an image encoding/decoding method and device can be provided that encodes/decodes object recognition information separately from image information.

Additionally, according to the present disclosure, a computer-readable recording medium storing a bitstream generated by the image encoding method or device according to the present disclosure may be provided.

1 is a block diagram showing a video encoding device.
Figure 2 is a block diagram showing a video decoding device.
3 to 10 are diagrams for explaining a video encoder and a video decoder according to embodiments of the present invention.
11 to 12B are diagrams for explaining a method of reconstructing a compression unit according to an embodiment of the present invention.
Figure 13 is a diagram for explaining inter-screen prediction according to an embodiment of the present invention.
14 and 15 are diagrams showing examples of object classification tables that can be applied to embodiments of the present invention.
Figure 16 is a flowchart showing an image encoding method according to an embodiment of the present invention.
Figure 17 is a flowchart showing an image decoding method according to an embodiment of the present invention.
Figure 18 is a block diagram schematically showing an electronic device including an image encoding/decoding device according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. Similar reference numbers in the drawings refer to identical or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. For a detailed description of the exemplary embodiments described below, refer to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description that follows is not to be taken in a limiting sense, and the scope of the exemplary embodiments is limited only by the appended claims, together with all equivalents to what those claims assert if properly described.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

When a component of the present invention is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in between. It must be understood that it may be possible. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

The components appearing in the embodiments of the present invention are shown independently to represent different characteristic functions, and do not mean that each component is comprised of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two of each component can be combined to form one component, or one component can be divided into a plurality of components to perform a function, and each of these components can perform a function. Integrated embodiments and separate embodiments of the constituent parts are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

The terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In the present invention, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof. In other words, the description of “including” a specific configuration in the present invention does not exclude configurations other than the configuration, and means that additional configurations may be included in the practice of the present invention or the scope of the technical idea of the present invention.

Some of the components of the present invention may not be essential components that perform essential functions in the present invention, but may simply be optional components to improve performance. The present invention can be implemented by including only essential components for implementing the essence of the present invention excluding components used only to improve performance, and a structure including only essential components excluding optional components used only to improve performance. is also included in the scope of rights of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing the embodiments of the present specification, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description will be omitted, and the same reference numerals will be used for the same components in the drawings. Redundant descriptions of the same components are omitted.

1 is a block diagram showing a video encoding device.

[Figure 1] Video encoding device

Referring to FIG. 1, the image encoding device 100 includes a picture segmentation unit 110, prediction units 120 and 125, a transformation unit 130, a quantization unit 135, a rearrangement unit 160, and an entropy encoding unit ( 165), an inverse quantization unit 140, an inverse transform unit 145, a filter unit 150, and a memory 155.

Each component shown in FIG. 1 is shown independently to represent different characteristic functions in the video encoding device, and does not mean that each component is comprised of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two of each component can be combined to form one component, or one component can be divided into a plurality of components to perform a function, and each of these components can be divided into a plurality of components. Integrated embodiments and separate embodiments of the constituent parts are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

Additionally, some components may not be essential components that perform essential functions in the present invention, but may simply be optional components to improve performance. The present invention can be implemented by including only essential components for implementing the essence of the present invention excluding components used only to improve performance, and a structure including only essential components excluding optional components used only to improve performance. is also included in the scope of rights of the present invention.

The picture division unit 110 may divide the input picture into at least one processing unit. At this time, the processing unit may be a prediction unit (PU), a transformation unit (TU), or a coding unit (CU). The picture division unit 110 divides one picture into a combination of a plurality of coding units, prediction units, and transformation units, and combines one coding unit, prediction unit, and transformation unit based on a predetermined standard (for example, a cost function). You can encode the picture by selecting .

For example, one picture may be divided into a plurality of coding units. To split the coding unit in a picture, a recursive tree structure such as the Quad Tree Structure can be used. Coding that is split into other coding units with one image or the largest coding unit as the root. A unit can be divided into child nodes equal to the number of divided coding units. A coding unit that is no longer divided according to certain restrictions becomes a leaf node. That is, assuming that only square division is possible for one coding unit, one coding unit can be divided into up to four different coding units.

Hereinafter, in the embodiments of the present invention, the coding unit may be used to mean a unit that performs encoding, or may be used to mean a unit that performs decoding.

A prediction unit may be divided into at least one square or rectangular shape of the same size within one coding unit, and any one of the prediction units divided within one coding unit may be a prediction unit of another prediction unit. It may be divided to have a different shape and/or size than the unit.

If the prediction unit for which intra prediction is performed based on the coding unit is not the minimum coding unit when generated, intra prediction can be performed without dividing the prediction unit into a plurality of prediction units NxN.

The prediction units 120 and 125 may include an inter prediction unit 120 that performs inter prediction and an intra prediction unit 125 that performs intra prediction. It is possible to determine whether to use inter prediction or intra prediction for a prediction unit, and determine specific information (eg, intra prediction mode, motion vector, reference picture, etc.) according to each prediction method. At this time, the processing unit in which the prediction is performed and the processing unit in which the prediction method and specific contents are determined may be different. For example, the prediction method and prediction mode are determined in prediction units, and prediction may be performed in transformation units. The residual value (residual block) between the generated prediction block and the original block may be input to the conversion unit 130. Additionally, prediction mode information, motion vector information, etc. used for prediction may be encoded in the entropy encoder 165 together with the residual value and transmitted to the decoder. When using a specific encoding mode, it is possible to encode the original block as is and transmit it to the decoder without generating a prediction block through the prediction units 120 and 125.

The inter prediction unit 120 may predict a prediction unit based on information on at least one picture among the pictures before or after the current picture, and in some cases, prediction based on information on a partially encoded region within the current picture. Units can also be predicted. The inter prediction unit 120 may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.

The reference picture interpolator may receive reference picture information from the memory 155 and generate pixel information of an integer number of pixels or less from the reference picture. In the case of luminance pixels, a DCT-based 8-tap interpolation filter with different filter coefficients can be used to generate pixel information of an integer pixel or less in 1/4 pixel units. In the case of color difference signals, a DCT-based 4-tap interpolation filter with different filter coefficients can be used to generate pixel information of an integer pixel or less in 1/8 pixel units.

The motion prediction unit may perform motion prediction based on the reference picture interpolated by the reference picture interpolation unit. Various methods such as FBMA (Full search-based Block Matching Algorithm), TSS (Three Step Search), and NTS (New Three-Step Search Algorithm) can be used to calculate the motion vector. The motion vector may have a motion vector value in units of 1/2 or 1/4 pixels based on the interpolated pixels. The motion prediction unit can predict the current prediction unit by using a different motion prediction method. As a motion prediction method, various methods such as the skip method, the merge method, the Advanced Motion Vector Prediction (AMVP) method, and the intra block copy method can be used.

The intra prediction unit 125 may generate a prediction unit based on reference pixel information around the current block, which is pixel information in the current picture. If the neighboring block of the current prediction unit is a block on which inter prediction has been performed and the reference pixel is a pixel on which inter prediction has been performed, the reference pixel included in the block on which inter prediction has been performed is the reference pixel of the block on which intra prediction has been performed. It can be used in place of information. That is, when a reference pixel is not available, the unavailable reference pixel information can be replaced with at least one reference pixel among available reference pixels.

In intra prediction, the prediction mode can include a directional prediction mode that uses reference pixel information according to the prediction direction and a non-directional mode that does not use directional information when performing prediction. The number of directional prediction modes may be equal to or more than 33 defined in the HEVC standard, and may be expanded to a number in the range of 60 to 70, for example. The mode for predicting luminance information and the mode for predicting chrominance information may be different, and intra prediction mode information used to predict luminance information or predicted luminance signal information may be used to predict chrominance information.

When performing intra prediction, if the size of the prediction unit and the size of the transformation unit are the same, intra prediction for the prediction unit is made based on the pixel on the left, the pixel on the top left, and the pixel on the top of the prediction unit. can be performed. However, when performing intra prediction, if the size of the prediction unit and the size of the transformation unit are different, intra prediction can be performed using a reference pixel based on the transformation unit. Additionally, intra prediction using N x N partitioning can be used only for the minimum coding unit.

The intra prediction method can generate a prediction block after applying an Adaptive Intra Smoothing (AIS) filter to the reference pixel according to the prediction mode. The type of AIS filter applied to the reference pixel may be different. To perform the intra prediction method, the intra prediction mode of the current prediction unit can be predicted from the intra prediction mode of prediction units existing around the current prediction unit. When predicting the prediction mode of the current prediction unit using predicted mode information from neighboring prediction units, if the intra prediction mode of the current prediction unit and neighboring prediction units are the same, predetermined flag information is used to predict the current prediction unit and neighboring prediction units. Information that the prediction modes of are the same can be transmitted, and if the prediction modes of the current prediction unit and neighboring prediction units are different, entropy encoding can be performed to encode the prediction mode information of the current block.

Additionally, based on the prediction units generated by the prediction units 120 and 125, a residual block may be generated that includes residual information that is the difference between the prediction unit and the original block of the prediction unit. The generated residual block may be input to the conversion unit 130.

The transform unit 130 transforms the residual block, including the original block and the residual value information of the prediction unit generated through the prediction units 120 and 125, into DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT and It can be converted using the same conversion method. Whether to apply DCT, DST, or KLT to transform the residual block can be determined based on intra prediction mode information of the prediction unit used to generate the residual block.

The quantization unit 135 may quantize the values converted to the frequency domain by the conversion unit 130. The quantization coefficient may change depending on the block or the importance of the image. The value calculated by the quantization unit 135 may be provided to the inverse quantization unit 140 and the realignment unit 160.

The rearrangement unit 160 may rearrange coefficient values for the quantized residual values.

The rearrangement unit 160 can change the coefficients in a two-dimensional block form into a one-dimensional vector form through a coefficient scanning method. For example, the realignment unit 160 can scan from DC coefficients to coefficients in the high frequency region using a zig-zag scan method and change it into a one-dimensional vector form. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional block-type coefficients in the column direction or a horizontal scan that scans the two-dimensional block-type coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, it can be determined which scan method among zig-zag scan, vertical scan, and horizontal scan will be used.

The entropy encoding unit 165 may perform entropy encoding based on the values calculated by the reordering unit 160. Entropy coding can use various coding methods, such as Exponential Golomb, Context-Adaptive Variable Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC).

The entropy encoding unit 165 receives the residual value coefficient information and block type information of the coding unit, prediction mode information, division unit information, prediction unit information and transmission unit information, and motion information from the reordering unit 160 and the prediction units 120 and 125. Various information such as vector information, reference frame information, block interpolation information, and filtering information can be encoded.

The entropy encoding unit 165 may entropy encode the coefficient value of the coding unit input from the reordering unit 160.

The inverse quantization unit 140 and the inverse transformation unit 145 inversely quantize the values quantized in the quantization unit 135 and inversely transform the values transformed in the transformation unit 130. The residual value generated in the inverse quantization unit 140 and the inverse transform unit 145 is restored by combining the prediction units predicted through the motion estimation unit, motion compensation unit, and intra prediction unit included in the prediction units 120 and 125. You can create a block (Reconstructed Block).

The filter unit 150 may include at least one of a deblocking filter, an offset correction unit, and an adaptive loop filter (ALF).

The deblocking filter can remove block distortion caused by boundaries between blocks in the restored picture. To determine whether to perform deblocking, it is possible to determine whether to apply a deblocking filter to the current block based on the pixels included in several columns or rows included in the block. When applying a deblocking filter to a block, a strong filter or a weak filter can be applied depending on the required deblocking filtering strength. Additionally, when applying a deblocking filter, horizontal filtering and vertical filtering can be processed in parallel when vertical filtering and horizontal filtering are performed.

The offset correction unit may correct the offset of the deblocked image from the original image in pixel units. In order to perform offset correction for a specific picture, the pixels included in the image are divided into a certain number of areas, then the area to perform offset is determined and the offset is applied to that area, or the offset is performed by considering the edge information of each pixel. You can use the method of applying .

Adaptive Loop Filtering (ALF) can be performed based on a comparison between the filtered restored image and the original image. After dividing the pixels included in the image into predetermined groups, filtering can be performed differentially for each group by determining one filter to be applied to that group. Information related to whether to apply ALF may be transmitted for each coding unit (CU), and the shape and filter coefficients of the ALF filter to be applied may vary for each block. Additionally, an ALF filter of the same type (fixed type) may be applied regardless of the characteristics of the block to which it is applied.

The memory 155 may store a reconstructed block or picture calculated through the filter unit 150, and the stored reconstructed block or picture may be provided to the prediction units 120 and 125 when inter prediction is performed.

Recently, demand for high-resolution, high-quality images such as HD (High Definition) images and UHD (Ultra High Definition) images is increasing in various application fields. Likewise, as object recognition technology using machine learning develops, the demand for developing technology with image compression is increasing. Therefore, based on data obtained through an algorithm that applies object recognition, high-resolution and high-quality image compression can be achieved.

[1] can use the data obtained by applying an object recognition algorithm for object-based image compression. By using data obtained through an object recognition algorithm, a unit that can divide the image according to the number or type of objects, such as a picture/slice/tile/tile group/CTU/CTU row/CTU column unit or object-based compression unit You can compress the video by adding or replacing . In addition, we present a method to efficiently compress images based on the obtained data. Related data obtained from the object recognition algorithm can be compressed and transmitted or stored through entropy encoding/decoding.

[2] The object recognition algorithm and image compression standardization technology can be used independently. Unlike [1], the information obtained through the object recognition algorithm and the bitstream of image compression standardization are transmitted separately. As a result, data obtained through object recognition algorithms in image compression standardization technology can be shared and compression performed.

In the prior art, there is no image compression technology using object recognition. Additionally, as object recognition technology becomes more concise, development of technologies that can be utilized, such as image compression technology, is necessary.

The purpose of the present invention is to use image compression technology together with the application of an object recognition algorithm. Based on the data obtained from the object recognition algorithm, compression of objects can be performed efficiently.

The present invention can provide effective image quality through a combination of data obtained through an object recognition algorithm and image compression.

The present invention provides data obtained through an object recognition algorithm, such as x-axis and y-axis information of the recognized object, width of the recognized object, height of the recognized object, type of object, recognized accuracy of the object, and object identification number. It is a video encoding/decoding method and device characterized by a method of improving video quality by using a video encoding/decoding method and device.

The present invention adds (includes) a unit of picture/slice/tile/tile group/CTU/CTU row/CTU column or object-based compression unit (Object-based Coding Unit) in compression unit reconstruction based on data obtained through an object recognition algorithm. Or, a video encoding/decoding method and device characterized by providing a replacement method.

The present invention is an encoding/decoding method and device that combines an object recognition algorithm and image compression technology to provide a method for efficiently compressing images for a recognized object area.

The present invention provides a method of adding (including) or replacing existing picture/slice/tile/tile group/CTU/CTU row/CTU column units or object-based compression units, and breaks the mold of existing video compression methods. Provides an encoding/decoding method and device that can

Figure 2 is a block diagram showing a video decoding device.

[Figure 2] Video decoding device

Referring to FIG. 2, the image decoding device 200 includes an entropy decoding unit 210, a reordering unit 215, an inverse quantization unit 220, an inverse transform unit 225, a prediction unit 230, 235, and a filter unit ( 240) and memory 245 may be included.

When a video bitstream is input from a video encoding device, the input bitstream can be decoded in a procedure opposite to that of the video encoding device.

The entropy decoding unit 210 may perform entropy decoding in a procedure opposite to the procedure in which entropy encoding is performed in the entropy encoding unit of the video encoding device. For example, various methods such as Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding) may be applied in response to the method performed in the image encoding device.

The entropy decoder 210 can decode information related to intra prediction and inter prediction performed by the encoder.

The reordering unit 215 may rearrange the bitstream entropy-decoded by the entropy decoding unit 210 based on the method in which the encoder rearranges the bitstream. Coefficients expressed in the form of a one-dimensional vector can be restored and rearranged as coefficients in the form of a two-dimensional block. The reordering unit 215 may receive information related to coefficient scanning performed by the encoder and perform reordering by reverse scanning based on the scanning order performed by the encoder.

The inverse quantization unit 220 may perform inverse quantization based on the quantization parameters provided by the encoder and the coefficient values of the rearranged blocks.

The inverse transform unit 225 may perform inverse transform, that is, inverse DCT, inverse DST, and inverse KLT, on the transform performed by the transformer, that is, DCT, DST, and KLT, on the quantization result performed by the image encoding device. Inverse transformation may be performed based on the transmission unit determined by the video encoding device. The inverse transform unit 225 of the video decoding device may selectively perform a transformation technique (eg, DCT, DST, KLT) according to a plurality of information such as a prediction method, the size of the current block, and the prediction direction.

The prediction units 230 and 235 may generate a prediction block based on prediction block generation-related information provided by the entropy decoder 210 and previously decoded block or picture information provided by the memory 245.

As described above, when performing intra prediction in the same manner as the operation in the video encoding device, when the size of the prediction unit and the size of the transformation unit are the same, the pixel existing on the left of the prediction unit, the pixel existing on the upper left, and the size of the transformation unit are the same. Intra prediction of the prediction unit is performed based on existing pixels, but when performing intra prediction, if the size of the prediction unit and the size of the transformation unit are different, intra prediction is performed using a reference pixel based on the transformation unit. can do. Additionally, intra prediction using N x N partitioning only for the minimum coding unit can be used.

The prediction units 230 and 235 may include a prediction unit determination unit, an inter prediction unit, and an intra prediction unit. The prediction unit discriminator receives various information such as prediction unit information input from the entropy decoder 210, prediction mode information of the intra prediction method, and motion prediction-related information of the inter prediction method, distinguishes the prediction unit from the current coding unit, and makes predictions. It is possible to determine whether a unit performs inter-prediction or intra-prediction. The inter prediction unit 230 uses the information required for inter prediction of the current prediction unit provided by the video encoding device to determine the current prediction unit based on information included in at least one of the pictures before or after the current picture including the current prediction unit. Inter prediction can be performed on prediction units. Alternatively, inter prediction may be performed based on information on a pre-restored partial region within the current picture including the current prediction unit.

To perform inter prediction, based on the coding unit, the motion prediction method of the prediction unit included in the coding unit is Skip Mode, Merge Mode, AMVP Mode, and Intra Block Copy Mode. You can judge whether it is a certain method or not.

The intra prediction unit 235 may generate a prediction block based on pixel information in the current picture. If the prediction unit is a prediction unit that has performed intra prediction, intra prediction may be performed based on intra prediction mode information of the prediction unit provided by the video encoding device. The intra prediction unit 235 may include an Adaptive Intra Smoothing (AIS) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a part that performs filtering on the reference pixels of the current block, and can be applied by determining whether or not to apply the filter according to the prediction mode of the current prediction unit. AIS filtering can be performed on the reference pixel of the current block using the prediction mode and AIS filter information of the prediction unit provided by the video encoding device. If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied.

If the prediction mode of the prediction unit is a prediction unit that performs intra prediction based on pixel values by interpolating the reference pixel, the reference pixel interpolator may interpolate the reference pixel to generate a reference pixel in pixel units of an integer value or less. If the prediction mode of the current prediction unit is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is DC mode.

The restored block or picture may be provided to the filter unit 240. The filter unit 240 may include a deblocking filter, an offset correction unit, and an ALF.

Information on whether a deblocking filter has been applied to the corresponding block or picture can be provided from the video encoding device, and when a deblocking filter has been applied, information on whether a strong filter or a weak filter has been applied. The deblocking filter of the video decoding device receives information related to the deblocking filter provided by the video encoding device, and the video decoding device can perform deblocking filtering on the corresponding block.

The offset correction unit may perform offset correction on the reconstructed image based on the type of offset correction applied to the image during encoding and offset value information.

ALF can be applied to the coding unit based on ALF application availability information, ALF coefficient information, etc. provided from the encoder. This ALF information may be included and provided in a specific parameter set.

The memory 245 can store the restored picture or block so that it can be used as a reference picture or reference block, and can also provide the restored picture to an output unit.

As described above, hereinafter, in the embodiments of the present invention, the term coding unit is used as a coding unit for convenience of explanation, but it may also be a unit that performs not only encoding but also decoding.

Video encoder/decoder - 1-1 structure

Figure 3 is a diagram for explaining a video encoder according to an embodiment of the present invention.

[Figure 3] Video encoder

Referring to FIG. 3, the image encoder 300 may encode an input image based on object recognition information obtained from the input image. The video encoder 300 may also be referred to as a video encoding device, and may have the same/similar structure as the video encoding device 100 described above with reference to FIG. 1.

Object recognition information may be generated by an object recognizer (A).

Specifically, the object recognizer A can recognize objects present in the input image by applying a predetermined object recognition algorithm to the input image. Depending on the embodiment, the object recognition algorithm may be designed to recognize only specific types of objects in the input image.

The object recognizer (A) can generate object recognition information by converting the object recognition result into data. Object recognition information may include the coordinates, width and height of the recognized object, object type, recognition accuracy, and object identification number. Here, the coordinates of the recognized object may represent any one of the center coordinates, upper-left corner coordinates, upper-right corner coordinates, lower-left corner coordinates, and lower-right corner coordinates of the object within the input image.

The coordinates, width, and height of a recognized object can be expressed as floating points. The value expressed as a floating point can be converted into an integer by multiplying it by the width or height of the input image. For example, the x-axis coordinate and width of the object can be converted into integers by multiplying them by the width of the image. Additionally, the y-axis coordinate and height of the object can be converted into integers by multiplying them by the height of the image. Through this integer representation, the encoding/decoding efficiency of object recognition information can be further improved. Depending on the embodiment, the value expressed as a floating point may be encoded as is without being converted to an integer.

Meanwhile, if the x-axis coordinate, y-axis coordinate, width, and height of the recognized object are individually transmitted, coding efficiency may decrease and signaling overhead may increase. In order to solve this problem, in one embodiment, the object recognizer (A) uses the previous encoding object (hereinafter, referred to as 'first object') for the first and subsequent encoding object in the encoding order (hereinafter, referred to as 'second object'). Object recognition information can be generated by differentiating it from the object recognition information for the object (referred to as 'object'). For example, assume that the coordinate values of the first object are (42, 118), the width and height are 50 and 30, respectively, the object type/class is 2 (e.g. vehicle), and the object identification number is 0. Also, assume that the coordinates of the second object are (70, 220), the width and height are 40 and 28, respectively, the object type/class is 2 (e.g. vehicle), and the object identification number is 1. In this case, the object recognizer A can generate first object recognition information by encoding the object information as is for the first object. On the other hand, the object recognizer A may generate second object recognition information for the second object as a difference from the first object recognition information. For example, if you calculate the difference value of the x-axis coordinate between the second object and the first object, you get 70-42=38. In the same way, if you calculate the difference value of the y-axis coordinate, you get 220-108=102. Also, if you calculate the difference values for height and width, you get -10 and -2. The difference value of the object type/class is 0. In order to encode each calculated difference value, the object recognizer A may encode a flag indicating the sign of the difference value and the absolute value of the difference value, respectively.

Even when there are three or more objects recognized in the input image, the above-described method can be applied. For example, the object recognizer A may generate third object recognition information as a difference between the second object recognition information and the third object.

The image encoder 100 may encode an input image based on object recognition information generated by the object recognizer (A).

Specifically, the image encoder 100 may reconstruct the compression unit based on object recognition information (310). The process determines whether to reconstruct the first compression unit of the input image, and when it is decided to reconstruct the first compression unit, the first compression unit is converted into a predetermined second compression unit using the object recognition information. This can be done by reconstructing. Here, the first compression unit is a unit used in existing video compression standards and may include picture/slice/tile/tile group/CTU/CTU row/CTU column units. On the other hand, the second compression unit is a new unit rather than a unit used in existing video compression standards, and may mean an object-based coding unit. In one embodiment, the second compression unit may include slice/tile/tile group/CTU/CTU row/CTU column as a higher-order unit of the picture. Alternatively, the second compression unit is a higher level unit of the slice and may include a tile/tile group/CTU/CTU row/CTU column. Alternatively, the second compression unit is a higher level unit of a tile or tile group and may include a CTU/CTU row/CTU column. Alternatively, the second compression unit may include a CTU column as a higher-order unit of the CTU row/CTU column. Specific details of the compression unit reconstruction 310 will be described in detail later.

Meanwhile, in FIG. 3, the object recognition information generated by the object recognizer (A) is shown as being transmitted only through the compression unit reconstruction process 310 of the image encoder 300, but this is only an example and does not apply to the implementation of the present invention. The examples are not limited to this. For example, the image encoder 300 may store the object recognition information received from the object recognizer A in a buffer and use it for each operation of image encoding.

The image encoder 300 can control the bit rate of the input image based on object recognition information (320). Bit rate control means controlling the amount of data of each picture that makes up an image. Through bit rate control, video transmission delay can be eliminated and excessive data amounts can be prevented from being suddenly transmitted.

In one embodiment, the bit rate of the input image may be determined in a predetermined unit, for example, a first compression unit or a second compression unit, based on object recognition information. Here, the first compression unit may include a group of picture (GoP) unit, a picture unit, and a coding unit (CU) unit. Additionally, the second compression unit may include a new unit reconstructed from the first compression unit.

The bit rate can be determined individually for each area in the input image where an object exists, and can have different values depending on the importance of each object. For example, the bit rate of a first area containing a recognized object in an input image may be higher than the bit rate of a second area not containing the recognized object. By controlling the bit rate of the first area to be higher than the bit rate of the second area, the area where the object exists in the input image can be compressed with higher resolution. Specific details of the bit rate control 320 will be described in detail later.

The image encoder 300 may perform intra-screen/inter-screen prediction on the input image based on object recognition information (330).

Specifically, the image encoder 300 may perform motion estimation/compensation based on object recognition information for an area where an object exists in the input image. In one embodiment, the motion prediction/compensation may be performed only for areas where objects exist in the input image. The basic contents of intra-screen/inter-screen prediction are as described above with reference to FIG. 1, and the specific contents of intra-screen/inter-screen prediction 330 based on object recognition information will be described in detail later.

The image encoder 300 may perform transformation/quantization based on object recognition information (340).

Specifically, the image encoder 300 distinguishes between an area where an object exists and an area where an object does not exist in the input image based on object recognition information, and based on quantization parameters determined differently depending on whether an object exists. Transformation/quantization can be performed. In one embodiment, the quantization parameter of the first area containing the recognized object in the input image may be adjusted to be smaller than the quantization parameter of the second area not containing the recognized object. By adjusting the quantization parameter of the first region to a smaller size, a larger high-frequency region within the input image can be used, making it possible to realize high-quality, high-resolution image compression. The basic contents of transformation/quantization are as described above with reference to FIG. 1, and the specific details of transformation/quantization 340 based on object recognition information will be described in detail later.

The image encoder 300 may perform entropy encoding based on object recognition information (350).

Specifically, the image encoder 300 may generate a bitstream by entropy encoding input image information (eg, transformed and quantized residual information) based on object recognition information. In one embodiment, the bitstream may further include object recognition information. The basic contents of entropy encoding are as described above with reference to FIG. 1, and the specific contents of entropy encoding 350 based on object recognition information will be described in detail later.

Meanwhile, the image encoder 300 can restore the input image by performing inverse transformation/inverse quantization 360 on the transformed/quantized input image information and then adding it with prediction information generated by intra-screen/inter-screen prediction. there is. The restored image can be in-loop filtered (370) and then used for inter-screen prediction (330) for the next encoding target image. The basic contents of the inverse transformation/inverse quantization 360 and the in-loop filtering 370 are the same as described above with reference to FIG. 1, and detailed description thereof will be omitted.

Figure 4 is a diagram for explaining a video decoder according to an embodiment of the present invention.

[Figure 4] Video decoder

The video decoder 400 may also be referred to as a video decoding device. The video decoder 400 corresponds to the video encoder 300 of FIG. 3 and may have the same/similar structure as the video decoding device 200 described above with reference to FIG. 2.

Referring to FIG. 4, the image decoder 400 can decode an image based on object recognition information.

Specifically, the image decoder 400 may perform entropy decoding on the received bitstream to obtain image information and object recognition information (410). The bitstream may have a single bitstream structure including image information (eg, residual information and various encoding information) and object recognition information. In one embodiment, the object recognition information may be encoded as a difference from the object recognition information of a previous encoding target object.

The video decoder 400 may perform intra-screen/inter-screen prediction 420 and inverse transformation/inverse quantization 430, respectively, based on the received information. Additionally, the image decoder 400 can restore the image by adding the predicted image and the inversely transformed/inversely quantized image. After going through in-loop filtering 440, the restored image can be used for intra-screen/inter-screen prediction of the next image to be decoded, and can be output through a display device such as a display. The basic contents of the entropy decoding 410 to the in-loop filtering 440 are the same as described above with reference to FIG. 1, and detailed description thereof will be omitted.

Meanwhile, entropy decoding 410 to inverse transformation/inverse quantization 430 may be performed based on object recognition information. The specific details are the same as described above with reference to FIG. 3. For example, inter-screen prediction 420 may be performed only on areas where objects exist in the image.

The video encoder 300 and the video decoder 400 described above with reference to FIGS. 3 and 4 can adaptively encode/decode an image based on object recognition information. Accordingly, image compression efficiency is dramatically improved, making it possible to encode/decode high-quality, high-resolution images more effectively.

Video encoder/decoder - Structure 1-2

Figure 5 is a diagram for explaining a video encoder according to another embodiment of the present invention.

Figure 6 is a diagram for explaining a video decoder according to another embodiment of the present invention.

[Figure 5] Video encoder

[Figure 6] Video decoder

Figure 5 shows a video encoder, and Figure 6 shows a video decoder.

Figure 5 shows [1]. At this time, [1] shares information obtained through an image encoder and an object recognition algorithm, performs compression, and transmits it through a single bitstream.

[1] In Figure 5, input images are received from two directions. At this time, the first performs compression based on information obtained through the application of the [E1] object recognition algorithm, and the second performs the process of encoding the image through [E3] compression unit reconstruction. At this time, [E2] bit rate control is performed based on the information obtained from the application of the [E1] object recognition algorithm, the existing intra/inter-screen [E4] prediction is performed, [E5] conversion and quantization, and [E6] After performing inverse transformation, inverse quantization, in-loop filter, and [E7] entropy coding, the bitstream is signaled.

This is a method of decoding what was encoded in FIG. 5 in FIG. 6. At this time, in Figure 5, an image decoding process can be performed by compressing information obtained through an object recognition algorithm and then decoding one bitstream.

Figure 6 receives the bitstream transmitted in Figure 5, performs [D1] entropy decoding, and passes the image obtained through intra/inter-screen [D3] prediction and [D2] inverse transformation and inverse quantization to a restored image through an in-loop filter. You will have

Video encoder/decoder - second structure

Figure 7 is a diagram for explaining a video encoder according to another embodiment of the present invention.

Figure 8 is a diagram for explaining a video decoder according to another embodiment of the present invention.

[Figure 7] Video encoder

[Figure 8] Video decoder

Figure 7 shows a video encoder, and Figure 8 shows a video decoder.

Figure 7 shows [2]. At this time, [2] is used independently by the image encoder and object recognition algorithm, so it can be transmitted through each bitstream.

[2] In Figure 7, input images are received from two directions. At this time, the object recognition encoder can transmit the information obtained through the application of the [E1] object recognition algorithm as a background bitstream. The video encoder controls the [E2] bit rate, performs existing intra/inter-screen [E4] prediction, [E5] transformation and quantization, [E6] inverse transformation and inverse quantization, in-loop filter, [E7] ] After entropy encoding is performed, the bitstream is signaled.

This is a method of decoding what was encoded in FIG. 7 in FIG. 8. At this time, in Figure 8, information obtained through an object recognition algorithm and each bitstream are retrieved from the image encoder. The object recognition encoder may encode and have the x-axis and y-axis information of the recognized object, the width of the recognized object, the height of the recognized object, the type of object, the recognized accuracy of the object, the object identification number, etc. At this time, the image encoder can generate a restored image using each necessary information.

Figure 8 receives the bitstream transmitted in Figure 7, performs [D1] entropy decoding, and passes the image obtained through intra/inter-screen [D3] prediction and [D2] inverse transformation and inverse quantization to a restored image through an in-loop filter. You will have The object recognition information is decoded in the object recognition decoder of FIG. 8 and used in the video decoder.

Video encoder/decoder - 3rd structure

Figure 9 is a diagram for explaining an image encoder and an object recognition layer according to another embodiment of the present invention.

Figure 10 is a diagram for explaining an image decoder and an object recognition layer according to another embodiment of the present invention.

[Figure 9] Video encoder

[Figure 10] Video decoder

Referring to FIGS. 9 and 10 together, the object recognition layer allows the object recognition algorithm to be used without affecting the creation of a restored image of the image encoder and image decoder. The information obtained using the object recognition algorithm in the object recognition layer includes the x-axis and y-axis information of the recognized object, the width of the recognized object, the height of the recognized object, the type of object, the recognized accuracy of the object, and the identification number of the object. etc. At this time, the obtained information can be used in a video encoder. Information obtained from the video encoder includes inter-screen prediction difference values, delta-qp, and quantization difference values. At this time, the obtained information can be transmitted through a bitstream.

The predicted difference value between screens refers to the value subtracted from the predicted value between screens used in existing video compression and the predicted value between screens based on object recognition.

delta-qp can be defined differently when quantizing different delta-qp values depending on the characteristics of the object based on information obtained through the object recognition algorithm.

The quantization difference value refers to the value obtained by subtracting the value obtained through inverse quantization used in existing image compression and the value obtained through object recognition-based inverse quantization.

Figure 10 shows that the image decoder can obtain a restored image through entropy decoding, intra/inter-screen prediction, inverse transformation and inverse quantization, and in-loop filter. At this time, a reconstructed image based on object recognition can be obtained by adding the inter-screen predicted difference or inverse quantization difference obtained from the object recognition layer.

Figure 10 receives a bitstream and can have inter-screen prediction difference values, inverse quantization difference values, delta-qp, etc. through the object recognition information decoding process.

Hereinafter, a method for applying an object recognition algorithm, compression unit reconstruction, bit rate control, intra/interscreen prediction, transformation/quantization, and entropy encoding/decoding methods according to embodiments of the present disclosure will be described in detail.

[E1] How to apply object recognition algorithm

Objects in the image can be identified by applying an object recognition algorithm from the input image. At this time, one type of object or the corresponding object can be identified through an algorithm.

Through the information obtained by applying the object recognition algorithm, compression can be performed by applying the information to the image. In other words, the method of applying video compression using information obtained through object recognition can be called Region of Interest-based Video Compression. At this time, various methods of compression can be performed depending on the type of information obtained through the object recognition algorithm.

The information obtained by applying the object recognition algorithm is as follows. The x-axis and y-axis information of the recognized object, the width of the recognized object, the height of the recognized object, the type of object, the recognized accuracy of the object, and the object identification number can be obtained through the object recognition algorithm.

For example, the x-axis and y-axis information of the recognized object, the width of the recognized object, and the height of the recognized object can be expressed as floating points. At this time, the value expressed as a decimal can be converted to an integer by multiplying the width or height of the image. The width of the x-axis and the recognized object can be integerized through the product of the width of the image, and the y-axis and the height of the recognized object can be integerized through the product of the height of the image. At this time, the integer representation is for efficient encoding/decoding. Information on the x-axis and y-axis of an object may represent the coordinates of the center of the object, the upper left corner, the upper right corner, the lower left corner, or the lower right corner.

As another example, the x- and y-axis information of the recognized object, the width of the recognized object, and the height of the recognized object can be expressed as decimals. At this time, values expressed as decimals can be encoded/decoded.

For example, transmitting the x-axis and y-axis information and the width and height of the recognized object separately may be burdensome, so for two or more objects, the difference value of the information between the first object and the second object and other objects must be obtained. Information can be encoded/decoded.

As a specific example, assuming there are two recognized objects, the information of the first object is as follows. The (x,y) axis coordinate value is (42,118), the width and height of the object are 50 and 30, respectively, the object type class is 2 (vehicle), and the object identification number is 0. The information about the second object is that the (x, y) axis coordinate value is (70,220), the width and height of the object are 40 and 28, respectively, the object type is 2 (vehicle), and the object identification number is 1. At this time, the information on the first object is encoded and transmitted, and when transmitting the information on the second object, the difference value for the two objects can be obtained and encoded before transmission. If you find the difference value of the x-axis coordinate between the second and first objects, you get 70-42=38. Likewise, if you calculate the difference value of the y-axis coordinate in the same way, you get 220-108=102. If the height and width are the same as the previous methods, the difference values for the width and height between the two objects are -10 and -2, respectively. The difference value for the type and class of the object is also 0. When encoding the difference values obtained in this way, a flag indicating the sign of the difference value can be transmitted, and the positive difference value can be encoded and transmitted.

At this time, if there are three or more objects, the same method is used. For the second object, the difference value is obtained using the above method with the second object, and for the second object, the difference value is obtained using the above method with the third object, encoded, and transmitted. there is.

For example, if the information of the recognized object is not changed, object_coincidence_flag becomes the first value and the type of recognized object, recognized accuracy of the object, identification number of the object, etc. may not be transmitted. However, the x- and y-axis information of the recognized object, the width of the recognized object, and the height of the recognized object can be transmitted.

For another example, if the information of the recognized object is not changed, object_coincidence_flag becomes the first value and may not transmit the type of recognized object, recognized accuracy of the object, identification number of the object, etc. However, if the image itself is a still image, the x- and y-axis information of the recognized object, the width of the recognized object, and the height of the recognized object may not be transmitted.

For another example, if the information of the recognized object changes, object_coincidence_flag becomes the second value, including the x- and y-axis information of the recognized object, the width of the recognized object, the height of the recognized object, and the The type, recognized accuracy of the object, identification number of the object, etc. can be transmitted.

At this time, the first value of object_coincidence_flag indicates 0 and the second value indicates 1.

At this time, the information obtained by applying the object recognition algorithm can be transmitted independently of [1] the high-level syntax elements of the picture/slice/tile/tile group/CTU/CTU row/CTU column during video compression or [2] the video compression stream. .

[E2] Reconstruction of compression units

Compression unit reconstruction adds a unit that stores information about a recognized object as a unit of an existing picture/slice/tile/tile group/CTU/CTU row/CTU column or as an object-based coding unit. (included) or replacement step. To explain specifically, when two or more objects exist in one picture and a unit for storing information about the objects is added, the picture/slice/tile/tile group/CTU/CTU row/CTU column It can be stored in units or object-based compression units.

At this time, the added object-based compression unit is a new unit rather than a unit used in existing video compression standards.

At this time, the units of picture/slice/tile/tile group/CTU/CTU row/CTU column are the units used in existing video compression standards.

At this time, the object-based compression unit may be a higher-order unit of the picture and may include slice/tile/tile group/CTU/CTU row/CTU column.

At this time, the object-based compression unit may be the upper unit of the slice and may include tile/tile group/CTU/CTU row/CTU column.

At this time, the object-based compression unit may be a higher-level unit of a tile or tile group, and may include a CTU/CTU row/CTU column.

At this time, the object-based compression unit may be a higher unit of the CTU row/CTU column and may include the CTU column.

[1] When using an object recognition algorithm in a high-level syntax element, based on the information obtained through the object recognition algorithm, a picture/slice/tile/tile group/CTU/CTU row/CTU column or object-based compression unit is used. It can be reconstructed. However, if there is no recognized object in the picture, the picture being compressed can be compressed like existing image compression, or the picture can be skipped and the next picture compressed.

[2] Alternatively, when transmitting objects as an independent stream rather than a video compressed bitstream, it is possible to apply the existing video compression standardization method as is.

At this time, it is possible to determine whether to reconstruct a picture/slice/tile/tile group/CTU/CTU row/CTU column or object-based compression unit through the object_based_coding_unit_reconstruction_flag flag that determines reconstruction of the compression unit. At this time, if the variable is the first value, reconstruction of the compression unit is not performed, and if the variable is the second value, reconstruction of the compression unit may be performed. At this time, the first value may be 0, and the second value may be 1.

Specifically, assuming that the object is recognized and the object's information can be known through [E1], picture/slice/tile/tile group/CTU/CTU row/CTU column or object-based compression is appropriate for the size of the object. This is a method of performing compression by adding (including) or replacing units.

At this time, when the width or height of the object is divided by 4, if the remainder is not 0, an arbitrary positive number (1 to 3) can be added to the width or height of the object so that the remainder becomes 0. Since the video compression standard uses a method of storing motion vectors in 4x4 units, it has the effect of adjusting the width or height of the object to be divisible by 4.

At this time, if two or more objects are recognized and the objects overlap, they can be considered as one object and the picture/slice/tile/tile group/CTU/CTU row/CTU column or object-based compression unit can be reconstructed.

At this time, when adding an object-based compression unit, picture/slice/tile/tile group/CTU/CTU row/CTU column can be included or replaced.

At this time, the object-based compression unit may be a lower level of picture/slice/tile/tile group/CTU/CTU row/CTU column. A lower level means that an object-based compression unit is included for the unit of that level. At this time, the unit of the corresponding step may be picture/slice/tile/tile group/CTU/CTU row/CTU column.

At this time, compression may not be performed on areas where objects are not recognized. Alternatively, compression can be performed quickly by skipping the RDO (Rate-Distortion Optimization) process.

As an example, an object-based compression unit can perform compression by configuring only where an object is recognized.

As another example, an object-based compression unit can perform compression by configuring a place where an object is recognized and a place where an object is not recognized to be distinguishable from each other.

Object-based compression units may be contiguous, but may not be contiguous.

11 to 12B are diagrams for explaining a method of reconstructing a compression unit according to an embodiment of the present invention.

[Figure 11] Compression unit reconstruction method

[Figure 12a] Compression unit reconstruction method

[Figure 12b] Compression unit reconstruction method

Figure 11 assumes that three objects are recognized in the input image. In the drawing, one square represents one CTU for convenience. At this time, a method of replacing CTU or slice using an object-based compression unit can be used. Specifically, Figures 12A and 12B can be said to be object-based compression units that replace slices. At this time, if the data obtained in [E1] is transmitted through a bitstream, the slice position may not be transmitted because the slice can be derived during the decoding process. However, if the data obtained in [E1] is not transmitted through a bitstream, the starting point of the object-based compression unit may need to be transmitted.

Figure 12a shows the area containing the object in red (the largest square in each area) using existing positions that may exist under the assumption that the picture is divided into CTUs for convenience of encoding/decoding, and this is the object. Can be used as a base compression unit. At this time, the object-based compression unit can be divided into CTUs to perform encoding/decoding.

In Figure 12b, an object-based compression unit can be reconstructed by referring to the upper, left, lower, and end edges of the object. Since the two objects on the left overlap, they can be created as one object-based compression unit. At this time, encoding and decoding can be done by considering the upper left corner of the object-based compression unit as the starting point without using the existing CTU location. Likewise, the bottom right of the object-based compression unit can be considered the endpoint. At this time, the starting point refers to the starting point for compression, and the end point refers to the final point reached for compression.

bit rate control

Bit rate control is a technology that constantly adjusts the amount of data of each picture that makes up an image. Therefore, it is a technology that prevents delays in transmitting video and prevents the user from receiving the video from suddenly transmitting an excessive amount of data so that the video can run smoothly.

Bit rate control is designed to be adjustable on a GOP, picture, or CU basis.

At this time, if object_based_coding_unit_reconstruction_flag, which determines reconstruction of the compression unit, is the second value, bit rate control can be designed with an object-based compression unit.

At this time, the object-based compression unit can replace GOP, picture, or CU.

Bit rate control sets the available bits on a GOP basis, and the GOP unit bits set accordingly can be separately set on a picture basis. And, depending on the bits assigned to the picture, the bits can be divided into CU units. At this time, an object-based compression unit may replace a GOP unit, picture unit, or CU unit. Alternatively, you can add an object-based compression unit as another unit. For example, assuming that there are 480 images and that the GOP unit consists of 16 pictures, 480/16=30 can be composed of 30 GOPs. At this time, the amount of bits available to encode 480 images is 300,000. If so, 100,000 bits can be used per GOP. Then, 100,000/16 = 6,250 bits in one picture unit. However, the allocation may not be equal depending on the importance or type of the picture or specific conditions.

As an example, when applying the [E1] object recognition algorithm, assuming that major objects are recognized in a specific GOP, in order to compress the video with better image quality, less bits are allocated to other GOPs and bits are allocated to the specific GOP. You can allocate more. At this time, it can be expressed through a weight called a_i. At this time, i is a value indicating the order of GOP. Assuming that it is composed of 30 GOPs, and the available bits are 300,000, each bit for one GOP can be obtained through (300,000/30) * a_i. a_i can be said to be a positive value. At this time, the value of a_i may be determined according to the number of objects in the GOP or according to the importance of the object.

As another example, it can be assumed that a GOP unit consists of 16 pictures, and that 6,250 bits are available for each picture unit. Alternatively, bits can be distributed differently for each picture through a weight called b_i for each picture, depending on the number of objects or the importance of the objects per picture. When setting the bits for each picture, each bit can be obtained through (100,000/16) * b_i. i is a value indicating the order of the picture, and b_i can be said to be a positive value.

As another example, it can be assumed that a GOP unit consists of 16 pictures, and that 6,250 bits are available for each picture unit. Assuming that one picture consists of 24 CUs, approximately 260 bits can be used per CU. In another method, bits can be distributed differently to each CU through a weight called c_i for each CU, depending on the number of recognized objects in the picture or the importance of the objects. When setting the bits for each CU, each bit can be obtained through (6,250/24)*c_i. i is a value indicating the order of CU, and c_i can be said to be a positive value.

As another example, it can be assumed that a GOP unit consists of 16 pictures, and that 6,250 bits are available for each picture unit. Assuming that one picture consists of 24 CUs, approximately 260 bits can be used per CU. In another method, bits can be distributed differently to each CU through a weight called c_i for each CU, depending on the number of recognized objects in the picture or the importance of the objects. When setting the bits for each CU, each bit can be obtained through (6,250/24)*c_i. i is a value indicating the order of CU, and c_i can be said to be a positive value.

As another example, an object-based compression unit can replace the CU stage. Therefore, it is possible to set the bits for each picture unit and set the bits to be used in the object-based compression unit. At this time, the object-based compression unit attempts to encode/decode only the area of interest where the object is located. Therefore, if you try to encode/decode in CU units, it consists of 24 CUs, but the object-based compression unit may differ for each picture because it configures the compression unit only where the object is located. Therefore, bit allocation can vary depending on the area of the object-based compression unit.

Compression can be performed by allocating more bits to the region where objects exist (Region of Interest), and compression can be performed by allocating fewer bits to regions where objects exist (Region of non-Interest). You can do it.

At this time, the total amount of bits can be allocated differently to the ROI area and the non-ROI.

At this time, the ROI area and non-ROI bit amount can be allocated using weights.

Intra/interscreen prediction

Table 1 shows the object-based inter-screen prediction method and the existing image compression process together in syntax.

isObjectflag[x0][y0] in Table 1 indicates the presence or absence of an object in the currently encoding/decoding block. At this time, x0 and y0 represent the left and top positions of the block, respectively.

The object_based_prediction_flag in Table 1 may have a different flag value depending on the efficiency of the Rate Distortion Optimization process compared to the existing video compression process. At this time, the first value of object_based_prediction_flag may be 0, and the second value may be 1.

For example, if object_based_prediction_flag is the first value, the prediction method of the existing image compression process can be performed.

For example, if object_based_prediction_flag is the second value, an object-based inter-screen prediction method can be performed.

[Table 1]

If an object exists, motion estimation/compensation can be performed using the information in [E1] or based on information obtained through an object recognition algorithm.

Alternatively, motion prediction/compensation can be performed only for the portion where the object exists.

If there is more than one object, motion prediction/compensation can be performed through the object's identification number, for example, as shown in Equation 1.

[Equation 1]

SAD stands for Sum of Absolute Difference, and during the encoding process, m _x and m _y can be used at the moment when SAD is the minimum value.

As an example, z can be 0 or F _k-1 (x+i,y+j) to simplify the encoding process by performing less motion prediction/compensation on blocks or areas where objects do not exist. .

As another example, existing image compression standardization technology can be used on blocks or areas where objects do not exist.

Alternatively, if there is more than one object, motion prediction/compensation can be performed through the identification number of the object, for example, as shown in Equation 2.

[Equation 2]

If an object exists This means using z in other cases. And, SAD stands for Sum of Absolute Difference, and is the moment when SAD is the minimum value during the encoding process. and You can use it.

As an example, to simplify the encoding process by performing less motion prediction/compensation on blocks or areas where objects do not exist, z is 0 or can be used.

As another example, existing image compression standardization technology can be used on blocks or areas where objects do not exist. At this time, z may be a motion vector value obtained through existing image compression standardization technology.

In relation to Equation 1 and Equation 2 above,

As an example, m _x and m _y can be derived through a motion prediction/compensation process, a method used in image compression standardization.

As another example, m _x and m _y can be derived based on information obtained through an object recognition algorithm. Specifically, it can be explained through FIG. 13.

Figure 13 is a diagram for explaining inter-screen prediction according to an embodiment of the present invention.

[Figure 13] Inter-screen prediction

(1) in FIG. 13 is a picture currently being encoded, and (2) in FIG. 13 can be said to be a reference picture. At this time, (3) in FIG. 13 can be viewed as showing the m _x and m _y values of the reference picture in the current picture. Since the x-axis and y-axis coordinate information can be known through the object recognition algorithm, the m _x and m _y values can be obtained through the difference value, as shown in (3) of FIG. 13. At this time, because the recognized object is a square and has values for four corners, motion prediction/compensation can be performed through all four values during the encoding process. In this way, motion prediction/compensation can be performed for objects of the same color (same identification number).

As another example, in (3) of FIG. 13, the difference values m _x and m _y can be obtained through the four corners of the recognized object in the current picture and the reference picture. At this _time , _m . At this time, motion prediction/compensation can be performed using the derived motion vector.

Transformation/quantization (inverse transformation/inverse quantization)

A method of changing the quantization parameter depending on whether an object exists can be used.

For example, assuming that an object-based compression unit is designed as shown in Figure 6 (1) through [E2] compression unit reconstruction, the quantization parameter value used in the picture can be derived and used in the object-based compression unit. . At this time, since the object is recognized in the object-based compression unit, in order to have better image quality, a random positive number N can be subtracted from the quantization parameter derived from the picture and compression can be performed by applying a lower quantization parameter to the compression unit to be currently encoded. there is. Likewise, in the decoding process, since information about the recognized object is included, inverse quantization can be performed by subtracting an arbitrary positive number N, as in the encoding process. As another example, assuming that an object-based compression unit is designed as shown in Figure 6 (2) through compression unit reconstruction, the quantization parameter value used in the picture can be derived and used in the object-based compression unit. At this time, since the object is recognized in the object-based compression unit, in order to have better image quality, a random positive number N can be subtracted from the quantization parameter derived from the picture and compression can be performed by applying a lower quantization parameter to the compression unit to be currently encoded. there is. Likewise, in the decoding process, since information about the recognized object is included, inverse quantization can be performed by subtracting an arbitrary positive number N, as in the encoding process.

As another example, when [E2] compression unit reconstruction is not used, when a recognized object exists in the slice/CTU column/CTU row that is currently being encoded, the quantization parameters derived from the picture are used to obtain better image quality. Compression can be achieved by subtracting an arbitrary positive number N and applying a lower quantization parameter to the slice/CTU column/CTU row currently being encoded. Likewise, in the decoding process, since information about the recognized object is included, inverse quantization can be performed by subtracting an arbitrary positive number N, as in the encoding process.

As another example, when [E2] compression unit reconstruction is not used, when the currently encoding CTU is included in the area of the recognized object, a random positive number is added to the quantization parameter derived from the slice in order to have better image quality. Compression can be achieved by subtracting N and applying a lower quantization parameter to the CTU currently being encoded. Likewise, in the decoding process, since information about the recognized object is included, inverse quantization can be performed by subtracting an arbitrary positive number N, as in the encoding process.

If a low quantization parameter is used, higher image quality can be obtained because the high frequency region of the image can be preserved. However, instead of increasing the image quality, it may also result in an increase in bits.

Entropy encoding/decoding

[E1] Can indicate a method of transmitting data obtained through application of an object recognition algorithm. Object recognition information (data) obtained through [E1] may be as follows. The number of recognized objects, x- and y-axis information of recognized objects, width of recognized objects, height of recognized objects, type of object, recognized accuracy of objects, object identification number, and other additional data can be transmitted. You can. At this time, among the methods presented in [E1], for two or more objects, a method of transmitting difference values for the x-axis and y-axis in the second object information can be used. However, if the process of deriving the difference value is not used, it can be used to transmit the existing x-axis and y-axis information, the width of the object, and the height of the recognized object. However, at this time, unnecessary data may not be transmitted.

As an example, assuming that there are N objects, when transmitting without using the difference value, the width of the recognized object, the height of the recognized object, the type of object, the recognized accuracy of the object, the object identification number, etc. All other information can be transmitted because it consists of positive values.

As another example, assuming that there are N objects and transmitting using difference values, the first objects can be transmitted assuming that all values are positive. Since the difference value is used for the second or higher object, the information such as x-axis and y-axis information of the recognized object, width of the recognized object, height of the recognized object, type of object, and recognized accuracy of the object are negative. Since it can have a value of , after transmitting a flag indicating the sign, the absolute value of the difference value can be transmitted.

At this time, N is two or more positive integer values.

After encoding, the value transmitted as a bitstream can be encoded through the process of [D1]. [D1] can be decrypted in the order transmitted in [E7]. At this time, when N or more objects are transmitted through difference values, the first object to be decoded is all transmitted as a positive value, so it can be decoded in the same way as the encoding process. At this time, the second or more objects can be decoded by applying a sign to the decoded value of the first object and then adding it.

[E2] object_based_coding_unit_reconstruction_flag that determines the reconstruction of the compression unit used for compression unit reconstruction. The flag can be encoded through the process of [E7] and then transmitted through the bitstream. At this time, after receiving the bitstream through the process of [D1], it can be decoded.

To use an object recognition algorithm, a high-level syntax element can determine which object recognition algorithm to use. The object recognition algorithm use flag or variable object_detection_algorithm_selection_index can be signaled.

At this time, if the variable is the first value, the object recognition algorithm is not used. If the variable is the second value, the first object recognition algorithm determined in advance is used. If the variable is a third value, a second predetermined object recognition algorithm can be used.

At this time, the first value may be 0, the second value may be 1, and the third value may be 2.

At this time, if N object recognition algorithms are provided, up to the (N+1)th number can be defined as above. The first value may be 0, the second value may be 1, the third value may be 2,... , the (N+1)th value may be N.

To use an object recognition algorithm, you can decide whether to use indexing information in a high-level syntax element to indicate what type of object to recognize. Index_list_of_object_names indicating the type of recognized object can be signaled.

At this time, the index indicating the type of recognized object may be set to a total of 10, 20, or 30.

14 and 15 are diagrams showing examples of object classification tables that can be applied to embodiments of the present invention.

[Figure 14] Object classification table 1

[Figure 15] Object classification table 2

Figures 14 and 15 are voc.data or coco.data, respectively, and are frequently used in the object recognition area.

As shown in Figures 14 and 15, the types of objects can be organized and provided by index. At this time, signaling can be provided through index_list_of_object_names during the encoding/decoding process.

Figure 16 is a flowchart showing an image encoding method according to an embodiment of the present invention.

[Figure 16] Video encoding method

Referring to FIG. 16, the method includes obtaining object recognition information in an input image based on an object recognition algorithm (S1310), and encoding the input image based on the object recognition information (S1320), When the object recognition information includes first object recognition information for a first object in the input image and second object recognition information for a second object in the input image, the second object recognition information is used to recognize the first object. It can be encoded as difference information from information.

Additionally, the bitstream generated by the video encoding method can be stored in a computer-readable recording medium and transmitted to an video decoding device.

Figure 17 is a flowchart showing an image decoding method according to an embodiment of the present invention.

[Figure 17] Video decoding method

Referring to FIG. 17, in the image decoding method performed by the image decoding device, obtaining object recognition information for the input image (S1410) and restoring the input image based on the object recognition information (S1410) S1420), and when the object recognition information includes first object recognition information for the first object in the input image and second object recognition information for the second object in the input image, the second object is recognized Information may be derived as difference information with the first object recognition information.

In one embodiment, reconstructing the input image includes determining whether to reconstruct a first compression unit of the input image, and if it is determined to reconstruct the first compression unit, based on the object recognition information. It may include reconstructing the first compression unit into a predetermined second compression unit, and restoring the input image based on the second compression unit.

In one embodiment, the second compression unit may be determined as a higher-order unit of a picture, slice, tile, tile group, CTU (coding tree unit) row, or CTU column.

In one embodiment, based on the fact that the recognized object does not exist in the input image, the steps of restoring the input image and filtering the restored input image may be skipped.

In one embodiment, whether to reconstruct the first compression unit may be determined based on a predetermined flag obtained from the bitstream.

In one embodiment, the steps of restoring the input image and filtering the restored input image may be selectively performed only for areas in the input image where an object is recognized.

In one embodiment, the step of restoring the input image and the step of filtering the restored input image are performed differentially for a first area in which an object is recognized and a second area in which an object is not recognized in the input image. It can be done.

In one embodiment, the second compression unit may be individually determined for each object recognized in the input image.

In one embodiment, for first objects that overlap each other among the objects recognized in the input image, the second compression unit may be determined to be a predetermined area that completely covers the first objects.

In one embodiment, the bitrate of the input image may be determined in predetermined units based on the object recognition information.

In one embodiment, the predetermined unit is determined to be one of a first compression unit and a second compression unit, and the first compression unit is a group of picture (GoP) unit, a picture unit, and a coding unit (CU). The second compression unit may be a unit reconstructed from the first compression unit based on the object recognition information.

In one embodiment, within the input image, a bit rate of a first area including a recognized object may be higher than a bit rate of a second area not including the recognized object.

In one embodiment, the bit rate may be determined individually for each area containing a recognized object within the input image.

In one embodiment, the bit rate may be determined differently based on the importance of objects included in each area.

In one embodiment, the reconstructed input image may be generated by performing inter prediction based on the object recognition information.

In one embodiment, the inter prediction may be performed only on an area containing a recognized object within the input image.

In one embodiment, the restored input image may be generated by performing inverse quantization based on the object recognition information.

In one embodiment, within the input image, a quantization parameter of a first area containing the recognized object may be smaller than a quantization parameter of a second area not containing the recognized object.

In one embodiment, the object recognition information may be obtained from a first bitstream including input image information.

In one embodiment, the object recognition information may be obtained from a second bitstream that is different from the first bitstream including input image information.

Figure 18 is a block diagram schematically showing an electronic device including an image encoding/decoding device according to an embodiment of the present disclosure.

[Figure 18] Electronic devices

Referring to FIG. 18, the electronic device 1800 is a comprehensive concept including a smartphone, tablet PC, smart wearable device, etc., and may include a display 1810, a memory 1820, and a processor 1830.

The display 1810 may include an Organic Light Emitting Diode (OLED), Liquid Crystal Display (LCD), or Plasma Display Panel (PDP) display, and may display various images on the screen. Additionally, the display 1810 may provide user interface functions. For example, the display 1810 may provide a means for a user to input various commands.

The memory 1820 may be a storage medium that stores data or multimedia data necessary for the operation of the electronic device 1800. The memory 1820 may include a storage device based on a semiconductor device. For example, the memory 1820 may include DRAM, synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), low power double data rate SDRAM (LPDDR SDRAM), graphics double data rate SDRAM (GDDR SDRAM), and DDR2 SDRAM. It may include a dynamic random access memory device such as DDR3 SDRAM, DDR4 SDRAM, or a resistive memory device such as PRAM (Phase change Random Access Memory), MRAM (Magnetic Random Access Memory), or RRAM (Resistive Random Access Memory). Additionally, the memory 1820 may include at least one of a solid state drive (SSD), a hard disk drive (HDD), and an optical drive (ODD) as a storage device.

The processor 1830 may perform the image encoding/decoding method described above with reference to FIGS. 1 to 17 based on a neural network. Specifically, the processor 1830 may obtain the first image feature from the input image based on a neural network. The processor 1830 may obtain block information features of the input image from block information of the input image based on a neural network. The processor 1830 may obtain a second image feature by removing noise and distortion of the first image feature based on the block information feature and a neural network. And, the processor 1830 may restore the input image based on a neural network based on the second image feature. At this time, the block information may include at least one of a block boundary map indicating a block division structure of the input image and a block distribution map indicating encoding information of the input image.

The processor 1830 may be a central processing unit (CPU), a microprocessor unit (MCU), a system-on-chip (SoC), etc., and communicates with the display 1810 and/or memory 1820 and various data through the bus 1840. It can be exchanged.

In the above-described embodiments, the methods are described based on flowcharts as a series of steps or units, but the present invention is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. You can. Additionally, a person of ordinary skill in the art will recognize that the steps shown in the flowchart are not exclusive and that other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You will understand.

The above-described embodiments include examples of various aspects. Although it is not possible to describe all possible combinations for representing the various aspects, those skilled in the art will recognize that other combinations are possible. Accordingly, the present invention is intended to include all other substitutions, modifications and changes falling within the scope of the following claims.

Embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person skilled in the art to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

Claims (3)

In an image decoding method performed by an image decoding device,
Obtaining object recognition information for an input image; and
Based on the object recognition information, restoring the input image,
When the object recognition information includes first object recognition information for the first object in the input image and second object recognition information for the second object in the input image, the second object recognition information is the first object recognition information. Derived as differential information from recognition information
Video decoding method. In an image encoding method performed by an image encoding device,
Obtaining object recognition information in an input image based on an object recognition algorithm; and
Based on the object recognition information, encoding the input image,
When the object recognition information includes first object recognition information for the first object in the input image and second object recognition information for the second object in the input image, the second object recognition information is the first object recognition information. Encoded as difference information from recognition information
Video encoding method. A computer-readable recording medium storing a bitstream generated by an image encoding method, the image encoding method comprising:
Obtaining object recognition information in an input image based on an object recognition algorithm; and
Generating a first bitstream by encoding the input image based on the object recognition information,
When the object recognition information includes first object recognition information for the first object in the input image and second object recognition information for the second object in the input image, the second object recognition information is the first object recognition information. It is encoded as difference information with recognition information,
The object recognition information is selectively included in either the first bitstream or the second bitstream that does not include input image information.
Computer-readable recording medium.