CN115297327B - Semantic priori coding and decoding method and system based on semantic structured coding - Google Patents

Abstract

The invention discloses a semantic priori coding and decoding method and system based on semantic structured coding, which combines the existing semantic structured coding method, and provides an improved method for improving coding and decoding performance, namely, fully utilizing semantic priori information in a structured code stream, and realizing better coding and decoding performance while maintaining the functionality of the semantic structured coding, namely, better reconstruction quality under the same code rate or lower code rate under the same reconstruction quality.

Description

Semantic a priori encoding and decoding method and system based on semantic structured coding

Technical Field

The present invention relates to the technical field of image compression coding, and in particular to a semantic a priori coding and decoding method and system based on semantic structured coding.

Background Art

In order to cope with the challenges of the explosion of visual data in the 5G era, and to more efficiently support machine intelligence application scenarios and human-machine hybrid intelligent application scenarios, and even support flexible editing of image content at the code stream level, existing methods have proposed semantically structured encoding of images. For example, the Chinese invention patent with announcement number CN110225341 B, "Semantic structured image encoding and decoding method based on deep learning", and the image video encoding MPEG-4Visual based on the visual object plane.

In the above patent "Semantic structured image encoding and decoding method based on deep learning", the regional decision network and alignment module of target detection are introduced to extract the bounding box of the area where the semantic object may exist in the compressed feature, and the feature is segmented at the spatial level to obtain a semantically structured code stream, that is, each segment of the code stream represents a semantic object separately. This method can support the decoding and reconstruction of some semantic objects, and use part of the semantic information to directly perform machine intelligence analysis tasks. Image and video encoding MPEG-4Visual divides the input image/video into various visual object planes (Visual Object Plane) according to semantic objects, and blocks, transforms, predicts, quantizes and entropy encodes each visual object plane separately to form a structured code stream. This method can support the separate decoding and reconstruction of some visual objects, and supports the editing of visual objects, such as scaling, shifting or rotating, and the recombination of visual objects from different images.

Although existing structured coding methods have strong functionality, there is still much room for improvement in encoding and decoding performance compared to general image coding methods (such as H.264, H.265, H.266, etc.).

Summary of the invention

The purpose of the present invention is to provide a semantic a priori encoding and decoding method and system based on semantic structured coding, which makes full use of semantic a priori information, improves the existing semantic structured coding method, and enhances the encoding and decoding performance of the semantic structured coding method.

The objective of the present invention is achieved through the following technical solutions:

A semantic a priori encoding and decoding method based on semantic structured coding, comprising:

At the encoding end, the input image passes through the semantic analysis module to obtain the position information and semantic category label corresponding to the semantic object in the compressed feature; based on the position information of the semantic object, the input image or the compressed feature corresponding to the input image is segmented into several parts containing only the semantic object at the spatial level; the segmented image or compressed feature is respectively input into the subsequent encoding module to obtain a structured bitstream; the encoding end maintains a semantic category label pool, determines the index value of the semantic category label according to the semantic category label obtained by the semantic analysis module, and fills the index value of the semantic category label into the specified position in the structured bitstream;

At the decoding end, a decoder pool consisting of decoders corresponding one-to-one to the labels in the semantic label pool is maintained, and the corresponding decoder is selected according to the index value of the semantic category label in the structured bitstream to decode the relevant bitstream in the structured bitstream.

A semantic a priori coding and decoding system based on semantic structured coding, comprising:

The coding network is applied to the coding end; at the coding end, the input image is passed through a semantic analysis module to obtain the position information and semantic category label corresponding to the semantic object in the compressed feature; based on the position information of the semantic object, the input image or the compressed feature corresponding to the input image is segmented into several parts containing only the semantic object at the spatial level; the segmented image or compressed feature is respectively input to the subsequent coding module to obtain a structured bitstream; the coding end maintains a semantic category label pool, determines the index value of the semantic category label according to the semantic category label obtained by the semantic analysis module, and fills the index value of the semantic category label into a specified position in the structured bitstream;

The decoding network is applied to the decoding end; at the decoding end, a decoder pool consisting of decoders corresponding one-to-one to the labels in the semantic label pool is maintained, and the corresponding decoder is selected according to the index value of the semantic category label in the structured bitstream to decode the relevant bitstream in the structured bitstream.

A processing device, comprising: one or more processors; a memory for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

It can be seen from the technical solution provided by the present invention that, in combination with the existing semantic structured coding method, an improved method for improving the coding and decoding performance is proposed, that is, making full use of the semantic prior information in the structured bit stream, while maintaining the functionality of the semantic structured coding, to achieve better coding and decoding performance, that is, better reconstruction quality under the same bit rate or lower bit rate under the same reconstruction quality.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

FIG1 is a flow chart of a semantic a priori encoding and decoding method based on semantic structured coding provided by an embodiment of the present invention;

FIG2 is a schematic diagram of a semantic a priori coding method based on semantic structured coding for performing semantic segmentation at a compressed feature level provided by an embodiment of the present invention;

3 is a schematic diagram of a semantic a priori coding method based on semantic structured coding for performing semantic segmentation at an image level provided by an embodiment of the present invention;

FIG4 is a schematic diagram of a semantic a priori encoding and decoding system based on semantic structured coding provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of a processing device provided in an embodiment of the present invention.

DETAILED DESCRIPTION

The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

First, the terms that may be used in this article are explained as follows:

The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

The following is a detailed description of a semantic a priori encoding and decoding method and system based on semantic structured coding provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer.

Embodiment 1

The embodiment of the present invention provides a semantic a priori encoding and decoding method based on semantic structured coding, as shown in FIG1 , which mainly includes:

At the encoding end, the input image passes through the semantic analysis module to obtain the position information and semantic category label corresponding to the semantic object in the compressed feature; based on the position information of the semantic object, the input image or the compressed feature corresponding to the input image is segmented into several parts containing only the semantic object at the spatial level; the segmented image or compressed feature is respectively input into the subsequent encoding module to obtain a structured bitstream; the encoding end maintains a semantic category label pool, determines the index value of the semantic category label according to the semantic category label obtained by the semantic analysis module, and fills the index value of the semantic category label into the specified position in the structured bitstream;

At the decoding end, a decoder pool consisting of decoders corresponding one-to-one to the labels in the semantic label pool is maintained, and the corresponding decoder is selected according to the index value of the semantic category label in the structured bitstream to decode the relevant bitstream in the structured bitstream.

In the embodiment of the present invention, the input image passes through the semantic analysis module to obtain the position information and semantic category label of the semantic object in the input image. Since the compressed features obtained by the universal encoder can be regarded as the result of the downsampling transformation of the input image, the position information and semantic category label of the semantic object in the input image can be mapped to the compressed features through the same downsampling operation, so that the position information and semantic category label of the semantic object in the compressed features can be obtained.

In the embodiment of the present invention, bytes specifically used to store index values are set in the structured bitstream. The index values are first converted into binary representations, and then the corresponding bytes storing the index values are filled.

The above-mentioned scheme provided by the embodiment of the present invention combines the existing semantic structured coding method, makes full use of semantic prior information, improves the existing semantic structured coding method, and improves the encoding and decoding performance of the semantic structured coding method, that is, better reconstruction quality under the same bit rate or lower bit rate under the same reconstruction quality.

In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, a semantic a priori encoding and decoding method based on semantic structured coding provided by an embodiment of the present invention is described in detail with reference to a specific embodiment below.

As shown in FIG. 2 and FIG. 3 , they are schematic diagrams of implementing the above method by performing semantic segmentation at the compression feature level and performing semantic segmentation at the image level, respectively.

In both Figures 2 and 3, the semantic analysis module performs semantic analysis on the input image to obtain the location information and semantic category labels of the semantic objects in the compressed features. The location information of the semantic objects includes: a bounding box of the semantic object or a semantic segmentation mask.

In an embodiment of the present invention, the semantic analysis module can be implemented by using commonly used neural network-based methods, such as Mask RCNN, CenterNet and other target detection or instance segmentation methods.

At the same time, in order to form a semantically structured code stream, based on the location information of the semantic object, the input image or the compressed features obtained by the general encoder will be segmented into several parts containing only the corresponding semantic information at the spatial level. All segmented images/compressed features will be input into the subsequent encoding module respectively to form a structured code stream. As shown in Figures 2 and 3, for the segmented compressed features, the subsequent encoding module includes: a quantizer and an entropy encoder, and the segmented compressed features are sequentially passed through the quantizer and the entropy encoder to obtain a structured code stream; for the segmented image, the subsequent encoding module includes: an encoder, a quantizer and an entropy encoder, and the segmented image is sequentially passed through the encoder, a quantizer and an entropy encoder to obtain a structured code stream.

The semantic structured coding process can refer to the prior art, for example, the technical solutions in the patents cited in the aforementioned background technology. The difference is that in the embodiment of the present invention, the semantic segmentation process in the semantic structured coding can be implemented at the image level or at the compression feature level. Furthermore, the semantic segmentation method can choose to use a coarse-grained bounding box as the basis for segmentation, or choose a more precise semantic segmentation map as the basis for segmentation.

In order to make full use of the acquired prior information such as the location information and semantic category label information of the semantic objects, so as to help improve the encoding performance. In an embodiment of the present invention, the semantic category label pool maintained by the encoding end stores all semantic category labels and the index values corresponding to each label, as shown in Figures 2 and 3, providing examples of a small number of semantic category labels and corresponding index values: fish - index value 1, car - index value 2, person - index value 3. In an embodiment of the present invention, the setting of the categories in the semantic category label pool is determined according to the specific application scenario. In the structured bitstream, a storage space of multi-bit (for example, 8-bit) index values (that is, supporting up to 1024 semantic categories) can be preset for use. If multiple semantic objects of the same semantic category appear in the same image, the method of encoding the number of codes and the corresponding index values can be used to reduce the transmission of repeated index values.

Correspondingly, the decoding end maintains a decoder pool composed of decoders corresponding to the labels in the semantic label pool one by one, as shown in Figures 2 and 3. The decoder pool stores the best decoder of the semantic category corresponding to the index value in the semantic category label pool and a general decoder. The best decoder of each semantic category has the best decoding performance when the compressed features of the corresponding semantic category are used as input, but it is not generalizable for other semantic categories, so it cannot obtain good performance. The general decoder is a decoder that is more generalizable than the decoder of the corresponding semantic category, and can obtain generally good performance on any image, but the decoding performance of the image of a specific semantic category is not as good as the best decoder of the semantic category. Generally speaking, the encoder used for the image part corresponding to each semantic object after segmentation in Figure 3 is generally a general encoder, but if the computing power and storage space of the encoding end allow, the encoder can also be designed for a specific semantic category to achieve the best encoder for the semantic category. The encoder of a specific semantic category must match the corresponding decoder in the decoder pool to be correctly decoded. In the embodiment of the present invention, the best encoder/decoder can be obtained by end-to-end training of a deep neural network CNN, and its optimization target is a trade-off loss function between bit rate and reconstruction distortion. The encoder/decoder trained on a specific semantic category data set can theoretically achieve the best adaptation to the data of the specific semantic category data set, thereby showing the best encoding performance on the current semantic category data.

At the encoding end, if the technical solution in the patent cited in the above-mentioned background technology already contains the semantic position information and semantic category information of the segmented part in the structured code stream, the semantic category information is mapped one-to-one to an index value in the present invention and then transmitted. If the structured coding method itself does not transmit the semantic category information, the method of the present invention will increase the transmission of the index value. The grammatical structure of the index value of the semantic category label includes: an identification of whether structured coding is supported, the total number of semantic categories contained in the input image, the number of types of a certain semantic category contained in the input image, and the index value corresponding to each semantic category. After obtaining the index value in the semantic structured code stream, the decoding end can select the best decoder for the image/compression feature to be decoded to complete the decoding operation. Tables 1 and 2 respectively show the description of key grammatical items in the semantic structure related to the semantic category index value and the definition of the grammatical structure (syntax) related to the semantic category index value. Considering that the present invention supports structured coding, the identification of whether structured coding is supported is set to 1.

Table 1: Description of key grammatical items in the semantic structure associated with semantic category index values

Table 2: Syntax definitions of semantic category index values

The definition of the bitstream structure in Table 2 also determines the process of the decoder reading the bitstream and obtaining the information in the bitstream. First, the detection_enabled_flag flag is used to determine whether the bitstream supports semantic structuring. If it does, continue to read the 8-bit bitstream object_class_max_num and obtain the number of semantic categories contained in the image based on the bitstream. According to the number of semantic categories, read the category labels of each semantic category of object_class_index in turn to specify the decoder.

It should be noted that the number of bits and the number of semantic categories shown in Table 1 and Table 2 are examples and do not constitute a limitation. In actual applications, users can set specific values of the number of bits and the number of semantic categories based on actual conditions or experience.

The above scheme of the embodiment of the present invention can also update the semantic category label pool and the decoder pool according to the situation. When a new semantic category that is not included in the semantic category label pool appears in the application scenario, priority is given to obtaining the relevant data of the new semantic category, and the best decoder is designed for the data set of the new category, and the index value and decoder corresponding to the new category are transmitted to the decoding end to update the decoder pool, and the semantic category and corresponding index value of the semantic category label pool are synchronously updated. When it is difficult for the encoding end to obtain the relevant data of the new category, the decoding of the image/compression features related to the new semantic category is all completed using a general decoder.

In the embodiment of the present invention, the decoder pool includes several decoders and general decoders that are optimal for a specific category, which are specifically manifested as model parameters (weights and biases of kernel functions) in the method related to deep learning. In order to reduce the storage space required for storing decoders, the model parameters of the general decoder and the residual value obtained by subtracting the general decoder from the model parameters of the best decoder of each category are stored.

The above solution of the embodiment of the present invention mainly has the following advantages:

(1) Based on the semantic structured coding method, an improved method for improving the encoding and decoding performance is proposed. That is, the semantic prior information in the structured bitstream is fully utilized to achieve better encoding and decoding performance while maintaining the functionality of the semantic structured coding.

(2) An improvement on the syntax in the semantically structured code stream is proposed to realize the transmission of index values, thereby realizing the selection of the best decoder to adapt to the decoding of each object code stream at the decoding end.

(3) An efficient storage method for each decoder in the decoder pool is proposed, which effectively reduces the storage space required for storing the best decoder of each category and the general decoder at the decoding end.

In summary, the solution provided by the present invention can effectively improve the encoding and decoding performance of semantic structured coding related methods.

Embodiment 2

The present invention also provides a semantic a priori coding and decoding system based on semantic structured coding, which is mainly implemented based on the method provided in the above embodiment. As shown in FIG4 , the system mainly includes:

The coding network is applied to the coding end; at the coding end, the input image is passed through a semantic analysis module to obtain the position information and semantic category label corresponding to the semantic object in the compressed feature; based on the position information of the semantic object, the input image or the compressed feature corresponding to the input image is segmented into several parts containing only the semantic object at the spatial level; the segmented image or compressed feature is respectively input to the subsequent coding module to obtain a structured bitstream; the coding end maintains a semantic category label pool, determines the index value of the semantic category label according to the semantic category label obtained by the semantic analysis module, and fills the index value of the semantic category label into a specified position in the structured bitstream;

The decoding network is applied to the decoding end; at the decoding end, a decoder pool consisting of decoders corresponding one-to-one to the labels in the semantic label pool is maintained, and the corresponding decoder is selected according to the index value of the semantic category label in the structured bitstream to decode the relevant bitstream in the structured bitstream.

Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

Embodiment 3

The present invention also provides a processing device, as shown in Figure 5, which mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the aforementioned embodiment.

Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

The output device may be a display terminal;

The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

Embodiment 4

The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (8)

1. A semantic priori coding and decoding method based on semantic structured coding is characterized by comprising the following steps:

At the encoding end, the input image passes through a semantic analysis module to obtain position information and semantic category labels corresponding to semantic objects in the compression characteristics; dividing an input image or compression characteristics corresponding to the input image into a plurality of parts only containing the semantic object on a space level based on the position information of the semantic object; the segmented images or compression features are respectively input into a subsequent coding module to obtain a structured code stream; the encoding end maintains a semantic class label pool, determines the index value of the semantic class label according to the semantic class label obtained by the semantic analysis module, and fills the index value of the semantic class label into a designated position in the structured code stream;

At the decoding end, a decoder pool formed by decoders corresponding to tags in the semantic tag pool one by one is maintained, and the corresponding decoder is selected according to the index value of the semantic category tag in the structured code stream, so that the related code stream in the structured code stream is decoded;

The method for maintaining a decoder pool formed by decoders corresponding to labels in the semantic label pool one by the encoding end comprises the following steps: the semantic category label pool maintained by the encoding end stores all semantic category labels and index values corresponding to the labels; the decoder pool maintained by the decoding end stores an optimal decoder and a universal decoder of semantic categories, wherein the optimal decoder and the universal decoder of the semantic categories are in one-to-one correspondence with index values in the semantic category label pool, the universal decoder can be suitable for each semantic category, and the optimal decoder of each semantic category has optimal decoding performance when compression characteristics of the corresponding semantic category are used as input;

The decoding end stores model parameters of the general decoder and residual values obtained by subtracting the model parameters of the optimal decoder of each semantic category from the general decoder.

2. The semantic prior encoding and decoding method based on semantic structured coding according to claim 1, wherein the location information of the semantic object comprises: a bounding box of a semantic object or a semantic segmentation map.

3. The semantic prior encoding and decoding method based on semantic structured coding according to claim 1, wherein the inputting of the segmented image or the compressed feature to the subsequent encoding module, respectively, comprises:

For the segmented image, the subsequent encoding module includes: an encoder, a quantizer, and an entropy encoder; the segmented image sequentially passes through an encoder, a quantizer and an entropy encoder to obtain a structured code stream;

for the segmented compression feature, the subsequent encoding module includes: a quantizer and entropy encoder; and the segmented compression characteristics sequentially pass through a quantizer and an entropy coder to obtain a structured code stream.

4. The semantic prior encoding and decoding method based on semantic structured coding according to claim 1, further comprising:

When a new semantic category which is not contained in the semantic category label pool appears in the application scene, designing an optimal decoder aiming at the new semantic category, updating the decoder pool by utilizing the index value of the new semantic category and the corresponding optimal decoder, and synchronously updating the semantic category of the semantic category label pool and the corresponding index value.

5. The semantic prior encoding and decoding method based on semantic structured coding according to claim 1, wherein the syntax structure of the index value of the semantic category label comprises:

Whether the identification of the structured coding is supported, the total number of semantic categories contained in the input image, the number of categories of a certain semantic category contained in the input image, and the index value corresponding to each semantic category.

6. A semantic prior codec system based on semantic structured coding, characterized in that it is implemented based on the method of any one of claims 1 to 5, the system comprising:

The coding network is applied to the coding end; at the encoding end, the input image passes through a semantic analysis module to obtain position information and semantic category labels corresponding to semantic objects in the compression characteristics; dividing an input image or compression characteristics corresponding to the input image into a plurality of parts only containing the semantic object on a space level based on the position information of the semantic object; the segmented images or compression features are respectively input into a subsequent coding module to obtain a structured code stream; the encoding end maintains a semantic class label pool, determines the index value of the semantic class label according to the semantic class label obtained by the semantic analysis module, and fills the index value of the semantic class label into a designated position in the structured code stream;

The decoding network is applied to the decoding end; at the decoding end, a decoder pool formed by decoders corresponding to the tags in the semantic tag pool one by one is maintained, and the corresponding decoders are selected according to the index values of the semantic category tags in the structured code stream, so that the related code stream in the structured code stream is decoded.

7. A processing apparatus, comprising: one or more processors; a memory for storing one or more programs;

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-5.

8. A readable storage medium storing a computer program, characterized in that the method according to any one of claims 1-5 is implemented when the computer program is executed by a processor.