CN115604475A - Multi-mode information source joint coding method - Google Patents

Abstract

A multi-modal information source joint encoding method. Firstly, multiple modal information sources are extracted through the corresponding first encoder to remove the internal redundancy of each modal signal, and the corresponding feature map is obtained; then multiple groups of feature maps are connected. Get up and input the second encoder, decoupled into a common feature map and a personality feature map; the common feature map represents the common part between different modal information sources, and the individual characteristic map represents the unique characteristics of each modal information source; finally The individual characteristic map and common feature map of multiple modal information sources are decoded by the corresponding decoder and the corresponding modal information sources are reconstructed, that is, they are converted into binary code streams for storage or transmission through entropy encoding respectively; the binary code at the decoding end After the stream is entropy decoded, the corresponding modal information source is restored by the corresponding decoder; the present invention utilizes the correlation between different information sources to reduce the repeated transmission of related information, reduce the transmission bandwidth, and reduce the storage space; the decoding end restores the Different modality sources, with modality scalability.

Description

A Joint Coding Method for Multimodal Information Sources

technical field

The invention relates to the technical field of information source coding, in particular to a multi-mode information source joint coding method.

Background technique

As a basic technology, source coding is widely used in various fields. Source coding is the product of the combination of multimedia technology and Internet technology in the information age. It aims to express the source with the fewest bits under the premise of allowing certain distortion or not allowing distortion. High-efficiency source coding technology can greatly improve the quality of decoded source and reduce storage space under limited bandwidth. For example, depending on the input, there are currently text compression, image compression (such as PNG, BMP, JPEG, BPG, WEBP and other compression standards), video compression (such as H.264/AVC, H.265/HEVC, H.266/VVC , VP9, AV1, AVS1, AVS2, AVS3, etc.), audio coding (such as AAC, etc.), etc., these standards have a common feature, only for a single type of input, for example, text compression is only for text input, image compression is only for Image and video compression is for images or videos, and audio encoding is only for audio input, and cannot process other forms, even if the processing requires pre-processing, and the efficiency is low. For example, video compression coding standards cannot intuitively compress text. Although text can be organized into video form through preprocessing, its content is very different from normal video and has no actual physical meaning. The technology in video codec standards does not target This is an unusual signal design, so even mandatory encoding would be inefficient.

In practice, data from several modalities are often combined for a certain expression. For example, the most common modalities such as TV dramas and movies include three modalities: video, audio, and subtitles. According to the above standards, the current solutions are almost all three The three modes are encoded separately, but in fact there is a correlation between the three mode signals, that is, there is a certain degree of redundancy, and the existing independent encoding method cannot eliminate such redundancy, so it is a limitation on bandwidth or storage space. a waste. Therefore, a method capable of jointly encoding signals of multiple modalities is needed to remove correlations between signals of different modalities and reduce redundancy, thereby achieving the purpose of reducing bandwidth and saving storage space.

Contents of the invention

In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a multi-modal information source joint coding method, which reduces the repeated transmission of related information by utilizing the correlation between different information sources in the encoding and compression process. Reduce the transmission bandwidth and storage space; the decoding end recovers different modal information sources according to the needs, that is, it has modal scalability.

In order to achieve the above object, the technical scheme that the present invention takes is:

A multimodal information source joint coding method, comprising the following steps:

1) passing multiple modal information sources through corresponding first encoders to extract features and remove internal redundancy of each modal signal to obtain corresponding feature maps;

2) In order to remove the correlation between different modal signals, multiple sets of feature maps are connected and input to the second encoder, and decoupled into common feature maps and individual feature maps; the common feature maps represent the differences between different modal sources. The common part, the personality map represents the characteristics unique to each modality source;

3) Decode the individual feature maps and common feature maps of multiple modal information sources through the corresponding decoder and reconstruct the corresponding modal information sources, that is, respectively undergo entropy encoding and convert them into binary code streams for storage or transmission; at the decoding end After the binary code stream is entropy decoded, the corresponding modal information sources are recovered through corresponding decoders respectively.

The knowledge base is introduced to jointly encode multi-modal information sources; the knowledge base is multi-modal or single-modal, and the multi-modal knowledge base refers to the storage of information in the knowledge base that contains a variety of different forms from different modal sources. ; Single or multiple modal information sources obtain the index of the retrieval knowledge base through "modal analysis", and "modal analysis" is used to obtain the node entity of the knowledge base for query and reasoning.

One form of representation of the multimodal knowledge base includes text and images, represented by nodes and edges, each node represents an entity or text or image, and each edge represents the relationship between different nodes.

The beneficial effects of the present invention are: the present invention proposes a multi-modal information source joint coding method, which characterizes each modal information source as common features and individual features, and the common features between different modal information sources are the same, and then Realize the joint coding of multiple modal sources. Compared with independent coding of multiple modal information sources, the present invention utilizes the correlation between different information sources in the process of encoding and compression to reduce repeated transmission of related information, thereby reducing transmission bandwidth and storage space. At the same time, the decoding end can recover information sources of different modalities according to needs, that is, it has the advantage of modal scalability.

On the basis of the above multimodal joint coding method, the present invention introduces a knowledge base (where there is known information strongly related to the information source to be encoded), increases prior knowledge, and explicitly associates information sources of different modes, In the encoding process, the prior knowledge in the knowledge base is used to guide the multimodal encoding process. Therefore, compared with multi-modal joint coding without knowledge base, it can further save storage space and reduce bandwidth.

Description of drawings

FIG. 1 is a flow chart of a joint encoding method for multi-modal information sources according to Embodiment 1 of the present invention.

FIG. 2 is a flow chart of a knowledge base-assisted joint encoding method for multi-modal information sources according to Embodiment 2 of the present invention.

Fig. 3 is an image and text multimodal knowledge base in Embodiment 2 of the present invention.

FIG. 4 is a flowchart of a knowledge base-assisted joint encoding method for multi-modal information sources according to Embodiment 3 of the present invention.

detailed description

The present invention will be described in detail below in conjunction with the accompanying drawings and embodiments.

Embodiment 1, embodiment 1 provides the example that two kinds of information sources are given as input, a kind of multimodal information source joint coding method, comprises the following steps:

1) Given two modal signal sources "modal 1" and "modal 2", denoted as src ₁ and src ₂ respectively, the two modal signals respectively pass through the first encoder A and the first encoder B to Extract features to remove the internal redundancy of each modal signal, and obtain the feature map feat ₁ and feature map feat ₂ , wherein the first encoder A and the second encoder B can be the convolutional neural network CNN in the neural network without special restrictions, It can also be a time series recurrent neural network RNN; feature map feat ₁ and feature map feat ₂ can be one-dimensional vectors, two-dimensional matrices or even higher-dimensional tensors;

2) In order to remove the correlation between different modal signals, the two sets of feature maps are connected and input to the second encoder C, decoupled into common feature maps and individual feature maps; the common feature maps represent the differences between different modal sources The common part of , usually at the semantic level; the personality feature map represents the unique features of each modal source; taking video and audio as examples, the common feature may be the words spoken by the characters in the video, The audio usually also contains this information; the personality characteristics of the video can be the appearance of the characters in the video or other background information other than the characters, such as flowers and plants, and the personality characteristics of the audio may include other irrelevant audio, or it can be the tone that is usually difficult to express in the video, etc. ;

In this embodiment, the common and individual features are decoupled, and the individual features feati ₁ of modality 1, the common features featc of the two modalities and the individual features feati ₂ of modality 2 are output, and the second encoder C may include a quantization process to Realize lossy coding, its structure has no special requirements, it can be CNN, RNN can also include hyper prior model; In addition, it should be noted that the internal characteristics of feati ₁ , featc and feati ₂ are not necessarily the same, such as feati ₁ The interior may contain side information featis ₁ and feature featii _1. The side information featis ₁ is used to assist the generation of featii ₁ , and featc and feati ₂ are the same;

featci₁和featc共同输入解码器B，以恢复模态2，记为

3) featc, feati ₁ and feati ₂ three types of features are respectively entropy coded and converted into binary code streams for storage or transmission; the binary code streams at the decoding end are restored to obtain feati ₁ , featc and feati ₂ after entropy decoding; after that feati ₁ and featc are jointly input to decoder A to recover modality 1, labeled as

Featci ₁ and featc are jointly input to decoder B to recover modality 2, denoted as

设计为以下形式：The above is the test process. In the training process, only paired multi-modal data can be used for training. During the training process, the encoder and decoder of multiple modalities perform end-to-end training together. The loss function

designed to be of the form:

和

用于衡量

和

转换为二进制码流所消耗的比特数量，通常可以通过估计获得。例如上述描述中，可以假设featc，feati₁和feati₂三类特征服从高斯分布，用featis₁中的部分特征表示高斯分布的均值，另外部分特征表示方差，即编码器采用变分自编码器VAE结构，则码率

和

可以用香农熵估计得到；公式中的λ₁，λ，λ₃属于超参数，λ₁控制模态1 和模态2重建质量之间的折中，即当更希望模态1的信源失真更小时λ₁可以设置比较小，反之亦然；λ₃在模态1和模态2之间进行码率分配，即两种模态总的带宽或者存储空间要求一定，λ₃较大时倾向于模态1码率更大和模态2码率更小，反之亦然；λ用于控制质量和码率之间的折中，通常质量越高所消耗的码率越大，质量越低所消耗的码率越小，即λ用于选取最终的码率点，λ越大则选取的码率点越低，适用于带宽越低的场景，相应的重建质量会越低，反之亦然。Among them, quality ₁ ( , ) and quality ₂ ( , ) are used to measure the quality loss of mode 1 and mode 2 caused by encoding, respectively. For example, for videos or images, PSNR (peak signal-to-noise ratio), MS-SSIM (multi-scale-structural similarity) or perceptual loss can be used to measure;

and

used to measure

and

The number of bits consumed by converting to a binary code stream can usually be estimated. For example, in the above description, it can be assumed that the featc, feati ₁ and feati ₂ types of features obey the Gaussian distribution, and some features in featis ₁ represent the mean value of the Gaussian distribution, and some other features represent the variance, that is, the encoder uses the variational autoencoder VAE structure, the code rate

and

It can be estimated by Shannon entropy; λ ₁ , λ, λ ₃ in the formula belong to hyperparameters, and λ ₁ controls the compromise between the reconstruction quality of mode 1 and mode 2, that is, when the source distortion of mode 1 is more expected When it is smaller, λ ₁ can be set relatively small, and vice versa; λ ₃ performs code rate allocation between mode 1 and mode 2, that is, the total bandwidth or storage space requirements of the two modes are certain, and when λ ₃ is larger, it tends to Because the code rate of mode 1 is larger and the code rate of mode 2 is smaller, and vice versa; λ is used to control the compromise between quality and code rate. Usually, the higher the quality, the greater the code rate consumed, and the lower the quality. The smaller the code rate consumed, that is, λ is used to select the final code rate point, the larger the λ, the lower the selected code rate point, which is suitable for scenarios with lower bandwidth, and the corresponding reconstruction quality will be lower, and vice versa.

Embodiment 2, referring to FIG. 2 , embodiment 2 introduces a knowledge base on the basis of embodiment 1, which can perform joint coding on multi-modal information sources more efficiently.

The knowledge base in Figure 2 can be either multi-modal or unimodal, and the multi-modal knowledge base means that the knowledge base stores different forms of information (usually from different modal sources); Figure 3 uses text and Image as an example gives an example of a multi-modal knowledge base. The multi-modal knowledge base has text and images, represented by nodes and edges. Each node represents an entity or represents text or represents an image. Each edge Represents the relationship between different nodes. For example, Claude Shannon is a guest of the World Computer Chess Championship, where "Claude Shannon" and "WorldComputer Chess Championship" are nodes, and the edge "guestOf" represents the relationship between the two. The image of Claude Shannon is shown in the lower right corner of Figure 3. "Claude Shannon" and its image are connected by the edge "imageOf" with direction; "Deep Thought" participates in the "World Computer Chess Championship" competition, and the two nodes represent "Deep Thought" and "World Computer Chess Championship", the relationship between the two is represented by "attend".

Embodiment 2 introduces the knowledge base on the basis of Embodiment 1. On the basis of Embodiment 1, the information source of mode 1 can obtain the index of the retrieval knowledge base through "analysis of mode 1", and the information source of mode 2 can be obtained through "analysis of mode 1" Mode 2 analysis" can also obtain the index of the retrieval knowledge base, and only one of the two can be used. There are two types of analysis that can retrieve more relevant information from the knowledge base or enhance robustness. For multi-modal information sources The coding efficiency improvement effect is greater. Among them, "modal 1 parsing" and "modal 2 parsing" are mainly for obtaining knowledge base node entities for query and reasoning. After the reasoning and query of the knowledge base, the relevant information can be embedded and encoded by the third encoder D to obtain the features of the knowledge base, and jointly encoded with the source features by the second encoder C to remove the redundancy between the source code and the knowledge base , so as to improve the coding efficiency. Correspondingly, in the decoding process, decoder A and decoder B also need to input knowledge base features to decode the mode 1 and mode 2 sources.

The purpose of the knowledge base introduced in Embodiment 2 is to increase prior knowledge; to explicitly associate information sources of different modalities.

The specific process of embodiment 2 is: a method for joint encoding of multi-modal information sources, including the following steps:

1) Given two modal signal sources "modal 1" and "modal 2", denoted as src ₁ and src ₂ respectively, the two modal signals respectively pass through the first encoder A and the first encoder B to Extract features to remove the internal redundancy of each modal signal, and obtain feature maps feat ₁ and feature maps feat ₂ ;

The information source of mode 1 obtains the index of the retrieval knowledge base through "analysis of mode 1", and the information source of mode 2 obtains the index of the retrieval knowledge base through "analysis of mode 2", where "analysis of mode 1" and "analysis of mode 2 "Analysis" is mainly to obtain knowledge base node entities for query and reasoning; after knowledge base reasoning and query, relevant information is embedded and coded by encoder D to obtain knowledge base features;

2) In order to remove the correlation between different modal signals, the two sets of feature maps are connected and input to the second encoder C, decoupled into common feature maps and individual feature maps; the common feature maps represent the differences between different modal sources The common part of , usually at the semantic level; the personality feature map represents the unique features of each modal source; taking video and audio as examples, the common feature may be the words spoken by the characters in the video, The audio usually also contains this information; the personality characteristics of the video can be the appearance of the characters in the video or other Beijing information other than the characters, such as flowers and plants, and the personality characteristics of the audio may include other irrelevant audio, or the tone that is usually difficult to express in the video ;

In this embodiment, the common and individual features are decoupled, and the individual features feati ₁ of modality 1, the common features featc of the two modalities and the individual features feati ₂ of modality 2 are output, and the second encoder C may include a quantization process to Implement lossy encoding;

The knowledge base feature and the information source feature are jointly encoded by the second encoder C to remove the redundancy of the information source encoding and the knowledge base, thereby improving the encoding efficiency;

feati₁和featc共同输入解码器B，以恢复模态2，记为

3) featc, feati ₁ and feati ₂ three types of features are respectively entropy coded and converted into binary code streams for storage or transmission; the binary code streams at the decoding end are restored to obtain feati ₁ , featc and feati ₂ after entropy decoding; after that feati ₁ and featc are jointly input to decoder A to recover modality 1, labeled as

Feati ₁ and featc are jointly input to decoder B to recover modality 2, denoted as

In the decoding process, Decoder A and Decoder B also need to input knowledge base features to decode Modal 1 and Modal 2 sources.

Embodiment 3, with reference to Fig. 4, embodiment 3 provides a kind of embodiment that introduces knowledge base, and the effect that knowledge base plays is that can obtain in the knowledge base according to " Claude Shannon " keyword query in " text " information source There is no need to encode the image part corresponding to Claude Shannon in the "image" source, so that the image and text can be encoded more efficiently.

With reference to Fig. 4, the input of this embodiment is two kinds of modal information sources of "text" and "image", respectively corresponding to "modal 1" and "modal 2" in Fig. 2 of embodiment 2, in the text information source The "Named Entity Recognition: BERT" corresponds to "Modality 1 Analysis", that is, the BERT technology in the field of natural language processing can be used to analyze the named entities in the text to obtain entity names, such as "Claude Shannon" and "Deep Thought", input to Query and reasoning are performed in the knowledge base, and the knowledge base features are generated after encoding. The features are usually embedded feature vectors; in Figure 4, mode 2 is not analyzed, that is, the "mode 2 analysis" in Figure 2 is not used. For the main branch, the "text" modality passes through a text encoder, such as GRU, which can be encoded into text features, and the "image" modality detects objects in the image and establishes the relationship between them through scene graph generation technology, which is passed through volume The product network generates image feature maps, which are labeled as image features. Afterwards, after the text features and image features are connected, they are sent together with the knowledge base features as input to the second encoder C for encoding to generate text individual features, image individual features, and common features of text and images. Figure 4 does not show the process of losslessly encoding features to generate binary code streams, and decoding the binary code streams to generate corresponding features. In addition, the parsed "entity name" also needs to be encoded and transmitted to the decoding end.

At the decoding end, the text individual features, common features and knowledge base features are used as input together, and the text is output through the text decoder; the image individual features, common features and knowledge base features are jointly used as input and the image output image is output through the image decoder. It can be seen from Figure 4 that by introducing the knowledge base, the encoding end does not need to transmit the part corresponding to Claude Shannon in the image, but only needs to transmit the parsed Claude Shannon entity, and the decoding end can obtain the image corresponding to Claude Shannon in the knowledge base through the knowledge base; In addition, transfer codes "Edmonton" and "1989" are also not required, which can be inferred through knowledge base transfer. The personality characteristics of the image on the encoding side mainly include the clothing, posture and location characteristics of "Feng-hsiung Hsu". Personality features in the text mainly include "Feng-hsiung Hsu" and "first prize"; common features include information such as "Claude Shannon" and "Deep Thought". Therefore, increasing the knowledge base makes coding more efficient. The training process of this embodiment is similar to that of Embodiment 1, and the design of the loss function is also similar.

Claims (3)

1. A method for multi-modal joint source coding, comprising the steps of:

1) A plurality of modal signal sources pass through corresponding first encoders to extract features to remove internal redundancy of each modal signal, and a corresponding feature map is obtained;

2) In order to remove the correlation among different modal signals, connecting a plurality of groups of feature maps, inputting the feature maps into a second encoder, and decoupling the feature maps into a common feature map and an individual feature map; the common characteristic diagram represents a common part among different modality information sources, and the individual characteristic diagram represents a characteristic unique to each modality information source;

3) The individual characteristic graphs and the common characteristic graphs of the plurality of modal information sources are decoded by corresponding decoders and corresponding modal information sources are rebuilt, namely, the individual characteristic graphs and the common characteristic graphs are respectively subjected to entropy coding and converted into binary code streams to be stored or transmitted; and after entropy decoding is carried out on the binary code stream at the decoding end, recovering the binary code stream through corresponding decoders respectively to obtain corresponding modal information sources.

2. The method of claim 1, wherein: introducing a knowledge base, and carrying out joint coding on the multi-mode information source; the knowledge base is multi-mode or single-mode, and the multi-mode knowledge base stores information containing a plurality of different forms from different mode information sources; the single or multiple modal sources obtain indexes for searching the knowledge base through modal analysis, and the modal analysis is used for obtaining knowledge base node entities for query and reasoning.

3. The method of claim 2, wherein: the multi-modal knowledge base has a representation form of text and images which are represented by nodes and edges, wherein each node represents an entity or represents the text or represents the image, and each edge represents the relationship between different nodes.

Images