CN101692712B - Error concealment method based on state of signal channel and visual perception side information - Google Patents

Abstract

The invention relates to the field of digital video signal compression transmission, and discloses an error concealment method based on channel state and visual perception side information, which mainly uses the current channel state information and the visual perception weight map transmitted in the form of side information to calculate the adjacent macro The reliability weight and reconstruction weight of the block are used to cover the error video images of different encoding types in the space domain and time domain respectively, so as to improve the video decoding image quality under error-prone transmission conditions.

Description

An Error Concealment Method Based on Channel State and Visual Perceptual Side Information

technical field

The invention relates to the field of digital video signal compression transmission, in particular to a method for improving video decoding image quality under error-prone transmission conditions by using error concealment technology.

Background technique

The error concealment method is a method for the video decoder to restore the data of the damaged area according to the characteristics of the video image signal itself and improve the quality of the video image after an error occurs in the video code stream during transmission. Video image data is dominated by low-frequency components. Except for violent scene switching and obvious edge areas, the brightness and chrominance values of adjacent video images in the time and space domains change relatively smoothly, so the video image data in the time domain and space domain smoothness. On the other hand, human eyes pay different attention to different video images, and compared with low frequency components, human eyes are less likely to detect distortion of high frequency components. The error concealment method is based on the smoothness of the video image and the visual perception characteristics of the human eye, using data similar to the original video image to cover up the damaged error video image data, slowing down the spread and spread of errors in time and space, thereby improving video decoding. Image Quality. The error concealment method is usually used on the decoding side, does not introduce delay, and does not require additional bandwidth.

After the video code stream is wrong, the video decoder usually adopts an error concealment method based on pixel interpolation and motion compensation to reduce the impact of the code stream error on the quality of the decoded video image. This error concealment method utilizes the smoothness property of the video image in the space domain and the time domain, and reconstructs the current erroneous macroblock according to the information of adjacent macroblocks. These neighboring macroblocks include correctly decoded macroblocks and reconstructed macroblocks with error concealment. Therefore, when using adjacent macroblocks to reconstruct an erroneous macroblock, these adjacent macroblocks should be given different reliability weight values.

Moreover, when the video code stream is wrong due to packet loss or bit error, the current error concealment method simply utilizes the smoothness of the video image without considering the content of the video coded image, so the effect on the details and subjective quality of the video image poor.

Contents of the invention

The present invention provides an error concealment method based on channel state and visual perception side information, which mainly uses the current channel state information and the visual perception weight map transmitted in the form of side information to calculate the reliability weight and reconstruction weight of adjacent macroblocks , respectively perform error concealment processing on the space domain and time domain for error video images of different coding types, and improve the video decoding image quality under error-prone transmission conditions.

An error concealment method based on channel state and visual perception side information, after receiving video code stream and side information data from the network, the video encoding type is intra-frame coding or inter-frame coding error image containing error macroblocks, Include the following steps:

(1) Calculate the reliability weight value of the adjacent macroblock of the erroneous macroblock according to the packet loss rate in the current channel state information;

Since the video code stream and side information are both received from the network, the current network status also affects the accuracy of the read data to a certain extent, and the status information of the current network must be obtained at the same time to realize the error concealment of the present invention method. The current channel state information refers to the state information of the current network.

Wherein the macroblock refers to a square pixel area block with equal length and width; after decoding the read video code stream, the existing technology can be used to find and judge the error macroblock, and then use the error macroblock as the current macroblock Perform error masking.

The adjacent macroblocks refer to macroblocks with borders adjacent to the current macroblock in the four directions above, below, left and right. The distortion of the reconstructed macroblock is inversely proportional to its reliability, the greater the distortion, the lower the reliability, and vice versa. The distortion caused by packet loss or bit error in the video decoding image can be predicted by the packet loss rate, so the reliability of reconstructed macroblocks can also be described by the packet loss rate. On the other hand, the error concealment method needs to use the adjacent macroblock data to reconstruct the erroneous macroblock, so the reliability of the reconstructed macroblock is related to the reliability of the adjacent macroblock.

According to the packet loss rate in the current channel state information, the reliability weight value of the adjacent macroblock is calculated, and its calculation formula is:

W _reliability = avg{W _reliability (dir)}(1-P), dir ∈ {top, left, bottom, right}

(1)

In the formula, W _reliability represents the reliability weight value of the macroblock, dir represents the upper, lower, left and right adjacent macroblocks, and P represents the packet loss rate of the current channel.

In the formula, calculate the average value of the reliability weight values of the upper, lower, left and right adjacent macroblocks, and then multiply it by (1-P) to obtain the reliability weight value of the macroblock.

As far as any adjacent macroblock is concerned, the method for determining its reliability weight value is as follows:

a) If the macroblock data is received correctly (can be judged by the decoder), its reliability weight value is 1;

b) If there is an error in the macroblock data and no error concealment has been performed on it, its reliability weight value is 0;

c) If the macroblock data is wrong, but it has been covered up, its reliability weight value is calculated by formula (1). Here, it will be explained that this is an iterative calculation process, because in the method of the present invention is The read-in macroblocks are processed sequentially. As long as the data of the first macroblock can be received correctly, the subsequent error macroblocks can obtain their reliability weight values through the iterative calculation of formula (1).

(2) Select adjacent macroblocks to participate in reconstruction;

The error concealment method requires sufficient boundary pixel data of adjacent macroblocks. First, it is judged whether the correctly decoded macroblock in the adjacent macroblock satisfies the condition. If the condition is satisfied, only correctly decoded adjacent macroblocks are used to restore the error macroblock; if the condition is not satisfied, the error-concealed macroblock among the adjacent macroblocks is selected to restore the error macroblock. Once the accumulation of reliability weight values W _reliability of adjacent macroblocks exceeds the threshold Δ _reliability , it indicates that the adjacent macroblocks involved in reconstructing the error macroblock are sufficient, and no other adjacent macroblocks need to be added.

This process can be described by formula (2)

$\underset{\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; top</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; left</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; bottom</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; right</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; reliability</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Greater\; Equal;</span>{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; reliability</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

The higher the value of Δ _reliability , the better the effect of error concealment, but the greater the amount of calculation. In general, the value of Δ _reliability can be set to 2.

In this step, when selecting adjacent macroblocks to participate in reconstruction, the macroblocks read in first are given priority in accordance with the sequence when the video code stream is received from the network in step (1).

(3) A visual perceptual weight map is proposed from side information. The visual perception weight map refers to a visual perception weight matrix in units of macroblocks.

When the visual perception weight map is proposed from the side information, the data of each byte in the side information is used as a visual perception weight value, and all the visual perception weight values are combined to obtain the visual perception weight map, wherein each The visual perception weight value corresponds to a macroblock in the visual perception weight matrix.

(4) Calculate the reconstruction weight value of the adjacent macroblock according to the visual perception weight map; the reconstruction weight value refers to the weight value of the adjacent macroblock participating in the reconstruction of the error macroblock in the error concealment.

The visual perception weight of each adjacent macroblock may be different from the visual perception weight of the current error macroblock, that is, the correlation between each adjacent macroblock and the current error macroblock in terms of subjective quality may be different. Therefore, when performing error concealment according to the smoothness of the video image signal, the reconstruction weight value in the error concealment method should be calculated according to the visual perception weight correlation between the adjacent macroblock and the current error macroblock. When the difference between the visual perception weight value of the adjacent macroblock and the current error macroblock is small, the reconstruction weight value of the adjacent macroblock in the error concealment method is larger; otherwise, the visual perception weight value of the adjacent macroblock and the current error macroblock When the value difference is large, the reconstruction weight value of adjacent macroblocks in the error concealment method is small.

(5) Analyze the video code stream and obtain the video encoding type of the wrong image, such as intra-frame encoding or inter-frame encoding;

(6) If the error image type is intra-frame coding, a space-domain error concealment method using pixel weighted interpolation;

Utilize the redundancy in the space domain of video image data, select the appropriate adjacent macroblocks, use pixels as the unit, reconstruct the weight value and the distance weight value according to the adjacent macroblocks, and use the interpolation method to realize the error concealment method in the space domain.

If the error image type is inter-frame coding, a time-domain error concealment method based on boundary distortion matching is adopted;

Using the redundancy of video image data in the time domain, the error video image data of the current frame is reconstructed according to the video image data of the reference frame, and the most suitable motion vector is selected from the motion vectors of adjacent macroblocks participating in the reconstruction as the error macroblock. Motion vector, the selection criterion is to minimize the boundary matching distortion of the luminance component.

According to the channel state information of the currently transmitted video code stream and the content of the video coded image, the present invention selects the adjacent macroblock data involved in reconstructing the error image and adjusts its weight value, so that the error macroblock can be accurately reconstructed and the error concealment effect is improved.

Description of drawings

Fig. 1 is the flowchart of the error concealment method of the present invention;

Detailed ways

The error concealment method based on the channel state and visual perception side information of the present invention will be described in detail below with reference to the accompanying drawings.

Referring to Figure 1, the video code stream and side information data are received from the network, the visual perception map is obtained from the side information, and the reconstruction weight value of the adjacent macroblock is calculated according to the visual perception weight map; the reliability weight value of the adjacent macroblock is calculated according to the channel Calculate the packet loss rate in the state information; select the adjacent macroblocks involved in the reconstruction according to the reliability weight value; obtain the video coding type of the error image from the video decoder; according to the video coding type of the error image, use the adjacent macroblock Data, the spatial domain error concealment based on pixel weighted interpolation and the time domain error concealment based on boundary matching distortion motion compensation are respectively performed.

The error concealment method based on channel state and visual perception side information, after receiving the video code stream and side information data from the network, processes the erroneous image containing the erroneous macroblock whose video coding type is intra-frame coding or inter-frame coding, Include the following steps;

(1) Calculate the reliability weight value of the adjacent macroblock of the erroneous macroblock according to the packet loss rate in the channel state information; wherein said macroblock refers to a square pixel area block whose length and width are 16 pixels; said Adjacent macroblocks refer to the macroblocks with adjacent borders in the four directions of the current macroblock, up, down, left, and right;

The calculation of the reliability weight value is shown in formula (1):

W _reliability = avg{W _reliability (dir)}(1-P), dir ∈ {top, left, bottom, right}

(1)

In the formula, W _reliability represents the reliability weight value of the reconstructed macroblock, dir represents the upper, lower, left and right adjacent macroblocks, and P represents the packet loss rate of the current channel.

As far as any adjacent macroblock is concerned, the method for determining its reliability weight value is as follows:

a) If the macroblock data is received correctly (can be judged by the decoder), its reliability weight value is 1;

b) If there is an error in the macroblock data and no error concealment has been performed on it, its reliability weight value is 0;

c) If the macroblock data is wrong, but it has been covered up, its reliability weight value is calculated by formula (1). Here, it will be explained that this is an iterative calculation process, because in the method of the present invention is The read-in macroblocks are processed sequentially. As long as the data of the first macroblock can be received correctly, the subsequent error macroblocks can obtain their reliability weight values through the iterative calculation of formula (1).

(2) Select the adjacent macroblocks involved in reconstruction according to the reliability weight;

A reliability weight accumulation threshold Δ _reliability is set. Once the accumulation of reliability weight values of adjacent macroblocks exceeds Δ _reliability , it indicates that the adjacent macroblocks involved in reconstructing the error macroblock are enough. In the method of the present invention, the value of Δ _reliability is 2. This process can be described by formula (2):

$\underset{\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; top</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; left</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; bottom</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; right</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; reliability</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Greater\; Equal;</span>{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; reliability</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

(3) Propose a visual perception weight map from side information;

When the visual perception weight map is proposed from the side information, each byte of data in the side information is used as a visual perception weight value, and all the visual perception weight values are combined to obtain a visual perception weight map, in which each visual perception weight The value corresponds to a macroblock in the visual perception weight matrix.

(4) Calculate the reconstruction weight value of the adjacent macroblock according to the visual perception weight map;

The calculation of the reconstruction weight value is shown in formula (3):

${\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; reconstructed</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; perceptual</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; perceptual</span>}}}{{\mathrm{<span\; class="notranslate">\; MAX</span>}}_{\mathrm{<span\; class="notranslate">\; perceptual</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; top</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; left</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; bottom</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; right</span>}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

In the formula, dir represents the four adjacent macroblocks up, down, left, and right, but in step (2) according to the reliability weight, when selecting adjacent macroblocks to participate in reconstruction, it may not be up, down, left, and right Four adjacent macroblocks all participate in reconstruction, so dir refers to the adjacent macroblocks involved in reconstruction determined in step (2) when calculating here;

W _{reconstructed} is the reconstruction weight value;

W _perceptual represents the visual perception weight value obtained from the visual perception weight map obtained in step (3);

MAX _perceputal represents the maximum visual perception weight value of the visual perception weight map obtained from step (3).

(5) Analyze the video code stream and obtain the video encoding type of the wrong image, such as intra-frame encoding or inter-frame encoding;

(6) If the error image type is intra-frame coding, a space-domain error concealment method using pixel weighted interpolation;

1) First, calculate the distance weight value of the adjacent macroblock according to the distance from the boundary pixel of the adjacent macroblock to the error pixel, as shown in formula (4)

$\underset{\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; top</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; left</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; bottom</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; right</span>}<span\; class="notranslate">\; \}</span>}{{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; dis</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; ce</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; SIZE</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

In the formula:

W _distance represents the distance weight value of adjacent macroblocks;

SIZE is the width of the macroblock where the error pixel is located, in pixels;

d is the distance between the error macroblock pixel and the boundary pixel of the adjacent macroblock;

dir represents four adjacent macroblocks up, down, left, and right, but in step (2) according to the reliability weight, when selecting adjacent macroblocks to participate in reconstruction, it may not be the four adjacent macroblocks up, down, left, and right Neighboring macroblocks all participate in reconstruction.

The adjacent macroblock boundary pixels refer to SIZE pixels on the border between the adjacent macroblock and the error macroblock, and SIZE is the width of the macroblock, in units of pixels;

The distance from the boundary pixel of the adjacent macroblock to the error pixel refers to the number of pixels between the boundary pixel of the adjacent macroblock and the error pixel;

2) Then combine the distance weight value and the reconstruction weight value to realize error concealment in the spatial domain through interpolation, as shown in formula (5):

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; conceal</span>}}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; top</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; left</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; bottom</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; right</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; dis</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; ce</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; reconstructed</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span>}{\underset{\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; top</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; left</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; bottom</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; right</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; dis</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; ce</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; reconstructed</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

In the formula:

Y _conceal represents the pixels of the error macroblock reconstructed by the error concealment method;

W _{reconstructed} represents the reconstruction weight value of adjacent macroblocks;

W _distance represents the distance weight value of adjacent macroblocks;

Y represents the pixel value of adjacent macroblocks;

dir represents four adjacent macroblocks up, down, left, and right, but in step (2) according to the reliability weight, when selecting adjacent macroblocks to participate in reconstruction, it may not be the four adjacent macroblocks up, down, left, and right Neighboring macroblocks all participate in reconstruction, so dir specifically refers to the adjacent macroblocks involved in reconstruction determined in step (2) when calculating here.

(7) If the error image type is inter-frame coding, a time-domain error concealment method based on boundary distortion matching is adopted;

1) Calculate the boundary matching distortion of the adjacent macroblocks participating in the reconstruction determined in step (2), as shown in formula (6):

$\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; dir</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; top</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; left</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; bottom</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; right</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; SIZE</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; SIZE</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; reconstructed</span>}}}^{\mathrm{<span\; class="notranslate">\; dir</span>}}{<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; Y</span>}}^{\mathrm{<span\; class="notranslate">\; dir</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; conceal</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

In the formula:

D is the boundary matching distortion;

SIZE is the width of the macroblock where the error pixel is located;

W _{reconstructed} represents the reconstruction weight value of adjacent macroblocks;

Y represents the pixel luminance value on the boundary adjacent to the error macroblock among the adjacent macroblocks participating in the reconstruction;

i represents the index (serial number) of the pixel on the boundary adjacent to the erroneous macroblock among the adjacent macroblocks participating in the reconstruction;

Y _conceal indicates that when the motion vector of the error macroblock selects the motion vector of the adjacent macroblock in the dir direction, the pixel brightness value on the boundary of the macroblock it points to (for example, when dir indicates the adjacent macroblock on the left side of the error macroblock , Y _conceal represents the brightness value of the pixel on the left border in the error macroblock).

dir represents four adjacent macroblocks up, down, left, and right, but in step (2) according to the reliability weight, when selecting adjacent macroblocks to participate in reconstruction, it may not be the four adjacent macroblocks up, down, left, and right Neighboring macroblocks all participate in reconstruction, so dir specifically refers to the adjacent macroblocks involved in reconstruction determined in step (2) when calculating here.

2) Compare the boundary matching distortion of adjacent macroblocks participating in the reconstruction, select the corresponding adjacent macroblock when the boundary matching distortion value is the smallest, and use the image data pointed to by the motion vector of the adjacent macroblock as the current error macroblock Reconstruct data and perform error masking on error macroblocks.

Claims (4)

1. An error concealment method based on channel state and visual perception side information. After receiving video code stream and side information data from the network, the video encoding type is intra-frame coding or inter-frame coding error image containing error macroblocks. Processing is characterized in that it comprises the following steps: (1) Calculate the reliability weight value of the adjacent macroblock of the erroneous macroblock according to the packet loss rate in the current channel state information, and its calculation formula is: W _reliability = avg{W _reliability (dir)}(1-P), dir ∈ {top, left, bottom, right} (1) In the formula, W _reliability represents the reliability weight value of the macroblock, dir represents the upper, lower, left and right adjacent macroblocks, and P represents the packet loss rate of the current channel; in the formula, the upper, lower, left and right The average value of the reliability weight value of four adjacent macroblocks is multiplied by (1-P) to obtain the reliability weight value of the macroblock; As far as any adjacent macroblock is concerned, the method for determining its reliability weight value is as follows: a) If the macroblock data is received correctly, its reliability weight value is 1; b) If there is an error in the macroblock data and no error concealment has been performed on it, its reliability weight value is 0; c) If the macroblock data is wrong, but the error has been concealed, its reliability weight value is calculated by formula (1); (2) According to the reliability weight, select the adjacent macroblocks to participate in the reconstruction, the calculation method is: Set the reliability weight value accumulation threshold Δ _reliability , when the reliability weight value accumulation of adjacent macroblocks exceeds Δ _reliability , stop selecting adjacent macroblocks participating in reconstruction; (3) Propose the visual perception weight map from the side information, the method is: use the data of each byte in the side information as a visual perception weight value, and combine all the visual perception weight values to obtain the visual perception weight map; (4) Calculate the reconstruction weight value of the adjacent macroblock according to the visual perception weight map; (5) parsing the video code stream and obtaining the video coding type of the wrong image therefrom; If the video coding type of the error image is intra-frame coding, combined with the reconstruction weight values of adjacent macroblocks, a spatial domain error concealment method of pixel weighted interpolation is used; If the video coding type of the error image is inter-frame coding, combined with the reconstruction weight values of adjacent macroblocks, a time-domain error concealment method based on boundary distortion matching is adopted. the 2. error concealment method as claimed in claim 1, is characterized in that: calculate the reconstruction weight value of adjacent macroblock according to the visual perception weight map described in step (4), its computing formula is:

In the formula: W _{reconstructed} is the reconstruction weight value; dir indicates the adjacent macroblocks participating in the reconstruction; W _perceptual represents the visual perception weight value obtained from the visual perception weight map obtained in step (3); MAX _perceputal represents the maximum visual perception weight value of the visual perception weight map obtained from step (3). 3. error concealment method as claimed in claim 1, is characterized in that: the process of adopting the space domain error concealment method of pixel weighted interpolation described in step (5) is; 1) First, calculate the distance weight value of the adjacent macroblock according to the distance from the boundary pixel of the adjacent macroblock to the error pixel, as shown in formula (4)

In the formula, W _distance represents the distance weight value of adjacent macroblocks, dir represents the participation in reconstruction of adjacent macroblocks, SIZE is the width of the macroblock where the error pixel is located, in pixels, and d is the boundary between the error macroblock pixel and the adjacent macroblock pixel distance; 2) Then combine the distance weight value and the reconstruction weight value to realize error concealment in the spatial domain through interpolation, as shown in formula (5):

In the formula: Y _conceal represents the pixels of the error macroblock reconstructed by the error concealment method; W _{reconstructed} represents the reconstruction weight value of adjacent macroblocks; W _distance represents the distance weight value of adjacent macroblocks; Y represents the pixel value of an adjacent macroblock; dir represents the adjacent macroblocks involved in reconstruction. 4. The error concealment method as claimed in claim 1, characterized in that: the process of adopting the time domain error concealment method based on boundary distortion matching described in step (5) is; 1) Calculate the boundary matching distortion of adjacent macroblocks in the up, down, left and right directions, as shown in formula (6):

In the formula: D is the boundary matching distortion; SIZE is the width of the macroblock where the error pixel is located; W _{reconstructed} represents the reconstruction weight value of adjacent macroblocks; Y represents the pixel luminance value on the boundary adjacent to the error macroblock among the adjacent macroblocks participating in the reconstruction; i represents the index of the pixel on the boundary adjacent to the error macroblock among the adjacent macroblocks participating in the reconstruction; Y _conceal indicates that when the motion vector of the wrong macroblock selects the motion vector of the adjacent macroblock in the dir direction, the pixel brightness value on the boundary of the macroblock it points to, and the reconstruction data of the wrong macroblock is selected from these data; dir indicates the adjacent macroblocks participating in the reconstruction; 2) Compare the boundary matching distortion of adjacent macroblocks participating in the reconstruction, select the corresponding adjacent macroblock when the boundary matching distortion value is the smallest, and use the image data pointed to by the motion vector of the adjacent macroblock as the current error macroblock Reconstruct data and perform error masking on error macroblocks. the