JP4624308B2 - Moving picture decoding apparatus and moving picture decoding method - Google Patents

Description

  The present invention relates to a moving image decoding apparatus and a moving image decoding method for inputting moving image data that has been digitally compressed and encoded to restore a moving image signal, and particularly when there is an error in the input moving image data. The present invention relates to a moving image decoding apparatus and a moving image decoding method having a function of suppressing deterioration in reproduction image quality.

  Conventionally, MPEG (Moving Picture Experts Group) and ITU-T (International Telecommunication Union-Telecommunication Standardization Sector). In an international standard video encoding system such as 26x, for each frame of a video signal, block data (hereinafter referred to as a macroblock) in which a luminance signal of 16 × 16 pixels and a corresponding color difference signal of 8 × 8 pixels are collected is used as a unit. A compression method based on a motion compensation technique and an orthogonal transform / transform coefficient quantization technique is employed. When video data compressed and encoded based on such a standard method is transmitted for broadcasting / communication purposes, bit errors are mixed in the video data included in the bitstream due to congestion on the transmission path or radio wave interference. There are things to do.

  In general, in moving picture coding, each piece of coded data included in a bit stream is represented by a variable length code by entropy coding designed based on the occurrence probability. For this reason, due to bit errors, not only the encoded data existing at the bit position where the error has occurred, but also the subsequent encoded data cannot be normally decoded. As a result, image quality degradation occurs such that an abnormal reproduced image far from the encoded signal is reproduced or a decoded image is lost.

  For the purpose of suppressing such image quality degradation as much as possible, an error concealment technique for concealing the degraded portion using a video signal that has been successfully decoded in the past has been developed. In general, the image content between adjacent frames of a moving image is often similar, and the motion compensation technique stores a signal of a past encoded frame in a frame memory and refers to it to predict in the time direction. To increase the compression efficiency. In order for motion compensation to function, it is necessary for the video decoding device side to hold the exact same signal as the content of the frame memory used in the video encoding device. In other words, even if a video signal deteriorates due to a bit error in a certain frame, a normal video signal that is very similar to the frame in which the bit error has occurred may be stored in the frame memory in the video decoding device. High nature.

  As an error concealment technique focusing on this point, in the technique shown in Non-Patent Document 1, a plurality of motion vector candidates for an image block in which an error has occurred are estimated based on the surrounding information, and the estimated motions are estimated. For each vector, a candidate for a repaired image is extracted from the corresponding location in the frame memory.

  FIG. 9 is a diagram for explaining candidate repair images. In FIG. 9, it is assumed that a block with a diagonal line at the center is a block that could not be normally decoded due to an error, including the motion vector at that position, and that the upper, lower, left, and right blocks could be normally decoded.

At this time, the four motion vectors (v _up , v _right , v _left , v _down ) are used as the motion vector candidate values of the central block, and each is regarded as a motion vector of the central block. Get image candidates.

  Next, among the plurality of obtained repair image candidates, the repair image candidate having the highest spatial signal continuity with the normally decoded neighboring blocks around the center block is adopted as the final repair image. To do.

W.-M.Lam, A.R.Reibman, and B.Liu, “Recovery of lost or erroneously received motion vectors,” in Proc.ICASSP, vol.5,1993, pp.417-420.

  The conventional moving image decoding apparatus is configured as described above, and the technique of Non-Patent Document 1 is known to have good error concealment performance with respect to a video signal having a high correlation in the time direction. For each of the motion vector candidates, it is necessary to extract a repaired image candidate from the frame memory, and it is not possible to perform error concealment processing that is efficient and error-resistant with less processing, and the required memory bandwidth per macroblock There is a problem that the reproduction quality cannot be efficiently maintained under the severe power consumption conditions such as broadcasting / communication for mobiles.

  The present invention has been made in order to solve the above-described problems. The motion vector reliability is evaluated in advance for a plurality of motion vector candidates, and the motion vector candidates for extracting the repaired image are determined as reliability. By narrowing down to a higher one, efficient and error-resistant error concealment processing can be performed with less processing, and the required memory bandwidth required per macroblock can be reduced. An object of the present invention is to obtain a moving picture decoding apparatus and a moving picture decoding method capable of efficiently maintaining reproduction quality under severe power conditions.

  The moving picture decoding apparatus according to the present invention extracts encoded data of a predetermined area from a bit stream that is encoded by dividing each frame of the moving picture into predetermined areas, and the decoding failure occurs in the process of extracting the encoded data. A variable-length decoding unit that detects a predetermined region to be repaired and detects a prediction residual image from encoded data of the predetermined region extracted by the variable-length decoding unit and generates a decoded image of the predetermined region An image restoration context for generating predetermined image decoding context information by inputting a predetermined region decoding processing unit, encoded data of a predetermined region from the variable length decoding unit, and a prediction residual image from the predetermined region decoding processing unit The validity of a predetermined area around the predetermined area to be repaired specified by the information generation section and the variable length decoding section is determined, and the image repair is performed from the predetermined predetermined area around the area determined to be effective. The image restoration context information generated by the image context information generation unit is acquired, an estimated motion vector used for repairing the predetermined area to be repaired is obtained, and a decoded image is obtained from a frame that has already been decoded based on the obtained estimated motion vector. An image restoration processing unit that obtains a repaired image by acquisition and a decoded image generated by the predetermined region decoding processing unit or the image restoration processing unit based on presence / absence of detection of decoding failure of the predetermined region by the variable length decoding unit And a selection unit that selects and outputs the generated repair image.

  With this invention, it is possible to perform error concealment processing that is efficient and error-resistant with less processing, and can reduce the required memory bandwidth required per macroblock. Below, the effect that the reproduction quality can be efficiently maintained is obtained.

Embodiment 1 FIG.
The moving picture decoding apparatus according to the first embodiment uses MPEG-4 AVC (Advanced Video Coding) / ITU-T H.264 as an input. An apparatus for receiving a moving image encoded bit stream compliant with the H.264 (hereinafter, AVC) standard and reproducing a moving image signal. In the first embodiment, each frame image of a moving image is divided into slice units, and further, moving image compressed data composed of a plurality of macroblock data in which a slice is variable-length encoded is input, A moving picture decoding apparatus that obtains a reproduced image while repairing image quality deterioration due to errors occurring in compressed data will be described.

  The feature of the moving picture decoding apparatus according to this embodiment is that a decoding failure due to an error is detected in a slice, macroblocks in which image quality deterioration has occurred due to a decoding failure are identified, and image data restoration of those macroblocks is performed with a neighboring macro The point is to adaptively execute based on the decoding state of the block.

  FIG. 1 is a block diagram showing a configuration of a moving picture decoding apparatus according to Embodiment 1 of the present invention. The moving image decoding apparatus includes a variable length decoding unit 1, a memory 2, a macroblock decoding processing unit (predetermined region decoding processing unit) 3, an image restoration context information generation unit 4, an image restoration processing unit 5, a switch (selection unit). 6) A deblocking filter 7 and a frame memory 8 are provided.

  In FIG. 1, a variable-length decoding unit 1 extracts encoded data 102 of a macroblock based on a bitstream syntax from a bitstream 101 that is encoded by dividing each frame of a moving image into macroblocks (predetermined areas). In addition, when a decoding failure is detected in the process of extracting the encoded data 102 of the macroblock, the macroblock to be repaired is specified, the decoding failure detection flag 103 is enabled, and the macroblock position information to be repaired 104 and image restoration execution instruction information 105 are output.

  The memory 2 stores an error repair status map 106 based on the macroblock position information 104 to be repaired from the variable length decoding unit 1.

  The macroblock decoding processing unit 3 decodes the prediction residual image 107 from the encoded data 102 of the macroblock extracted by the variable length decoding unit 1 and performs prediction using the reference image 113 stored in the frame memory 8 An image is generated, and a first decoded image 108 of the macroblock is generated from the decoded prediction residual image 107 and the generated prediction image.

  The image restoration context information generation unit 4 receives the macroblock encoded data 102 from the variable length decoding unit 1 and the prediction residual image 107 from the macroblock decoding processing unit 3, and uses image restoration processing for image restoration processing. Context information 109 is generated. In this case, the image restoration context information generation unit 4 generates the image restoration context information 109 regarding the motion vector, the reference image index, and the prediction error amount from the encoded data 102 of the macroblock.

  The image restoration processing unit 5 refers to the error restoration status map 106 based on the image restoration execution instruction information 105 from the variable length decoding unit 1, determines an area including an area to be restored as an image restoration area, The position of the target macroblock to be repaired is determined in the determined image repair area, the macroblocks around the determined macroblock to be repaired are determined to be valid / invalid, and the surrounding macroblocks determined to be valid The image restoration context information 109 generated by the image restoration context information generation unit 4 is obtained, and an estimated motion vector used for restoration of the macroblock to be restored is obtained, and the decoding has already been performed based on the obtained estimated motion vector. The decoded image 114 is acquired from the immediately preceding frame in the frame memory 8 and the repaired image 110 is generated.

  When the decoding failure detection flag 103 from the variable length decoding unit 1 indicates no failure (= 0), the switch 6 selects the first decoded image 108 from the macroblock decoding processing unit 3 When output as the second decoded image 111 and the decoding failure detection flag 103 indicates failure (= 1), the repaired image 110 from the image repair processing unit 5 is selected and the second decoded image 111 is selected. Output as.

  The deblocking filter 7 inputs the second decoded image 111 from the switch 6, refers to the error correction situation map 106, and corresponds to the first decoded image 108 or the repaired image 110 as the second decoded image 111. The deblocking filter processing that has been performed is executed, and the final decoded image 112 is output.

  The frame memory 8 stores the final decoded image 112 from the deblocking filter 7 for use in generating a predicted image of a subsequent frame, and outputs it as a reference image 113 based on an instruction from the macroblock decoding processing unit 3. Based on the instruction of the repair processing unit 5, the decoded image 114 is output from the immediately preceding frame.

  FIG. 2 is a block diagram showing an internal configuration of the macroblock decoding processing unit 3. The macroblock decoding processing unit 3 includes an inverse quantization unit 31, an inverse integer conversion unit 32, a predicted image generation unit 33, and an adder 34.

  In FIG. 2, encoded data 102 of a macroblock input from the variable length decoding unit 1 to the macroblock decoding processing unit 3 includes encoding mode information 201, intra prediction mode information 202, a motion vector 203, and a reference picture index 204. , Integer transform coefficient data 205 representing a prediction residual signal, quantization step parameter 206, and the like are included.

  The inverse quantization unit 31 receives the integer transform coefficient data 205 representing the prediction residual signal and the quantization step parameter 206, inversely quantizes the integer transform coefficient data 205 based on the quantization step parameter 206, and performs inverse quantization. The integer conversion coefficient data 207 is output.

  The inverse integer transform unit 32 performs inverse integer transform on the integer transform coefficient data 207 after inverse quantization from the inverse quantization unit 31 to decode the prediction residual image 107.

  When the coding mode information 201 indicates the inter-frame motion prediction mode (inter mode), the predicted image generation unit 33 stores the frame image in the frame memory 8 based on the coding mode information 201, the motion vector 203, and the reference picture index 204. The prediction image 208 is generated using the reference image 113, and when the encoding mode information 201 indicates the intra-frame encoding mode (intra mode), the prediction image 208 is generated based on the intra prediction mode information 202.

  Since the prediction residual image 107 from the inverse integer transform unit 32 is a prediction residual image obtained as a result of intra prediction or interframe motion compensation prediction, the addition unit 34 is generated according to the coding mode information 201. The predicted image 208 is added to the prediction residual image 107 from the inverse integer transform unit 32 to generate the first decoded image 108.

  FIG. 3 is a block diagram showing the internal configuration of the image restoration processing unit 5. The image restoration processing unit 5 includes an image restoration area determining unit 51, a restoration target macroblock determining unit 52, a peripheral block validity / invalidity determining unit 53, a motion vector estimating unit 54, and a repaired image generating unit 55.

  In FIG. 3, the image repair area determination unit 51 refers to the error repair status map 106 stored in the memory 2 and determines a circumscribed square area including the area to be repaired as the image repair area 211.

  The restoration target macroblock determination unit 52 determines a restoration target macroblock to be restored in the image restoration area 211 as the current macroblock 212.

The peripheral block validity / invalidity determination unit 53 refers to the error repair status map 106 relating to the current frame, determines the validity or invalidity of the surrounding macroblocks located above, below, left and right of the current macroblock 212, and validates the surrounding macroblocks. The invalid state 213 is output. In this case, the peripheral block validity / invalidity determination unit 53 determines that the validity of the peripheral macroblock is “valid” when it is not a restoration target and when it is not out of the screen, and “invalid” otherwise. Is determined.

  The motion vector estimation unit 54 refers to the motion vector and reference image index from the image restoration context information generation unit 4 for the macroblock determined to be “valid” with reference to the valid / invalid state 213 from the neighboring block valid / invalid determination unit 53. Then, the image restoration context information 109 relating to the prediction error amount is acquired, and an estimated motion vector 214 used for restoration of the current macroblock 212 is obtained. In this case, the motion vector estimation unit 54 scales the motion vector based on the reference image index, and weights and averages the scaled motion vector using a weighting coefficient determined based on the prediction error amount. An estimated motion vector 214 used to repair a predetermined area is obtained.

  The repair image generation unit 55 acquires the decoded image 114 from the immediately preceding frame in the frame memory 8 based on the estimated motion vector 214 and generates the repair image 110.

  FIG. 4 is a flowchart showing a process flow of the moving picture decoding apparatus according to Embodiment 1 of the present invention. In FIG. 4, steps ST11 to ST13 indicate variable length decoding processing (A), steps ST14 to ST16 indicate macroblock decoding processing (B) when no decoding failure is detected, and steps ST17 to ST18 detect decoding failure. In this case, the macroblock decoding process (C) is shown, and steps ST19 to ST20 show the image restoration process (D).

First, the variable length decoding process (A) will be described.
The video decoding device receives a bit stream 101 that is a video encoding bit stream compliant with the AVC standard in units of slices, and the variable length decoding unit 1 decodes slice header information from the bit stream 101 in step ST11. The slice header information includes the position information on the screen of the first macroblock of the slice, the quantization parameter reference value of the macroblock included in the slice, and the like. At this time, the variable length decoding unit 1 invalidates the decoding failure detection flag 103 to the switch 6 and resets it to “0”.

  The process then proceeds to the individual macroblock decoding process. In each macroblock decoding process, first, in step ST12, the variable length decoding unit 1 analyzes the bitstream 101 according to the bitstream syntax of the AVC standard, and extracts and restores the encoded data 102 of the macroblock. The macroblock coded data 102 includes coding mode information 201, intra prediction mode information 202, motion vector 203, reference picture index 204, integer transform coefficient data 205 representing a prediction residual signal, quantization step parameter 206, and the like. It is included.

Further, the variable length decoding unit 1 detects a decoding failure as will be described later in the process of analyzing the bitstream 101. Decoding failure refers to a state in which normal bitstream analysis cannot be continued until decoding synchronization is restored again. Examples of the decoding failure to be detected include the following.
(A) In the AVC standard, an undefined codeword or an undefined decoded value that cannot be generated is detected in the bitstream syntax. (B) In the AVC standard, the decoded value is fixed in the bitstream syntax. (C) In the AVC standard, in the AVC standard, the range of the decoded value is determined based on the data-specific value or under the restriction of the decoded value of another data. (D) When the decoded data is clearly inconsistent with the previous decoding process, for example, within the screen of the first macroblock of a slice As a result of decoding the position, when the value is discontinuous with the position in the screen of the previously decoded slice, etc. (e) Decoded in the AVC standard When a limit is set on the number of similar data to be decoded, the number of decoded data exceeds that number. For example, integer conversion employed in AVC is applied to a 4 × 4 or 8 × 8 pixel block. Therefore, only 16 or 64 integer transform coefficients are generated per block, but this number is exceeded during decoding.

  In step ST13, the variable length decoding unit 1 determines whether or not the above-described decoding failure is detected in the process of analyzing the bitstream 101, and switches the processing flow according to the determination result. As long as no decoding failure is detected during the decoding process of the slice, the processing of step ST14 described in the macroblock decoding process (B) in the case where a decoding failure described later is not detected continues to be executed for each individual macroblock. When a decoding failure is detected in such a macroblock, steps ST17 and ST18 described in the macroblock decoding process (C) when a decoding failure described later is detected are executed. When the processing for one frame is completed, the restoration of the part where the decoding failure occurred in step ST19 described in the image restoration process (D) described later and the deblocking filter process in step ST20 are executed.

  In the configuration of the moving picture decoding apparatus shown in FIG. 1, the first generated by the switcher 6 in the process (B) described later depending on whether or not a decoding failure is detected based on the decoding failure detection flag 103. Although it is illustrated that the second decoded image 111 is determined by switching the decoded image 108 and the repaired image 110 generated by the process (C) described later, this is related to whether or not a decoding failure is detected. It does not mean that both the first decoded image 108 and the repaired image 110 are generated, but is illustrated in this way for the sake of clarity in describing the outline of the present invention. When executing the process (B) described later, among the components shown in FIG. 1, the process in the image repair processing unit 5 is not performed and the repaired image 110 is not generated. In addition, when transitioning to processing (C) described later, a macroblock that is determined as a repair target in the variable-length decoding unit 1 (a macroblock that is a target for which the image repair execution instruction information 105 is “repair execution”) 2), the processing in the inverse quantization unit 31, the inverse integer transform unit 32, and the predicted image generation unit 33 in the internal configuration of the macroblock decoding processing unit 3 in FIG. Is not generated.

Next, the macroblock decoding process (B) when no decoding failure is detected will be described.
When the decoding failure as described above is not detected, the variable length decoding unit 1 keeps the decoding failure detection flag 103 at “0”. The specific value of the decryption failure detection flag 103 may be determined in any way as long as the presence or absence of the decryption failure can be identified.

  In step ST14 of FIG. 4, the macroblock decoding processing unit 3 generates a first decoded image 108 of the macroblock using the extracted encoded data 102 of the macroblock. For this process, functional blocks such as an inverse quantization unit 31, an inverse integer transform unit 32, and a predicted image generation unit 33 in the configuration of the macroblock decoding processing unit 3 shown in FIG. 2 are used. First, the integer transform coefficient data 205 and the quantization step parameter 206 are sent to the inverse quantization unit 31, and after the inverse transform of the integer transform coefficient data 205 is performed, the inverse residual transform unit 32 performs the prediction residual image 107. Is decrypted.

  When the encoding mode information 201 indicates the inter-frame motion prediction mode (inter mode), the prediction image generation unit 33 stores the frame in the frame memory 8 based on the encoding mode information 201, the motion vector 203, and the reference picture index 204. On the other hand, when the prediction image 208 is generated and output using the reference image 113, and the coding mode information 201 indicates the intra-frame coding mode (intra mode), the prediction image 208 is predicted based on the intra prediction mode information 202. 208 is generated and output. Since the prediction residual image 107 output from the inverse integer transform unit 32 is a prediction residual image signal obtained as a result of performing intra prediction or interframe motion compensation prediction, a prediction image generated according to the coding mode information 201 is used. 208 is added to the prediction residual image 107 from the inverse integer transform unit 32 in the adding unit 34 to generate and output the first decoded image 108.

  As long as the decoding process of (B) is executed, the decoding failure detection flag 103 is maintained in the state of “no decoding failure”. Therefore, the switch 6 uses the first decoded image 108 as it is. The decoded image 111 is output.

  In step ST15 of the decoding process (B), the image restoration context information generation unit 4 generates image restoration context information 109 used for an image restoration process described later. In FIG. 4, it is described that the process of step ST15 is performed following step ST14, but actually step ST15 may be performed while steps ST12 and ST14 are being executed. The image restoration context information 109 is extracted or generated by the image restoration context information generation unit 4 and written in a storage area included in the image restoration context information generation unit 4, in response to a request from the image restoration processing unit 5. Is output. The image restoration context information 109 is information referred to by an image restoration algorithm (described later) used in the image restoration processing unit 5. In the first embodiment, the image restoration context information 109 for one frame is replaced with information in units of 8 × 8 blocks and held in the storage area. Of course, it may be configured so as to be held in units of 16 × 16 macroblocks according to the image restoration algorithm to be used.

The individual contents of the image restoration context information 109 will be described below.
○ Motion vector context information ctx_mv
When the encoding mode information 201 indicates an inter-frame motion prediction mode (inter mode), a predicted image 208 is generated based on the motion vector 203. In AVC, the motion vector 203 may be 16 × 16 block unit information or 4 × 4 block unit information. If it is a motion vector 203 for a block larger than 8 × 8, the same value is assigned as motion vector context information ctx_mv to all 8 × 8 blocks included in the region. In the case of the motion vector 203 for a block smaller than the 8 × 8 block, the motion vector context information ctx_mv in units of 8 × 8 blocks is obtained by averaging the portions covering the 8 × 8 block area or taking the median value. When the encoding mode information 201 indicates a skip macroblock, the motion vector 203 for the skip macroblock is generated according to the definition of the skip macroblock of the AVC standard, and is used as the motion vector context information ctx_mv. When the coding mode information 201 indicates the intra mode, motion vector context information ctx_mv obtained by a method equivalent to zero or a skip macroblock is used.

○ Reference image index context information ctx_refidx
In the AVC standard, a plurality of reference images 113 are stored in the frame memory 8, and the reference image 113 used for predictive image generation is changed for each block to which the motion vector 203 is applied (however, a block larger than the 8 × 8 block size). it can. Since which reference image 113 is used for predictive image generation needs to be transmitted to the video decoding device, a reference picture index 204 is assigned to each bitstream 101 for each motion vector 203. In the first embodiment, the reference picture index 204 is read in units of 8 × 8 blocks and stored as reference image index context information ctx_refidx. When the encoding mode information 201 indicates the intra mode, it is set to zero, that is, an index value indicating the immediately preceding frame.

○ Prediction error amount context information ctx_err
For the prediction residual image 107 that is the output of the inverse integer transform unit 32, a scalar value obtained by adding the absolute value of each pixel value in units of 8 × 8 blocks is generated and held as prediction error amount context information ctx_err. When the encoding mode information 201 indicates the intra mode, the prediction error amount context information ctx_err is set to a fixed value C because it is not a prediction error due to the motion vector 203.

  Finally, in step ST16 of FIG. 4, it is determined whether or not the decoding of the frame is completed depending on whether or not the current macroblock 212 is at the end of the frame. If it is at the end, the process proceeds to step ST19. Return to the slice header decoding process of ST11.

(C) Macroblock decoding processing when a decoding failure is detected When the variable length decoding unit 1 detects a decoding failure, in step ST17, the decoding failure detection flag 103 is set to 1 to indicate that a decoding failure has occurred. And then search for a resynchronization point to restore decoding synchronization and resume normal decoding. The resynchronization point corresponds to, for example, a unique word indicating the head of the next slice or the head of the next frame.

  Since the encoded data 102 of the macroblock is lost after the decoding failure is detected until the resynchronization point is found, it is necessary to supplement the predetermined range of image data by restoration. In the first embodiment, since this image restoration is performed collectively for one frame based on the image restoration algorithm which is a feature of the present invention, in step ST18, here, from the decoding failure detection point to the resynchronization point. The macro block position information 104 to be repaired in the meantime is stored in the memory 2 as the error repair status map 106 as “repair target”.

FIG. 5 is a diagram showing a state of the error correction situation map 106 stored in the memory 2. The memory 2 holds error correction status maps 106 corresponding to the number of reference images that can be stored in the frame memory 8. As shown in FIG. 5, in the error repair status map 106, normal macroblocks that have undergone the process (B) are marked as “areas that do not need to be repaired”, and macroblocks that have detected a decoding failure are detected. Is marked as “image restoration target area”. The next normal macroblock after the macroblock in which the decoding failure is detected becomes the head of the next slice at the resynchronization point.
After detecting the resynchronization point and storing the region to be repaired, the process returns to step ST11 and decoding from the resynchronization point is resumed.

(D) Image Restoration Process After executing the processes (B) and (C) for one frame, in step ST19, the image restoration processing unit 5 is based on the image restoration execution instruction information 105 from the variable length decoding unit 1. Then, referring to the error repair situation map 106, the area including the area to be repaired is determined as the image repair area, and the repair macroblock position for executing the repair process in the determined image repair area is determined and determined. Validity / invalidity of surrounding macroblocks of the repaired macroblock is determined, image restoration context information 109 is obtained for the macroblock position determined to be valid, and an estimated motion vector 214 used for macroblock restoration is obtained, and the obtained estimation is obtained. Based on the motion vector 214, the decoded image 114 is obtained from the immediately preceding frame in the frame memory 8, and the repaired image 1 is obtained. 10 is output.

The point of the present invention is the image restoration algorithm executed by the image restoration processing unit 5. Usually, as a process of repairing a macroblock lost due to an error, time concealment for generating a repaired image 110 by using a previously decoded reference image stored in the frame memory 8 as in Non-Patent Document 1 above. There are two types: processing and spatial concealment that generates the repaired image 110 using normally decoded information present in the spatial vicinity within the same frame. As described in the following document, for example, the degree of correlation between the reference image 110 in the frame memory 8 and the current frame is evaluated based on, for example, whether or not the current frame is a scene change location. In general, the method is designed so that the temporal concealment is used when the correlation is high, and the spatial concealment is used when the correlation is low. The image restoration algorithm in the present invention particularly relates to a temporal concealment technique.
Akio Yoneyama, Hiromasa Yanagihara, Yasuyuki Nakajima, “VIDEO ERROR CONCEALMENT BY THE COMBINATION OF SPATIO-TEMPORAL RECOVERY AND POST FILTER FOR H.264”, IEEE International Conference on Consumer Electronics 2005, PAPER NO.9.3-2, Jan. 10-12 , 2005.

  When the variable length decoding unit 1 determines that the current frame needs to be subjected to the image restoration process, the variable length decoding unit 1 instructs the image restoration processing unit 5 to execute the image restoration process by the image restoration execution instruction information 105. The image restoration processing unit 5 starts the restoration process in response to the instruction.

FIG. 6 is a flowchart showing the flow of the image restoration algorithm process executed by the image restoration processing unit 5.
In step ST <b> 21, the image restoration area determination unit 51 illustrated in FIG. 3 refers to the error restoration situation map 106 relating to the current frame, and determines a circumscribed square area including the area to be restored as the image restoration area 211.

  FIG. 7 is a diagram showing a circumscribed square area including an area to be repaired determined by the image repair area determining unit 51. The following processing is executed only in the image restoration area 211. Since there is no area lost due to an error in the macroblock outside the image restoration area 211, the following processing is skipped.

  In step ST <b> 22, the restoration target macroblock determination unit 52 determines a restoration target macroblock to be restored in the image restoration area 211 as the current macroblock 212.

Although the following method can be considered as a method of determining the current macroblock 212, any method may be used.
As one method, the restoration target macroblock determination unit 52 verifies the inside of the image restoration area 211 in the raster scan order, and sequentially repairs the macroblocks marked with “repair target”. Although a special determination regarding the processing order is not necessary, if the “restoration target” macroblocks are locally concentrated, there is a possibility that the usable image restoration context information 109 cannot be obtained sufficiently.

  As another method, the restoration target macroblock determination unit 52 sequentially verifies the image restoration area 211 from the macroblock located at the end of the area, and sequentially repairs the macroblocks marked with “repair target”. To do. While it is necessary to make a judgment on the processing order one by one, there are many normally decoded macroblocks at the edge of the area, and when the “restoration target” macroblocks are concentrated locally, usable image restoration is possible. The context information 109 for use may be easily obtained.

  In step ST23, the peripheral block validity / invalidity determination unit 53 refers to the error repair status map 106 related to the current frame and determines whether the peripheral macroblocks located above, below, left, and right of the current macroblock 212 are “repair targets”. By verifying, if the macroblock subject to valid / invalid judgment is “repair target” or goes out of the screen, the macroblock is judged as “invalid”, otherwise it is judged as “valid” Then, the validity / invalidity of the peripheral macroblock is determined and the valid / invalid state 213 is output. In the error repair status map 106, a macroblock indicating a “repaired” state, which will be described later, is determined to be “valid”.

  In step ST <b> 24, the motion vector estimation unit 54 refers to the valid / invalid state 213, acquires the image restoration context information 109 for the macroblock position determined to be “valid”, and estimates motion used for restoration of the current macroblock 212. Vector 214 is determined. If no macroblock is “valid” based on the valid / invalid state 214, the motion vector estimation unit 54 sets the estimated motion vector 214 to zero.

  Hereinafter, as an example, a method for estimating the estimated motion vector 214 when all the upper, lower, left, and right macroblocks are determined as “valid” in step ST24 will be described. Bold notation indicates a vector value. At this time, since the 8 × 8 blocks in the adjacent upper, lower, left, and right macroblocks are effective for the current macroblock, a total of eight pieces of context information can be used.

The motion vector context information is ctx_mv [8], the reference image index context information is ctx_refidx [8], and the prediction error amount context information is ctx_err [8]. First, for each ctx_mv [k], the motion vector value is scaled to the motion vector for the previous frame. Assuming that the motion is linear, the scaled motion vector scaled_ctx_mv [k] is obtained by the following equation (1) using ctx_refidx [k].

ここで、mbpos_in_pelは、画素単位でのカレントマクロブロックの位置である。 By using scaled_ctx_mv [k], position information refpos that defines which macroblock position the area pointed to by ctx_mv [k] in the immediately preceding frame corresponds is defined by the following expression (2).

Here, mbpos_in_pel is the position of the current macroblock in pixel units.

  FIG. 8 is a diagram for explaining processing for obtaining the estimated motion vector 214 by the motion vector estimation unit 54.

ここで、Kは利用可能なctx_mv[k]の数であり、上記式（３）は、利用可能な全てのscaled_ctx_mv[k]について、それぞれに対応する予測誤差量ctx_err[k]で重みを付加し、重み付平均をとったものである。つまり、予測誤差量の小さいctx_mv[k]のほうが動きベクトルの信頼性が高いとみなし、より強い重みが掛かるようになっている。また、ctx_mv[k]はそれぞれ参照画像インデックスctx_refidx[k]が異なるが、この方法により、参照画像を全て直前フレームに統一することができ、修復画像の絵柄の連続性も保証できる。 Next, referring to the error correction status map 106 of the immediately preceding frame held in the memory 2, if the macroblock at the position corresponding to refpos [k] in the immediately preceding frame is “no need for repair”, ctx_mv It is determined that [k] can be used for the subsequent processing. Otherwise, ctx_mv [k] is not used for further processing. Based on the above results, the final estimated motion vector (emv) 214 is obtained by the following equation (3).

Here, K is the number of ctx_mv [k] that can be used, and the above equation (3) adds a weight with the corresponding prediction error amount ctx_err [k] for all available scaled_ctx_mv [k] And a weighted average. That is, ctx_mv [k] having a smaller prediction error amount is considered to have higher motion vector reliability and is given a stronger weight. Further, ctx_mv [k] has a different reference image index ctx_refidx [k], but by this method, all the reference images can be unified in the immediately preceding frame, and the continuity of the pattern of the repaired image can be guaranteed.

  If the macroblock at the position corresponding to refpos [k] in the immediately preceding frame is “no need for repair” with reference to the error repair status map 106 of the immediately preceding frame held in the memory 2, ctx_mv [k] is determined to be usable for the subsequent processing. If it is determined that all ctx_mv [k] cannot be used, the estimated motion vector (emv) 214 is set to zero.

  Further, the value of the estimated motion vector 214 and the value of the reference image index (= 0) determined here are held as the image restoration context information 109 in the current macroblock, and the current macroblock position of the error restoration situation map 106 is stored. Change the status from "Repair target" to "Repaired". These are used for subsequent image restoration processing.

  In step ST <b> 25, the repair image generation unit 55 acquires the repair image 110 from the immediately previous frame in the frame memory 8 based on the estimated motion vector 214 and outputs it from the image repair processing unit 5. The repair image generation unit 55 may be configured to use the prediction image generation unit 33 of the macroblock decoding processing unit 3 in order to perform a process equivalent to the prediction image generation in the inter-frame motion compensation prediction. .

(D) Final decoded image generation processing When the decoding failure detection flag 103 indicates no failure (= 0), the switch 6 outputs the first decoded image 108 as the second decoded image 111 (already As described above, the switch 6 does not always receive the first decoded image 108 and the restored image 110 as inputs).
In step ST20 of FIG. 4, the second decoded image 111 is input to the deblocking filter 7, and the deblocking filter 7 performs a deblocking filter process based on a process defined in the AVC standard to obtain a final decoded image 112. The final decoded image 112 is written to the frame memory 8 based on the position of the macroblock in the screen, for use in generating predicted images for subsequent frames.

When the decoding failure detection flag 103 indicates that there is a failure (= 1), the switch 6 outputs the repaired image 110 as the second decoded image 111 (as described above, the switch 6 is always the first The decoded image 108 and the repaired image 110 are not input.
In step ST20 of FIG. 4, the second decoded image 111 is input to the deblocking filter 7, and the deblocking filter 7 performs a deblocking filter process based on a process defined in the AVC standard to obtain a final decoded image 112. However, at this time, since the second decoded image 111 is likely to be different from the original decoded image data, the deblocking filter 7 is executed in the mode with the strongest filter strength. Thereby, the discontinuity of the boundary can be reduced at the block boundary where the repaired image 110 and the normal image are adjacent to each other. The final decoded image 112 is written to the frame memory 8 based on the position of the macroblock in the screen, for use in generating predicted images for subsequent frames.

  As described above, according to the first embodiment, even when a fatal error that causes a decoding failure is included during decoding of a macroblock, a motion vector is stably estimated based on surrounding context information. By performing image restoration, it is not necessary to frequently access the frame memory 8 to obtain a plurality of predicted image candidates as in Non-Patent Document 1, and error concealment is efficient and error-resistant with less processing. By performing processing, the required memory bandwidth required per macroblock can be reduced, and the reproduction quality can be efficiently maintained under severe power consumption conditions such as broadcasting / communication for mobile units.

  In the first embodiment, a video decoding device that decodes a bitstream that complies with the AVC standard has been described, but the present invention relates to various international standard schemes that perform motion compensation prediction using macroblock units, MPEG-1, MPEG-2, MPEG-4, H.264, and the like that can extract context information as described in the first embodiment. 261, H.H. Needless to say, the present invention can also be applied to a moving picture decoding apparatus that supports a standard such as H.263. Note that the reference picture index 204 is employed only in the AVC standard, but in other standard systems, processing may be performed by always considering this as zero.

It is a block diagram which shows the structure of the moving image decoding apparatus by Embodiment 1 of this invention. It is a block diagram which shows the internal structure of the macroblock decoding process part of the moving image decoding apparatus by Embodiment 1 of this invention. It is a block diagram which shows the internal structure of the image restoration process part of the moving image decoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the flow of a process of the moving image decoding apparatus by Embodiment 1 of this invention. It is a figure which shows the mode of the error correction condition map memorize | stored in the memory of the moving image decoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the flow of a process of the image restoration algorithm performed by the image restoration process part of the moving image decoding apparatus by Embodiment 1 of this invention. It is a figure which shows the circumscribed square area | region containing the area used as the repair object determined by the image restoration area determination part of the moving image decoding apparatus by Embodiment 1 of this invention. It is a figure explaining the process which calculates | requires the estimated motion vector by the motion vector estimation part of the moving image decoding apparatus by Embodiment 1 of this invention. It is a figure for demonstrating the candidate of a restoration image.

Explanation of symbols

  1 variable length decoding unit, 2 memory, 3 macroblock decoding processing unit, 4 image restoration context information generation unit, 5 image restoration processing unit, 6 switch, 7 deblocking filter, 8 frame memory, 31 dequantization unit, 32 inverse integer transform unit, 33 predicted image generation unit, 34 adder, 51 image restoration area determination unit, 52 restoration target macroblock decision unit, 53 peripheral block validity / invalidity determination unit, 54 motion vector estimation unit, 55 restoration image generation unit , 101 bit stream, 102 encoded data of macroblock, 103 decoding failure detection flag, 104 macroblock position information to be repaired, 105 image repair execution instruction information, 106 error repair status map, 107 prediction residual image, 108 first Decoded image, 109 image restoration context information, 110 restoration Image, 111 second decoded image, 112 final decoded image, 113 reference image, 114 decoded image of previous frame, 201 encoding mode information, 202 intra prediction mode information, 203 motion vector, 204 reference picture index, 205 integer conversion Coefficient data, 206 quantization step parameter, 207 integer transform coefficient data after inverse quantization, 208 predicted image, 211 image restoration area, 212 current macroblock, 213 valid / invalid state, 214 estimated motion vector.

Claims (4)

Extracting encoded data of a predetermined area from a bit stream that is encoded by dividing each frame of a moving image into predetermined areas, and detecting a decoding failure in the process of extracting the encoded data, A variable length decoding unit to be identified;
A predetermined region decoding processing unit that decodes the prediction residual image from the encoded data of the predetermined region extracted by the variable length decoding unit and generates a decoded image of the predetermined region;
An image restoration context information generation unit that inputs encoded data of a predetermined region from the variable length decoding unit and a prediction residual image from the predetermined region decoding processing unit, and generates image restoration context information;
Image restoration generated by the image restoration context information generation unit from the surrounding predetermined region determined to be valid by determining the validity of the predetermined region around the predetermined region to be repaired specified by the variable length decoding unit Image restoration processing that obtains context information, obtains an estimated motion vector used for restoration of a predetermined area to be restored, obtains a decoded image from a frame that has already been decoded based on the obtained estimated motion vector, and generates a restored image And
A selection unit that selects and outputs a decoded image generated by the predetermined region decoding processing unit or a repaired image generated by the image restoration processing unit based on whether or not a decoding failure of the predetermined region is detected by the variable length decoding unit; A video decoding device comprising: The image restoration context information generation unit generates image restoration context information related to a motion vector, a reference image index, and a prediction error amount from encoded data of a predetermined region,
The image restoration processing unit scales the motion vector based on the reference image index and uses a weighting factor determined based on the prediction error amount to perform weighted averaging of the scaled motion vector to be a restoration target. 2. The moving picture decoding apparatus according to claim 1, wherein an estimated motion vector used for repairing the predetermined area is obtained. The image restoration processing unit according to claim the effectiveness of a given region around a predetermined region of the repaired, and judging effect when not in and out of the screen when not in the repaired 1 The moving picture decoding apparatus described. Extracting encoded data of a predetermined area from a bit stream that is encoded by dividing each frame of a moving image into predetermined areas, and detecting a decoding failure in the process of extracting the encoded data, A first step to identify;
A second step of decoding a prediction residual image from the encoded data of the predetermined region extracted in the first step and generating a decoded image of the predetermined region;
A third step of inputting the encoded data of the predetermined area extracted in the first step and the prediction residual image decoded in the second step, and generating context information for image restoration;
The validity of the predetermined area around the predetermined area to be repaired specified in the first step is determined, and the image restoration context information generated in the third step from the predetermined peripheral area determined to be effective Obtaining an estimated motion vector to be used for repairing a predetermined area to be repaired, and acquiring a decoded image from a frame that has already been decoded based on the determined estimated motion vector to generate a repaired image; ,
Fifth step of selecting and outputting the decoded image generated in the second step or the repaired image generated in the fourth step based on the presence or absence of detection of the decoding failure of the predetermined area in the first step A video decoding method comprising:

Images