KR100984650B1 - H. 24 Encoding method that enables highly efficient partial decoding of 264 and other transform coded information. - Google Patents

Abstract

A method and apparatus for processing multimedia data that enables efficient partial decoding of transform coded data is disclosed. The decoder device receives the transform coefficients, where the transform coefficients are associated with the multimedia data. The decoder device determines the set of multimedia samples to be reconstructed. In one aspect, the set of samples to be reconstructed is a subset of the matrix of transformed multimedia samples. The decoder device determines the set of transform coefficients used to reconstruct the multimedia samples. In one aspect, the transform coefficients are used to scale the partial base images associated with the encoding method used to generate the transform coefficients, resulting in reconstructed multimedia samples.

Multimedia data, multimedia samples, transform coefficients, dequantization, decoder device

Description

H.264 AND OTHER TRANSFORM CODED INFORMATION} VIDEO ENCODING METHOD ENABLING HIGHLY EFFICIENT PARTIAL DECODING OF H.264 AND OTHER TRANSFORM CODED INFORMATION

Cross Reference to Related Application

Claim priority under U.S.C. §119

This patent application claims priority to Provisional Application No. 60 / 721,377, filed September 27, 2005, which is assigned to this assignee and expressly incorporated herein by reference.

background

Field of invention

The present invention relates to multimedia signal processing, and more particularly to video encoding and decoding.

Description of the related technology

Multimedia signal processing systems, such as video encoders, may encode multimedia data using encoding methods based on international standards such as the MPEG-x and H.26x standards. Such encoding methods are generally directed to compressing multimedia data for transmission and / or storage. Compression is usually the process of removing redundancy from data.

The video signal may be described by a sequence of pictures, including frames (full picture), or fields (eg, interlaced video signal includes a field that alternates odd or even lines of the picture). As used herein, the term "frame" refers to an image, frame or field. Video encoding methods compress video signals by using a lossless or lossy compression algorithm to compress each frame. Intra-frame coding (herein referred to as intra-coding) means encoding a frame using that frame. Inter-frame coding (herein referred to as inter-coding) means encoding a frame based on other "reference" frames. For example, video signals often exhibit spatial redundancy with at least a portion where some of the video frame samples close to each other within the same frame match or at least approximately match each other.

A multimedia processor, such as a video encoder, may encode a frame by dividing it into blocks, or “macroblocks” of, for example, 16 × 16 pixels. The encoder may also split each macroblock into subblocks. Each subblock may also include additional subblocks. For example, subblocks of one macroblock may include 16x8 subblocks and 8x16 subblocks. Subblocks of 8x16 subblocks may include 8x8 subblocks and the like. As used herein, the term "block" means either a macroblock or a subblock.

One compression technique, based on evolving industry standards, is commonly referred to as "H.264" video compression. H.264 technology defines the syntax of an encoded video bitstream along with a method of decoding this bitstream. In one aspect of the H.264 encoding process, one input video frame is presented for encoding. The frame is processed into units of macroblocks corresponding to the original image. Each macroblock may be encoded in intra or inter mode. The predicted macroblock is formed based on some of the already reconstructed neighboring blocks in the same frame, known as already reconstructed frames or causal neighbors. In intra mode, a macroblock is formed from causal samples in the current frame that have been previously encoded, decoded, and reconstructed. Multimedia samples of one or more causal neighboring macroblocks are subtracted from the current macroblock to be encoded to produce a residual or difference macroblock D. This residual block D is transformed using a block transform and quantized to produce a set X of quantized transform coefficients. These transform coefficients are rearranged and entropy encoded. The entropy encoded coefficients, along with other information for decoding the macroblock, become part of the compressed bitstream transmitted to the receiving device.

Unfortunately, during the transmission process, an error may be introduced in one or more macroblocks. For example, one or more poor transmission effects, such as signal fading, may cause loss of data in one or more macroblocks. As a result, error concealment becomes important when delivering multimedia content over an error prone network such as wireless channels. The error concealment method uses the spatial and temporal correlation present in the video signal. When faced with an error, recovery may occur during entropy decoding. For example, when faced with a packet error, all or part of the data belonging to one or more macroblocks or video slices (usually groups of neighboring macroblocks) may be lost. When one slice of video data is lost, resynchronization of decoding may occur in the next slice, and missing blocks of the lost slice may be concealed using spatial concealment.

Since the decoded data available to the decoder device includes causal neighbors that have already been decoded and reconstructed, spatial concealment usually uses causal neighbors to conceal lost blocks. One reason to use causal neighbors to conceal lost blocks is that concealment of the lost portion of the current slice after non-sequential reconstruction of the next slice would be very inefficient, especially when using a highly pipelined video hardware decoder core. Can be. Non-causal neighbors can provide valuable information for improved spatial concealment. What is needed is an efficient way to provide non-sequential reconstruction of non-causal neighboring multimedia samples.

summary

Each of the systems, methods, and devices of the present invention has several aspects, not just one of which is solely responsible for the desired attributes. More important features will now be briefly described without departing from the scope of the present invention as expressed by the following claims. After considering this description, in particular after reading the section entitled “Detailed Description of Specific Aspects”, a method is provided in which the sample features of the present invention provide advantages for multimedia encoding and decoding including improved error concealment, and improved efficiency. I will understand.

A method of processing multimedia data is provided. The method includes receiving transform coefficients, the transform coefficients being associated with the multimedia data. The method includes determining a set of reconstructed multimedia samples, determining a set of received transform coefficients based on the reconstructed multimedia samples, and generating reconstructed samples corresponding to the determined set of multimedia samples. Processing the determined set of transform coefficients.

A multimedia data processor is provided. The processor is configured to receive transform coefficients, the transform coefficients being associated with the multimedia data. The processor may also determine a set of multimedia samples to be reconstructed, determine a set of received transform coefficients based on the multimedia samples to be reconstructed, and determine to generate reconstructed samples corresponding to the determined set of multimedia samples. And to process the set of transform coefficients.

An apparatus for processing multimedia data is provided. The apparatus includes a receiver for receiving transform coefficients, the transform coefficients being associated with the multimedia data. The apparatus includes a first determiner for determining a set of multimedia samples to be reconstructed, a second determiner for determining a set of received transform coefficients based on the multimedia samples to be reconstructed, and a reconstructed corresponding to the determined set of multimedia sets. And a generator for processing the determined set of transform coefficients to generate samples.

In execution, a machine readable medium is provided that includes instructions that cause a machine to process multimedia data. The instructions cause the machine to receive transform coefficients, which are associated with the multimedia data. The instructions also cause the machine to determine a set of multimedia samples to be reconstructed, to determine a set of received transform coefficients based on the multimedia samples to be reconstructed, and to reconstruct the sample corresponding to the determined set of multimedia samples. To process the determined set of transform coefficients to generate them.

Brief description of the drawings

1 is a block diagram illustrating a multimedia communication system according to one aspect.

FIG. 2A is a block diagram illustrating an aspect of a decoder device that may be used in a system such as that shown in FIG. 1.

FIG. 2B is a block diagram illustrating an example of a computer processor system of a decoder device that may be used in a system such as that shown in FIG. 1.

3 is a flowchart illustrating an example of a method of decoding a portion of a video stream in a system such as that shown in FIG. 1.

4 is a flowchart illustrating another example of a method of decoding a portion of a video stream in a system such as that shown in FIG. 1 in more detail.

5 is a detail view of a 4x4 block and causal neighboring pixels around it.

FIG. 6 is a directional mode diagram illustrating 9 (0 to 8) directional modes used to describe directional characteristics of one block in H.264.

7 is a diagram illustrating an example of an intra-coded 4x4 pixel block just below and to the right of one or more slice boundaries.

8 is a list of neighboring pixels and pixels within an intra-coded 4x4 pixel block.

9 is a diagram illustrating an example of an intra-coded 16 × 16 Luma (Luma) macroblock just below and to the right of one slice boundary.

FIG. 10 is a diagram illustrating an example of an intra-coded 8x8 Chroma block just below and to the right of one slice boundary.

11 is a diagram illustrating a portion of multimedia samples located directly below one slice boundary.

12 is a block diagram illustrating another example of a decoder device that may be used in a system such as that shown in FIG. 1.

FIG. 13 is a block diagram illustrating another example of a decoder device 150 that may be used in a system such as that shown in FIG. 1.

Detailed Description of Specific Aspects

The following detailed description relates to some specific sample aspects of the present invention. However, the present invention can be implemented in many different ways as defined and covered by the claims. This description is made with reference to the drawings, in which like parts are denoted by like numerals.

Video signals are characterized by a series of pictures, frames, or fields. As used herein, the term "frame" is a broad term that may include either frames of a progressive video signal or frames or fields of an interlaced video signal.

Aspects include a system and method for enhancing processing at an encoder and decoder in a multimedia transmission system. The multimedia data may include one or more of video, audio, still images, or any other suitable type of audio-visual data. Aspects include an apparatus and method for encoding video data in an efficient manner to reconstruct non-causal multimedia samples and to provide improved error concealment by performing spatial concealment of lost or incorrectly encoded multimedia data using the reconstructed samples. do. For example, according to one aspect, before estimating multimedia concealment data for lost or erroneous data, it has been found that the generation of reconstructed causal and / or causal neighboring samples can improve the quality of spatial concealment. . In some examples, the reconstructed multimedia samples and the directional indicator from which the reconstructed samples were initially encoded are used for estimation of the multimedia concealment data. In another aspect, it has been found that reconstruction of a subset of the matrix of multimedia samples used for spatial error concealment can further improve processing efficiency. In some examples, reconstruction of multimedia samples and estimation of multimedia concealment data are performed in the pre-processor. The multimedia concealment data may then be conveyed with the first encoded non-causal multimedia data decoded in an efficient video core processor, further improving processing efficiency.

Multimedia communication systems

1 is a functional block diagram illustrating a multimedia communication system 100 according to an aspect. The multimedia communication system 100 includes an encoder device 110 in communication with a decoder device 150 via a network 140. In one example, the encoder device receives a multimedia signal from an external source 102 and encodes the signal for transmission over the network 140.

In this example, encoder device 110 includes a processor 112 coupled to memory 114 and transceiver 116. The processor 112 encodes data from the multimedia data source and provides it to the transceiver 116 for communication over the network 140.

In this example, decoder device 150 includes a processor 152 coupled to memory 154 and transceiver 156. Processor 152 may include one or more of a general purpose processor and / or a digital signal processor and / or an application specific hardware processor. Memory 154 may include one or more of solid or disk-based storage or any readable and writable random access memory device. The transceiver 156 is configured to receive multimedia data via the network 140 and make it available to the processor 152 for decoding. In one example, the transceiver 156 includes a wireless transceiver. Network 140 may be a wired or wireless communication system including one or more of Ethernet, telephone (eg, POTS), cable, powerline, and fiber optic systems, and / or code division multiple access (CDMA or CDMA2000) communication systems, Frequency Division Multiple Access (FDMA) systems, Time Division Multiple Access (TDMA) systems such as GSM / General Packet Radio Service (GPRS) / Enhanced Data GSM Environment (EDGE), Terrestrial Trunked Radio (TETRA) mobile phone systems, Wideband code division multiplexing One or more of an access (WCDMA) system, a high-speed data (1xEV-DO or 1xEV-DO Gold Multicast) system, an IEEE 802.11 system, a MediaFLO system, a DMB system, an Orthogonal Frequency Division Multiple Access (OFDM) system, or a DVB-H system. It may also include one or more of the wireless systems included.

FIG. 2A is a functional block diagram illustrating an aspect of a decoder device 150 that may be used in a system such as the system 100 shown in FIG. 1. In this aspect, the decoder 150 includes the receiver element 202, the multimedia sample determiner element 204, the transform coefficient determiner element 206, the reconstructed sample generator element 208, and the multimedia concealment estimator element 210. Include.

Receiver 202 receives encoded video data (eg, data encoded by encoder 110 of FIG. 1). Receiver 202 may receive encoded data via a wired or wireless network, such as network 140 of FIG. 1. In one aspect, the received data includes transform coefficients representing source multimedia data. Transform coefficients are transformed in the region where the correlation of neighboring samples is significantly reduced. For example, images typically exhibit high spatial correlation in the spatial domain. On the other hand, the transform coefficients are usually orthogonal to each other, which shows zero correlation. Some examples of transforms that can be used for multimedia data include, but are not limited to, the Discrete Cosine Transform (DCT), the Discrete Fourier Transform (DFT), Adamar, as used in H.264. (Hadamard, or Walsh-Adamar), Discrete Wavelet, DST (Discrete Sine Transform), Harar, Slant, and Karhunen-Loeve (KL) And integer conversion. Transforms are used to transform a matrix or array of multimedia samples. Two-dimensional matrices are usually used, but one-dimensional arrays may be used. The received data also includes information indicating how the encoded blocks were encoded. Such information may include inter-coding reference information such as motion vectors and frame sequence numbers, intra-coding reference information including block size and spatial prediction directional indicators, and the like. Some received data includes quantization parameters indicating how each transform coefficient is rounded, non-zero indicator indicating how many transform coefficients in the transformed matrix are non-zero, and the like.

The multimedia sample determiner 204 determines which multimedia samples are to be reconstructed. In one aspect, the multimedia sample determiner 204 determines neighboring multimedia samples or pixels that are close to multimedia data that may be lost and hidden and / or that are in a border region of the multimedia data. In one example, the multimedia sample determiner identifies pixels adjacent to one slice's border or another group of blocks where some of the data was lost due to an error or channel loss. In some examples, multimedia sample determiner 204 identifies the fewest number of pixels associated with reconstructing spatial predicted neighboring blocks from the determined pixels. For example, the compressed multimedia data may include blocks of transform coefficients resulting from the transformation of individual blocks (eg, 8x8 pixel blocks and / or 4x4 pixel blocks) or matrices. . The multimedia sample determiner 204 is a particular subset of the multimedia samples of the transformed block that is reconstructed to be used to conceal lost data or to reconstruct other encoded multimedia samples in other blocks predicted from the samples. Can be identified. The determined multimedia samples may include non-causal samples and / or causal samples.

Transform coefficient determiner 206 determines a set of transform coefficients used to reconstruct some or all of the multimedia samples determined to be reconstructed by multimedia sample determiner 204. The determination of transform coefficients for use depends on the encoding method used to generate the transform coefficients. Transform coefficient determination also depends on which multimedia samples are being reconstructed and whether there are transform coefficients with zero values (thus negating the potential need to use them). Details are described below that the transform coefficients may be sufficient to reconstruct the multimedia samples.

The reconstructed sample generator 208 reconstructs the multimedia samples based on the samples determined by the multimedia sample determiner 204. The set of reconstructed samples may be an entire set, such as the entire N × N (where N is an integer) matrix of samples. The set of samples may be a subset of samples from an N × N matrix, such as a row, column, part of a row or column, a diagonal line, or the like. Reconstructed sample generator 208 uses the transform coefficients determined by transform coefficient determiner 206 when reconstructing the samples. Reconstructed sample generator 208 also uses information based on the encoding method used to encode the transform coefficients in reconstructing the multimedia samples. Details of the operations performed by the reconstructed sample generator 208 are described below.

The multimedia concealment estimator 210 uses reconstructed samples calculated by the reconstructed sample generator 208 to form hidden multimedia samples to replace or conceal areas of multimedia data that are lost or changed in error during transmission / reception. do. The multimedia concealment estimator 210 uses the reconstructed sample values to form the hidden multimedia data in one aspect. In another aspect, the multimedia concealment estimator 210 uses the reconstructed sample values and the received spatial prediction directional mode indicator in estimating the multimedia concealment data. Further details on spatial error concealment can be found in Application No. 11 / 182,621 (US Patent Publication No. 2006/0013320), “METHODS AND APPARATUS FOR SPATIAL ERROR CONCEALMENT”, assigned to the assignee herein.

In some aspects, one or more of the elements of decoder 150 of FIG. 2A may be rearranged and / or combined. The elements may be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. Details of the operations performed by the elements of decoder 150 will be described below with reference to the methods shown in FIGS. 3 and 4.

FIG. 2B is a block diagram illustrating an example of a computer processor system of a decoder device that may be used in a system such as that shown in FIG. 1. Decoder device 150 of this example includes pre-processor element 220, RAM element 222, digital signal processor (DSP) element 224, and video core element 226.

Pre-processor 220 is used, in one aspect, to perform one or more of the operations performed by the various elements of FIG. 2A. The pre-processor parses the video bitstream and writes the data to RAM 222. Also, in one aspect, pre-processor 220 implements the operations of multimedia sample determiner 204, transform coefficient determiner 206, reconstructed sample generator 208, and multimedia concealment estimator 210. By performing their more efficient, less computationally intensive operations in the pre-processor 220, more computationally intensive video decoding can be done in a causal order in the highly efficient video core 226.

The DSP 224 retrieves the parsed video data stored in the RAM 222 and reorganizes it for processing by the video core 226. Video core 226 performs inverse quantization (also known as rescaling or scaling), inverse transform and deblocking functions, as well as other video decompression functions. Video cores are usually implemented in a highly optimized and pipelined manner. Because of this, video data can be decoded in the fastest way when it is decoded in a causal order. By performing out-of-order reconstruction of the multimedia samples and subsequent spatial concealment in the pre-processor, a causal order is maintained for decoding in the video core, allowing for improved overall decoding performance.

3 is a flowchart illustrating an example of a method of decoding a portion of a video stream in a system such as that shown in FIG. 1. The method 300 may be performed by a decoding device, such as the examples shown in FIGS. 2A and 2B. The method 300 enables reconstruction of the selected multimedia samples. The method 300 may be used to reconstruct the multimedia samples in a causal order in which other encoded multimedia data is predicted from the causal data, and may require reconstruction of the causal data before reconstructing itself. The method 300 may be used to reconstruct the multimedia samples in an in causal order. In one aspect, the non-causal data is reconstructed in a manner that allows subsequent reconstruction of both multimedia data (both causal and non-causal) in a more efficient and timely manner.

The method 300 begins at block 305 where a decoder device receives transform coefficients associated with a multimedia data bitstream. The decoder device may receive the transform coefficients via a wired and / or wireless network, such as the network 140 shown in FIG. 1. The transform coefficients may represent multimedia samples that include color and / or brightness parameters, such as chroma and luminance, respectively. The transforms used to generate the transform coefficients are, but are not limited to, DCT (discrete cosine transform), DFT (discrete Fourier transform), Adama (or Walsh-Adamar) transform, discrete as used in H.264. Wavelet transforms, DST (discrete sine transforms), Har transforms, gradient transforms, KL (Karunen Rube) transforms, and integer transforms may be included. Multimedia samples may be transformed in groups such as one-dimensional arrays and / or two-dimensional matrices when the transform coefficients are generated during encoding. Transform coefficients may be intra-coded and may or may not include spatial prediction. If spatial prediction was used in the generation of the transform coefficients, the transform coefficients may represent a residual value that is an error of the predictor provided by the reference value. Transform coefficients may be quantized. Transform coefficients may be entropy encoded. The receiver element 202 of FIG. 2A may perform an act at block 305.

After receiving the transform coefficients, the method 300 proceeds to block 310 where the decoder device determines the set of multimedia samples to be reconstructed. The multimedia samples to be reconstructed may include luma and chroma samples. In some examples, the set of multimedia samples to be reconstructed is determined in response to a loss of synchronization while decoding the multimedia bitstream received at block 305. The loss of synchronization may be caused by the loss or incorrect reception of some or all of the encoded data corresponding to the multimedia samples included in the first slice of macroblocks. The determined multimedia samples to be reconstructed may be included in the second slice of macroblocks. The second slice of macroblocks abuts at least a portion of the missing portion of the first slice of macroblocks. The determined multimedia samples may be causal or non-causal with respect to the missing portion of the multimedia samples, as described above.

In an aspect, the multimedia samples determined to be reconstructed at block 310 may enable reconstruction of other multimedia samples that abut the lost portion of the hidden multimedia data. For example, intra-coded macroblocks below the other slice of macroblocks may be spatially predicted with respect to the determined set of multimedia samples determined to be reconstructed at block 310. Thus, by reconstructing the determined set of determined multimedia samples that strongly correlate with the intra-coded blocks, the intra-coded blocks themselves can be reconstructed through the concealment process. In another aspect, the multimedia samples determined to be reconstructed at block 310 may include samples located at or near the slide border. The samples to be reconstructed may include the entire matrix of associated multimedia samples transformed as a group during encoding. The samples to be reconstructed may also include part of a matrix of associated multimedia samples, such as rows, columns, diagonals, or some and / or combinations thereof. The multimedia sample determiner 204 of FIG. 2A may perform the action at block 310. Details of the subset of multimedia samples that may be reconstructed are described below.

The method 300 proceeds to block 315, where the decoder device determines a set of transform coefficients associated with the multimedia samples determined to be reconstructed at block 310. The determination of transform coefficients for use for reconstruction depends on the encoding method used to generate the transform coefficients. The transform coefficient determination also depends on which multimedia sample is being reconstructed. For example, it may be determined that the entire set of multimedia samples determined at block 310 may be reconstructed, or alternatively one subset may be determined to be reconstructed. The transform coefficient determination at block 315 depends on whether or not there are transform coefficients with zero values (thus negating the potential need to use them). Details of which transform coefficients may be sufficient to reconstruct the multimedia samples are described below. The transform coefficient determiner of FIG. 2A may perform the action at block 315.

After determining the set of multimedia samples to be reconstructed in block 310, and after determining the set of transform coefficients associated with the multimedia samples determined in block 315, the method 300 proceeds to block 320. At block 320, the decoder device processes the determined set of transform coefficients to generate reconstructed multimedia samples. The processing performed depends on the encoding methods used to generate the transform coefficients. The processing may further include other actions, including but not limited to entropy decoding, inverse quantization (also called rescaling or scaling), and the like. Details of examples of the processing performed at block 320 are described below with reference to FIG. 4.

In some example systems, some or all of the actions of method 300 are performed in a pre-processor, such as pre-processor 220 shown in FIG. 2B. It should be appreciated that some of the blocks of method 300 may be combinations, omissions, rearrangements, or any combination thereof.

4 is a flowchart detailing another example of a method of decoding a portion of a video stream in a system such as that shown in FIG. Exemplary method 400 includes all of the operations performed at blocks 305-320 included in method 300. Blocks 305, 310 and 315 remain unchanged from the examples shown in FIG. 3 and described above. Block 320 of method 300 of processing transform coefficients to generate reconstructed samples is described in more detail in method 400, which includes four blocks 405, 410, 420, and 425. The method 400 also includes additional blocks, such as block 430 in which hidden multimedia samples are estimated and block 435 in which transform coefficients based on the estimated hidden multimedia samples are generated.

The decoder device performs the operations at blocks 305, 310 and 315 in a similar manner as described above. The detailed example of block 320 shows that the transform coefficients are associated with a base image to efficiently reconstruct the multimedia samples. At block 405, the decoder device divides the transform coefficients into groups, and the groups of transform coefficients are associated with multimedia samples that are determined to be reconstructed at block 305. In one aspect, the groups of transform coefficients include transform coefficients that transform (or weight) the common base image during the inverse transform process in reconstruction. Details of how the transform coefficients are divided into groups are described below with respect to an example using H.264.

At block 410, the decoder device calculates a weight value associated with each divided group based on the encoding method that generated the coefficients. In one aspect, the weighting is the sum of the scaled transform coefficients of each group. Scaling duplicates the inverse transform characteristics of the encoding method. Examples of scaling and calculating weight values are described below with respect to H.264 examples.

At block 420, a base image is determined for each of the groups based on the encoding conversion method. The base image is typically two dimensional orthogonal matrices, but one dimensional arrays may be used. Some of the two-dimensional base images are used, where some depend on which multimedia samples are being reconstructed (as determined in block 310). The values calculated for each group in block 410 are used to transform (or weight) the associated base images in block 425. By combining all of the weighted base images, the multimedia samples are reconstructed at block 425. Details for blocks 420 and 425 are described below with respect to the H.264 example.

After generating the reconstructed multimedia samples, the method 400 proceeds to block 430 where the decoder device estimates hidden multimedia samples based on the reconstructed samples in some examples. In one aspect, the reconstructed sample values of the multimedia samples are used to form the hidden multimedia data. In another aspect, the reconstructed sample values and the received spatial prediction directional mode indicator are used to form the multimedia concealment data. Further details on spatial error concealment can be found in Application No. 11 / 182,621 (US Patent Publication No. 2006/0013320), “METHODS AND APPARATUS FOR SPATIAL ERROR CONCEALMENT”, assigned to the assignee herein.

In some examples, the estimated hidden multimedia samples are inserted into a frame buffer containing reconstructed data of the same frame that is used directly and later displayed. In another example, the estimated hidden multimedia samples are transformed in a manner that repeats the encoding process to generate transform coefficients representing the estimated hidden multimedia samples at block 435. These transform coefficients are then inserted into the undecoded (still encoded) bitstream as if they were their normal encoded samples. The entire bitstream may then be forwarded to video decoder code such as video core 226 in FIG. 2B to be decoded. In these examples, all or part of method 400 may be performed in a pre-processor such as pre-processor 220 of FIG. 2B. This method of performing reconstruction and concealment estimation is particularly useful for reconstructing non-causal portions that are later used to conceal other portions of multimedia data lost due to channel errors. Now, details of the methods used to improve the efficiency of reconstruction of multimedia samples will be described with respect to an H.264 encoded multimedia bitstream.

H.264 In the bitstream  High efficiency part Intra  decoding

H.264 uses spatial prediction that utilizes spatial correlation between neighboring blocks of pixels. The spatial prediction mode uses causal neighbors of 4x4, 8x8, or 16x16 pixel blocks on the left and top for spatial prediction. H.264 provides two modes for spatial prediction on luminance values, one of 4x4 pixel blocks (herein referred to as intra-4x4 coding) and the other of 16x16 pixel macroblocks. (Herein referred to as intra-16 × 16 coding). Note that other causal and non-causal neighbor samples may be used for spatial prediction.

FIG. 5 shows a detailed view of the 4x4 pixel block 502 and the causal neighboring pixels 504 on the left and top that surround the 4x4 pixel block. For example, during the H.264 encoding process, causal neighboring pixels 504 are used to generate various predictors, values, and / or parameters that describe the blocks 502 pixels. Block 502 includes pixels p0 to p15, and causal neighboring pixels 504 use reference indicators n3, n7, n11, n12, n13, n14, and n15. Where the number corresponds to similar locations of the block 502 pixels.

The spatial prediction modes provided in H.264 use various directional modes to spatially predict block 502 from various causal neighboring pixels 504. FIG. 6 shows a directional mode diagram 600 showing nine directional modes 0-8 used to describe the directional characteristic of an intra-coded block in H.264. Nine directional modes (or indicators) are used to describe the directional characteristic of the spatial prediction of block 502. For example, mode 0 describes the vertical directional characteristic, mode 1 describes the horizontal directional characteristic, and mode 2 describes the DC characteristic in which the average value of available causal neighboring pixels is used as a reference for prediction. do. In the DC mode, causal neighboring pixels (just above and to the left of a 4x4, 8x8, or 16x16 pixel block) that exist in the same slice are used in the calculation of the mean. For example, if the block to be encoded touches over one slice, the pixels on the left are averaged. If the block to be encoded touches the left and top of the other slice, then a value of 128 is used as the DC average (half of the 8-bit range of values provided in H.264). The modes shown in the directional mode diagram 600 are used in the H.264 encoding process to generate prediction values for block 502.

In Intra-4 × 4 coding of H.264, the luminance values can be encoded with respect to the left and top pixels of the 4 × 4 block, using any of nine directional modes. In Intra-16 × 16 coding, the luminance values are divided into four modes: i) vertical (mode 0), ii) horizontal (mode 1), iii) DC (mode 2), and iv) plane (mode 3). Can be encoded with respect to the left and top pixels of the entire 16x16 pixel block. In planar prediction mode, luminance values vary spatially and smoothly across macroblocks and references are formed based on planar equations. In the case of chromaticity, there is one prediction mode, 8x8. In intra-8x8 chroma coding, the 8x8 block has the same modes as used in intra-16x16 coding: i) vertical (mode 0), ii) horizontal (mode 1), iii) DC ( Mode 2), and iv) plane (mode 3). Details on reconstructing the predicted blocks encoded in H.264 will now be described.

The reconstructed signal in the (intra or inter) coded 4 × 4 (luminance or chromaticity) block of prediction is

는 각각 재구성된 신호 (원래의 압축해제된 신호 s 에 근사), 예측 신호, 및 압축된 잔여 신호 (원래의 압축해제된 잔여 신호에 근사:

= s - p (여기서, s 는 원래의 신호이다)) 를 나타내고, 이들 모두는, 이 예에서 정수 값의 4×4 행렬들이다. 잔여 값들

은, 변환 계수들의 역변환에 의해 재구성될 수 있다. 예측 값들 p 은, 그들을 인코딩하기 위해 사용된 공간 예측 모드에 따라 인과적인 이웃 화소들로부터 획득된다.And r, p and

Are respectively approximated to the reconstructed signal (approximate to the original decompressed signal s), the predictive signal, and the compressed residual signal (the original decompressed residual signal:

= s-p (where s is the original signal), all of which are 4x4 matrices of integer values in this example. Residual values

Can be reconstructed by the inverse transform of the transform coefficients. The prediction values p are obtained from causal neighboring pixels according to the spatial prediction mode used to encode them.

The following are observations that affect the reconstruction of the pixels (inhumane neighbors of H.264) in intra-4x4 coded macroblocks located just below the slice boundary. In a 16x16 macroblock, these blocks contain the best four 4x4 blocks located just below the slice boundary. For example, blocks with indices b0, b1, b4, and b5 in the 16x16 pixel macroblock shown in FIG. 9 represent blocks just below slice boundary AA '.

7 illustrates an aspect of an intra-4 × 4 coded block just below the slice boundary. Line AA 'represents the slice boundary mentioned above and 4x4 block 702 is the current block being reconstructed. The nine neighboring pixels 704 above slice boundary line AA 'that can usually be used to perform spatial prediction in intra-4x4 coding are because they are located on the other side of the slice boundary, and they are another slice. It is not available because it belongs to. Spatial prediction in addition to the coding dependence of any other prediction across slice boundaries is not allowed in H.264 because slices serve as resynchronization points.

8 shows a list of pixels and neighboring pixels in an intra-4x4 coded block. Since the pixels above slice boundary AA 'are not available for spatial prediction, the neighboring pixels of block 702 available for prediction are pixels {I, J, K, L}. This indicates that the intra-4x4 coding prediction modes allowed for the 4x4 block 702 are i) mode 1 (horizontal), ii) mode 2 (DC), and iii) mode 8 (horizontal-up). it means. If line BB 'of FIG. 7 represents another slice boundary, none of the pixels {I, J, K, L} or {M, A, B, C, D, E, F, G, H} is spaced It will not be available for prediction. In this case, the allowed intra-4 × 4 coding prediction mode available is Mode 2 (DC) with a reference value of 128 for all the pixels in block 702.

Thus, in the most common case, information for decoding and reconstructing some or all of the pixels of an intra-4x4 coded block located directly below the slice boundary is

1. Intra-4 × 4 prediction mode indicator;

2. residual information (quantized transform coefficients); And

3. values of four neighboring pixels {I, J, K, L} in FIG. 8 located directly to the left of the 4x4 block;

. This sufficient data set may enable reconstruction of all pixel values {a, b, c, ..., n, o, p} of the current 4x4 block. In addition, this data set is sufficient to reconstruct the values {d, h, l, p} of the pixel subset, which values are then used to reconstruct the next 4x4 block located immediately to the right. May be used.

The following are observations that affect the reconstruction of intra-16 × 16 coded macroblocks (inhumane neighbors of H.264) located just below the slice boundary. Here again, the best four 4 × 4 blocks (ie those with block indices b0, b1, b4, and b5 of FIG. 9) of an intra-16 × 16 coded macroblock located just below the slice boundary. have an interest in.

9 illustrates an aspect of an intra-16 × 16 coded macroblock located below a slice boundary. Line AA 'represents the slice boundary mentioned above, and the four 4x4 blocks labeled b0, b1, b4, and b5 constitute part of the 16x16 macroblock under consideration for reconstruction. Usually, 17 neighboring pixels on line AA 'that can be used to perform intra-16x16 spatial prediction are available because they are located on the other side of the slice boundary, and because they belong to another slice. Not. In this example the potential availability of 16 neighboring pixels, located directly to the left of line BB ', indicates that the intra-16 by 16 coding spatial prediction modes allowed for the current macroblock are: i) mode 1 (horizontal), And ii) mode 2 (DC). For example, if the line BB 'represents another slice boundary (or the left boundary of the video frame), the sixteen neighboring pixels located immediately to the left of the line BB' also have seventeen pixels located above the line AA '. If no pixels are also available, the allowed intra-16x16 prediction mode is mode 2 (DC).

If the current macroblock is encoded using intra-16 × 16 prediction mode 1 (horizontal), the best four neighboring pixels located to the left of line BB 'and directly below line AA' are the current 16 × 16 pixels. It is sufficient to decode and reconstruct the four best 4x4 blocks in the macroblock. This is consistent with the structure described above, which enables decoding of the best four 4x4 blocks in an intra-4x4 coded macroblock.

However, if the current macroblock is encoded using intra-16 × 16 spatial prediction mode 2 (DC), and it is not directly on the right side of the slice boundary nor on the left frame boundary, then all 16 located immediately to the left of the line BB '. Neighbor pixels are used to decode and reconstruct the best four 4x4 blocks (in addition to all others in the column) in the current MB. This is an undesirable situation. In one aspect, it is beneficial to avoid encoding with intra-16 × 16 spatial prediction mode 2 (DC) just below the slice boundary. The best four neighboring pixels may be used for reconstruction of the pixels below the slice boundary (eg, pixels I, J, K and L of FIG. 8).

In one aspect, intra-16 × 16 coding of macroblocks located directly below the slice boundary should be limited to spatial prediction mode 1 (horizontal), unless they are located directly to the right of the slice boundary or at the left frame boundary. . This allows computationally efficient reconstruction of the rightmost four pixels of all the best 4x4 blocks of the row. This allows later computationally efficient reconstruction of the best four pixels of all the best 4x4 blocks of the row.

10 illustrates one aspect of an 8x8 chromaticity block located directly below the slice boundary. Line AA 'represents a slice boundary, and two 4x4 blocks just below line AA' and to the right of line BB 'constitute data for one of two chroma channels Cr and Cb. Nine neighboring pixels above slice boundary line AA 'are not available for spatial prediction in this example because they are located on the other side of the slice boundary, and because they belong to another slice. The availability of eight neighboring pixels, located directly to the left of line BB ', indicates that the chromaticity channel intra prediction modes allowed for the current MB are limited to i) mode 0 (DC) and ii) mode 1 (horizontal). Means that. When line BB 'is also a slice boundary or the left boundary of a video frame, neither the eight neighboring pixels located directly to the left of line BB' nor the nine pixels located directly above line AA 'are available for spatial prediction. not. In this case, the allowed chromaticity channel intra prediction mode is mode 0 (DC).

When the chroma channels of the current intra coded macroblocks are encoded using the intra-8 × 8 chromatic horizontal prediction mode, the four best neighboring pixels located directly to the left of the line BB 'are the best two fours in the current MB. It may be necessary to decode and reconstruct × 4 chromaticity blocks. It should be noted that there are two 8x8 chromaticity blocks corresponding to one 16x16 luma macroblock.

Similarly, when the current intra-coded macroblock chromaticity channels Cr and Cb are encoded using intra-8 × 8 chroma prediction mode 2 (DC), eight neighbors located immediately to the left of the line BB '. The availability of the pixel is suitable for decoding and reconstructing the best two 4x4 blocks. This is also consistent with the structure described above.

In one aspect, intra-8 × 8 coding of chroma channels (Cr and Cb) of intra-coded macroblocks, located just below the slice boundary, means that they are not located directly to the right of the slice boundary, or the left frame boundary. If not, it should be limited to spatial prediction mode 1 (horizontal). This allows computationally efficient reconstruction of the rightmost four pixels of all the best 4x4 blocks of the row. This allows later computationally efficient reconstruction of the best four pixels of all the best 4x4 blocks of the row. This is consistent with the structure described above that allows decoding of the best four 4x4 blocks in intra-coded macroblocks chromaticity channels (Intra-4x4 coded macroblocks with limitations, and intra All 16x16 coded macroblocks used intra-16x16 DC spatial prediction mode as described above).

H. At 264 Intra Efficient partial decoding of coded samples

Partial decoding of the four rightmost pixels of the 4x4 pixel blocks has been shown to allow decoding of some and / or all of the pixels of the intra-coded blocks on the right side of the initial 4x4 block. The problem of efficiently decoding the fourth, ie, last, column of the residual component of the 4x4 intra-coded block contributing to the reconstruction of the final pixel values for positions {d, h, l, p} of FIG. It will be covered now. This example uses base images of H.264 integer conversion. However, it should be noted that the base images of the other variants can be manipulated in a similar manner, allowing similar efficient partial decoding. Other variants that may be partially decoded using these methods include, but are not limited to, DCT (Discrete Cosine Transform), DFT (Discrete Fourier Transform), Adama (or Walsh-Adamar) Transform, Discrete Wavelet Transform , DST (Discrete Sine Transformation), Har Transform, Slope Transformation, and KL (Karunen Rube) Transformation.

In general, the forward transform of an N × N matrix [Y] of multimedia samples using the transform matrix [T] resulting in a transform coefficient matrix [w] takes the form:

The corresponding inverse transform for reconstructing the multimedia sample matrix [Y] is of the form:

The transforms represented by equations (3) and (4) can be regarded as two one-dimensional (1D) transforms, respectively, resulting in a two-dimensional (2D) transform. For example, the [Y] [T] matrix multiplication operation may be considered a 1D row transformation and the [T] ^T [Y] matrix multiplication operation may be considered a 1D column transformation. The combination forms a 2D transform. Another way to think about the 2D transformation of the N × N matrix [Y] is to perform the transform coefficients by performing the 2D base image and the N ² dot product of [Y] corresponding to the 2D transformation characterized by the transformation matrix [T]. Derive the same set of N ² values as the set of?

The base images of a given transform [T] can be calculated by setting one of the transform coefficients to 1 and all others to 0, and taking the inverse transform of the resulting coefficient matrix. For example, using the 4x4 transform coefficient matrix [w], w ₁₁ coefficients are set to 1 and all others are set to 0, and using H.264 integer transform [T _H ], equation (4) is ,

Cause.

Sum 16 (N ² , where N = 4) matrices formed using the individual transform coefficients (weights) in [w] to weight (scale) 16 (N ² , where N = 4) base images By doing so, the entire reconstructed matrix [Y] can be calculated. This is not as efficient as the fast conversion method for calculating the entire matrix. However, reconstruction of a subset, such as a row or column, can be done much more efficiently than a fast transform using base images.

The 16 base images associated with the H.264 4 × 4 integer conversion process for the remaining 4 × 4 blocks can be determined as follows, where sij (if i, j ∈ {0, 1, 2, 3}) is The base image associated with the i th horizontal and j th vertical frequency channels.

Looking carefully at these 16 basic images, it can be seen that their last column actually contains four distinct vectors, except for the scale factor. This should be intuitively obvious because the last column, which is a 4x1 matrix / vector, lies in four-dimensional vector space, and because it can be represented exactly 4 fundamental vectors.

의 마지막 열에 대한 재구성 표현 식이,If quantized transform coefficients (ie levels, zij i, j∈ {0, 1, 2, 3}) are received in the bitstream, they are coefficients w'ij i, j ∈ {0, 1, 2 , 3} to be rescaled (requantized). These quantized transform coefficients (w'ij i, j ∈ {0, 1, 2, 3}) can then be parsed into groups that are combined, and the inverse transform process is emulated (ie, the base in the synthesis process). Multiplied by the last column (or vector) of base images to generate weights to weight the images. This observation shows that 4 × 4 residual signals corresponding to positions {d, h, l, p} in FIG. 8.

The reconstruction expression for the last column of,

It can be recorded as.

Once four different combinations of scalar quantities w'ij of the four sets of parentheses are calculated, right shift and addition / subtraction can be used to complete the scaling / calculation of each base vector. The calculation of the reconstructed samples is then straight forward. By starting at the far left of the frame or just to the right of the slice boundary, spatial prediction mode 2 (DC) may be used and all pixel values have a reference (or prediction) value equal to 128 (see p in equation 1 above). It is known. Reconstructed samples [r _d , r _h , r _l , r _p ] corresponding to positions {d, h, l, p} for this first leftmost block,

은 식 (7) 로 계산된다. 이 블록의 우측의 4×4 블록들은 그 후, 식 (1) 의 예측 신호 컴포넌트 p 를 발생시키기 위해 좌측의 블록으로부터 적절한 재구성된 값들을 이용함으로써 계산될 수 있다 (발생된 예측 신호 값들은, 어느 공간 예측 모드가 재구성되는 4×4 블록을 인코딩하도록 사용되었는지에 의존한다). 슬라이스 경계 아래에 위치되는 4×4 블록들에 대한 예측 값들을 계산하는 예가 이제 설명된다.Can be calculated as: where reconstructed residual values

Is calculated by equation (7). The 4x4 blocks on the right side of this block can then be calculated by using the appropriate reconstructed values from the block on the left to generate the prediction signal component p of equation (1) (the generated prediction signal values Depends on whether the spatial prediction mode was used to encode the 4x4 block to be reconstructed). An example of calculating prediction values for 4x4 blocks located below the slice boundary is now described.

을 사용하여 미리 재구성된 위치들을 나타낸다. 화소 위치들 {d, h, l, p} 에 대한 잔여 신호 컴포넌트 값들

의 재구성 후에, 식 (1) 에 따라 재구성을 완결하기 위해 동일 세트의 위치들 {d, h, l, p} 에 대한 예측 신호 컴포넌트 값들

이 발생될 것이다. 화소들 {d, h, l, p} 을 포함하는 인트라-4×4 코딩된 4×4 블록이 슬라이스 경계 바로 아래에 있다고 가정하면, 이 4×4 블록에 대해 예측 신호를 발생시키기 위해 사용될 수 있었던 인트라-4×4 공간 예측 모드들은 다음 중 하나일 수 있다 :11 shows some of the multimedia samples located just below the slice boundary. The pixels may include luminance and chromaticity values. Pixel positions {q, r, s, t} are pixel values (e.g., calculated using equation (7) above)

To indicate the pre-reconstructed positions. Residual signal component values for pixel positions {d, h, l, p}

After reconstruction of, the prediction signal component values for the same set of positions {d, h, l, p} to complete the reconstruction according to equation (1)

Will occur. Assuming that an intra-4x4 coded 4x4 block containing pixels {d, h, l, p} is just below the slice boundary, it can be used to generate a prediction signal for this 4x4 block. Intra-4 × 4 spatial prediction modes that were present may be one of the following:

1, intra-4 × 4 spatial prediction mode 1 (horizontal):

Referring to FIG. 11, the prediction signal component values are

And includes zero addition, zero arithmetic shift, and zero multiplication.

2. Intra-4 × 4 Spatial Prediction Mode 2 (DC):

If the pixels at positions {q, r, s, t} are not available, the prediction signal component values are

And includes zero addition, zero arithmetic shift, and zero multiplication.

If {q, r, s, t} is available, the prediction signal component values are

Is given by 4 additions, 1 arithmetic shift and 0 multiplication.

Intra-4 × 4 Spatial Prediction Mode 8 (Horizontal-Up):

The predictive signal component values are

6 add, 4 arithmetic shift, and 0 multiplication, or 8 add, 2 arithmetic shift, and 0 multiplication.

One or more observations regarding the rescaling process (inverse quantization of zij i, j ∈ {0, 1, 2, 3} to generate w'ij i, j ∈ {0, 1, 2, 3}) yield significant calculations It may reveal other sources of savings. In addition to the dependence on protons and parameters, the rescaling factor vij i, j ∈ {0, 1, 2, 3} used to scale zij ij ij {0, 1, 2, 3} is also To process the following positional structure in a 4x4 matrix:

Here, three groups of rescaling factors including [v00, v20, v02, v22], [v11, v31, v13, v33] and [v10, v30, v01, v21, v12, v32, v03, v23], respectively. Have the same value for a given quantization parameter QP _Y. This can preferably be used to reduce the number of multiplications associated with the occurrence of w'ij from zij as follows. In the given weighted principle vector sum formula (Equation (7)) for reconstructing the last column of the 4x4 residual signal, the first weight that weights the base vector [1 1 1 1] ^T is the two weights Note that we include the sum of w'00 and w'20 rather than the individual values of. Thus, instead of computing two values, usually w'00 and w'20, usually involving two integer multiplications, instead of adding them up, first add z00 and z20 and then add this sum to v00 = v20 Rescaling to obtain the same final value for (w'00 + w'20) through one integer multiplication.

In addition to their direct reduction in the computational steps for performing this partial decoding, a fast algorithm for calculating the desired last column and the first (best) row of the 4x4 residual signal can also be designed.

Another practical fact that may result in low computational steps for this partial decoding process is that, of a maximum of 16 quantized coefficients in the residual signal block, usually a fraction, typically less than 5, is actually non-zero. In connection with this fact, in the above, it can be used to further reduce or almost halve the accompanying multiplication number.

Those skilled in the art will appreciate that a formula similar to Equation (7) above may be derived to reconstruct any column, row, diagonal or any portion and / or combination thereof. For example, the upper row values of the base images (Equations 6a-6p above) correspond to corresponding transform coefficients w ' _ij to reconstruct the pixels directly below the slice boundary (see pixel positions {ABCD} in FIG. 11). Can be combined with and depends on the same four pixel positions {dhlp} on the left side of the block. Other subsets of multimedia samples that can be reconstructed using these methods will be apparent to those skilled in the art.

12 is a functional block diagram illustrating another example of a decoder device 150 that may be used in a system such as that shown in FIG. 1. This aspect includes: means for receiving transform coefficients associated with multimedia data, first determiner means for determining a set of multimedia samples to be reconstructed, and second determiner for determining a set of received transform coefficients based on the multimedia samples to be reconstructed Means, and generator means for processing the determined set of transform coefficients to generate reconstructed samples corresponding to the determined set of multimedia samples. Some examples of this aspect include that the receiving means comprises a receiver 202, the first determiner means comprises a multimedia sample determiner 204, the second determiner means comprises a transform coefficient determiner 206, and generator means. And having this reconstructed sample generator 208.

FIG. 13 is a functional block diagram illustrating another example of a decoder device 150 that may be used in a system such as that shown in FIG. 1. This aspect includes: means for receiving transform coefficients associated with multimedia data, first determiner means for determining a set of multimedia samples to be reconstructed, and second determiner for determining a set of received transform coefficients based on the multimedia samples to be reconstructed Means, and generator means for processing the determined set of transform coefficients to generate reconstructed samples corresponding to the determined set of multimedia samples. Some examples of this aspect include a module 1302 that the receiving means receives, the first determiner means comprises a module 1304 for determining samples for reconstruction, and the second determiner means determines the transform coefficients. A module 1306, the generator means comprising a module 1308 for processing the transform coefficients.

Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the description above may be voltages, currents, electromagnetic waves, magnetic fields or magnetic particles. , Optical systems or optical particles, or any combination thereof.

Those skilled in the art may also implement various illustrative logical blocks, modules, and algorithm steps described in conjunction with the examples disclosed herein in electronic hardware, firmware, computer software, middleware, microcode, or a combination thereof. To clearly illustrate hardware and software compatibility, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed methods.

The various illustrative logic blocks, components, modules, and circuits, along with the examples disclosed herein, are general purpose processors, digital signal processors (DSPs), custom integrated circuits (ASICs), FPGAs, designed to perform the functions described herein. (Field programmable gate array) or other programmable logic device, separate gate or transistor logic, separate hardware components, or any combination thereof. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented in a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core or ASIC core, or any other configuration.

The steps of a method or algorithm described in conjunction with the examples disclosed herein may be contained directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module resides in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, optical storage medium, or any other form of storage medium known in the art. You may. An example storage medium is coupled to the processor such that a process can read information from and write information to the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a wireless modem. In the alternative, the processor and the storage medium may reside as discrete components in a wireless modem.

The previous description of the disclosed examples is provided to enable any person skilled in the art to make or use the disclosed methods and apparatus. Various modifications to these examples are readily known to those skilled in the art, and the principles defined herein may be applied to other examples and additional elements may be added.

Thus, methods and apparatus for performing highly efficient partial decoding of multimedia data are described.

Claims (65)

Receiving transform coefficients associated with the multimedia data; Determining a set of multimedia samples to be reconstructed, wherein the set of multimedia samples to be reconstructed comprises at least one sample adjacent a portion of the lost multimedia; Determining the set of received transform coefficients based on the reconstructed multimedia samples; Processing the determined set of transform coefficients to generate reconstructed samples corresponding to the determined set of multimedia samples; And Estimating a set of concealment multimedia samples for the portion of the lost multimedia based on the reconstructed samples. The method of claim 1, And said processing comprises scaling said determined set of transform coefficients. The method of claim 2, Scaling the transform coefficients comprises dequantizing. The method of claim 1, And wherein said multimedia samples in said determined set include multimedia samples referenced when other multimedia samples are encoded. The method of claim 1, And wherein said multimedia samples of said determined set comprise multimedia samples within a first slice of multimedia data in contact with a second slice of multimedia data. The method of claim 1, Wherein the received transform coefficients are associated with a matrix of transformed multimedia samples as a set, the reconstructed samples comprising a subset of the matrix of multimedia samples. The method of claim 1, And said processing includes dividing said determined set of transform coefficients into a plurality of groups. The method of claim 7, wherein The processing further includes calculating a value for each group, And said calculation is based on an encoding method that generated said transform coefficients. The method of claim 8, The processing step, Determining an array for each group based on the encoding method that generated the transform coefficients; And Generating a set of reconstructed samples of the multimedia data based on the values and the arrays. delete The method of claim 1, Generating a set of transform coefficients corresponding to the estimated set of hidden multimedia samples. The method of claim 1, And the reconstructed samples are non-causal to the estimated set of hidden multimedia samples. The method of claim 1, Receiving a directivity mode indicator associated with each reconstructed sample; And Estimating a set of hidden multimedia samples based on the reconstructed samples and the directional mode indicators. A multimedia data processor for multimedia data concealment, Receive transform coefficients associated with the multimedia data; Determine a set of multimedia samples to be reconstructed; Determine the set of received transform coefficients based on the reconstructed multimedia samples; Process the determined set of transform coefficients to generate reconstructed samples corresponding to the determined set of multimedia samples, Estimate a set of hidden multimedia samples for the portion of the lost multimedia based on the reconstructed samples, Wherein the set of reconstructed multimedia samples comprises at least one sample adjacent to a portion of the lost multimedia. The method of claim 14, The multimedia data processor is further configured to scale the determined set of transform coefficients. The method of claim 14, The multimedia data processor is further configured to dequantize the determined set of transform coefficients. The method of claim 14, And the multimedia samples of the determined set include multimedia samples that are referenced when other multimedia samples are encoded. The method of claim 14, And the multimedia samples of the determined set include multimedia samples within a first slice of multimedia data abutting a second slice of multimedia data. The method of claim 14, And the received transform coefficients are associated with a matrix of transformed multimedia samples as a set, wherein the reconstructed samples comprise a subset of the matrix of multimedia samples. The method of claim 14, The multimedia data processor is further configured to divide the determined set of transform coefficients into a plurality of groups. The method of claim 20, The multimedia data processor is further configured to calculate a value for each group, And said calculation is based on an encoding method that generated said transform coefficients. The method of claim 21, The multimedia data processor also, Determine an array for each group based on the encoding method that generated the transform coefficients; And generate a set of reconstructed samples of the multimedia data based on the values and the arrays. delete The method of claim 14, And the multimedia data processor is further configured to generate a set of transform coefficients corresponding to the estimated set of hidden multimedia samples. The method of claim 14, And the reconstructed samples are in causal to the estimated set of hidden multimedia samples. The method of claim 14, The multimedia data processor also, Receive a directional mode indicator associated with each reconstructed sample; And estimate a set of hidden multimedia samples based on the reconstructed samples and the directional mode indicators. A receiver for receiving transform coefficients associated with the multimedia data; A first determiner for determining a set of multimedia samples to be reconstructed, wherein the set of multimedia samples to be reconstructed comprises at least one sample adjacent a portion of the lost multimedia; A second determiner for determining the set of received transform coefficients based on the reconstructed multimedia samples; A generator for processing the determined set of transform coefficients to generate reconstructed samples corresponding to the determined set of multimedia samples; And And an estimator for estimating a set of hidden multimedia samples for the portion of the lost multimedia based on the reconstructed samples. 28. The method of claim 27, And the generator to scale the determined set of transform coefficients. 28. The method of claim 27, And the generator inverse quantizes the determined set of transform coefficients. 28. The method of claim 27, And said multimedia sample of said determined set comprises multimedia samples referenced when other multimedia samples are encoded. 28. The method of claim 27, And wherein said multimedia samples in said determined set comprise multimedia samples in a first slice of multimedia data in contact with a second slice of multimedia data. 28. The method of claim 27, And the received transform coefficients are associated with a matrix of transformed multimedia samples as a set, the reconstructed samples comprising a subset of the matrix of multimedia samples. 28. The method of claim 27, And wherein the generator divides the determined set of transform coefficients into a plurality of groups. The method of claim 33, wherein The generator calculates a value for each group, And said calculation is based on an encoding method that has generated said transform coefficients. The method of claim 34, wherein Wherein the generator determines an array for each group based on the encoding method that generated the transform coefficients, and generates a set of reconstructed samples of the multimedia data based on the values and the arrays. Concealment device. delete 28. The method of claim 27, And the estimator generates a set of transform coefficients corresponding to the estimated set of hidden multimedia samples. 28. The method of claim 27, And the reconstructed samples are in causal to the estimated set of hidden multimedia samples. 28. The method of claim 27, The receiver receives a directional mode indicator associated with each reconstructed sample, And an estimator for estimating a set of hidden multimedia samples based on the reconstructed samples and the directional mode indicators. Means for receiving transform coefficients associated with the multimedia data; First determiner means for determining a set of multimedia samples to be reconstructed, the set of multimedia samples to be reconstructed comprising at least one sample adjacent a portion of the lost multimedia; Second determiner means for determining the set of received transform coefficients based on the reconstructed multimedia samples; Generator means for processing the determined set of transform coefficients to generate reconstructed samples corresponding to the determined set of multimedia samples; And Means for estimating a set of hidden multimedia samples for the portion of the lost multimedia based on the reconstructed samples. 41. The method of claim 40, And the generator means for scaling the determined set of transform coefficients. 41. The method of claim 40, And the generator means inverse quantizes the determined set of transform coefficients. 41. The method of claim 40, And said multimedia sample of said determined set comprises multimedia samples referenced when other multimedia samples are encoded. 41. The method of claim 40, And wherein said multimedia samples in said determined set comprise multimedia samples in a first slice of multimedia data in contact with a second slice of multimedia data. 41. The method of claim 40, And the received transform coefficients are associated with a matrix of transformed multimedia samples as a set, wherein the reconstructed samples comprise a subset of the matrix of multimedia samples. 41. The method of claim 40, The generator means for dividing the determined set of transform coefficients into a plurality of groups. The method of claim 46, The generator means calculates a value for each group, And said calculation is based on an encoding method that has generated said transform coefficients. 49. The method of claim 47, The generator means for determining an array for each group based on the encoding method that generated the transform coefficients and generating a set of reconstructed samples of the multimedia data based on the values and the arrays. Data concealment device. delete 41. The method of claim 40, And the means for estimating generates a set of transform coefficients corresponding to the set of estimated hidden multimedia samples. 41. The method of claim 40, And the reconstructed samples are in causal to the estimated set of hidden multimedia samples. 41. The method of claim 40, The receiving means receives a directional mode indicator associated with each reconstructed sample, And means for estimating a set of hidden multimedia samples based on the reconstructed samples and the directional mode indicators. At run time, let the machine Receive transform coefficients associated with the multimedia data; Determine a set of multimedia samples to be reconstructed; Determine the set of received transform coefficients based on the reconstructed multimedia samples; Process the determined set of transform coefficients to generate reconstructed samples corresponding to the determined set of multimedia samples; Instructions for concealing multimedia data, causing to estimate a set of hidden multimedia samples for the portion of the lost multimedia based on the reconstructed samples, And the set of reconstructed multimedia samples comprises at least one sample adjacent to a portion of the lost multimedia. 54. The method of claim 53, And the instructions further cause the machine to scale the determined set of transform coefficients. 54. The method of claim 53, The instructions further cause the machine to dequantize the determined set of transform coefficients. 54. The method of claim 53, And said multimedia samples in said determined set comprise multimedia samples referenced when other multimedia samples are encoded. 54. The method of claim 53, And the multimedia samples of the determined set comprise multimedia samples within a first slice of multimedia data abutting a second slice of multimedia data. 54. The method of claim 53, And the received transform coefficients are associated with a matrix of transformed multimedia samples as a set, wherein the reconstructed samples comprise a subset of the matrix of multimedia samples. 54. The method of claim 53, And the instructions further cause the machine to divide the determined set of transform coefficients into a plurality of groups. The method of claim 59, The instructions also cause the machine to calculate a value for each group, And said calculation is based on an encoding method that generated said transform coefficients. The method of claim 60, The instructions also cause the machine to: Determine an array for each group based on the encoding method that generated the transform coefficients; Generate a set of reconstructed samples of the multimedia data based on the values and the arrays. delete 54. The method of claim 53, And the instructions further cause the machine to generate a set of transform coefficients corresponding to the estimated set of hidden multimedia samples. 54. The method of claim 53, And the reconstructed samples are in causal to the estimated set of hidden multimedia samples. 54. The method of claim 53, The instructions also cause the machine to: Receive a directional mode indicator associated with each reconstructed sample; And estimate a set of hidden multimedia samples based on the reconstructed samples and the directional mode indicators.

Images