JP2001016113A - Multimedia decoder provided with single external storage memory - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce memory band width of an external memory required for an integrated circuit in which plural functions of a video decoder, an audio decoder and a system controller or the like are integrated on one chip. SOLUTION: While a video decoder 24 performs decoding processing and continuously outputs image information, an audio decoder 23 performs the decoding processing except for the processing which most request memory band width. When the video decoder 24 is turned into blanking period, the audio decoder 23 simultaneously performs the processing most requiring the memory band width, namely, block switching processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a decoding apparatus for decoding image / audio data digitally compressed according to the MPEG standard or the like, and more particularly to a decoding apparatus for decoding digitally compressed image data and digitally compressed audio data simultaneously. The present invention relates to a semiconductor integrated circuit for performing display.

[0002]

2. Description of the Related Art With the spread of digital broadcasting and DVD and the advance of semiconductor integrated technology, a video decoder represented by an MPEG2 video decoder and an AC-3 decoder, which have conventionally been realized by separate integrated circuits, have been developed. An audio decoder represented by an MPEG audio decoder is being integrated into a single semiconductor integrated circuit. Video at low cost
In order to implement the audio decoder, not only the video decoder and the audio decoder are implemented in a single integrated circuit, but also the data memory used by the video decoder and the data memory used by the audio decoder are physically one. It is desirable to have a unified memory architecture that shares two memories.

On the other hand, it is generally known that a high data rate is required for memory access in order to handle video information. As an example, NT as shown in FIG.
FIG. 8 shows the time variation of the required memory bandwidth of the MPEG-2 video decoder of the SC standard. Processing for requesting memory access in the MPEG2MP @ ML video decoder of the NTSC standard is of four types: reading a bit stream, reading a reference image for motion compensation, storing a decoded image, and reading a decoded image for image display.

[0004] Assuming that the maximum average bit rate of the bit stream is 15 Mbps, and the reference image is read out for four fields in one frame period, the memory bandwidth required for each memory access is approximately 8 MB per second. The reading of both reference images is about 45 MB per second, the storage of decoded images is about 16 MB per second, the reading of decoded images for image display is about 54 MB per second, and a memory access of a maximum of 123 Mbytes per second is required. However, the video output data format has a period generally called a vertical blanking period, and no data is transmitted during this period.

In a standard called CCIRR601 for outputting digital data of the NTSC standard, a vertical blanking period is once every 16.7 ms and continues for about 1.3 ms. During this period, the required memory access amount decreases to about 69 Mbytes per second, which is about 50% of the peak rate.

In some cases, audio decoding requires high-rate memory access. As an example, consider a 5.1 channel MPEG2 AACLC profile decoder. The MPEG2 AAC decoder collectively processes 1024 samples of the sampled audio signal. This is called a block. Considering the case where the sampling frequency is 48 kHz, the period for one block is 21 ms.
The decoding process for one channel must be completed in about 3.5 ms to 4.2 ms per channel.

[0007] An example of a simplified block diagram showing the processing of the MPEG2 AAC decoder is shown in FIG. Here, the processing of MPEG2 AAC decoding is shown in FIG.
Assume that the processing is performed sequentially by a processor having a local memory of 8 KB as shown in FIG. ISO / IE
C JTCI / SC29 / WG N2005 “Revised Report on Complexity
According to "MPEG-2 AAC Tools", the AAC decoder requires 43.75 KB of RAM and 13.85103 ROM. Here, the ROM is an internal ROM.
Assume that the 4375 KB RAM stored at 34 is stored in external memory.

The local RAM is used not only as a work memory of the processor but also as a read buffer when data is transferred from the external memory to the processor and as a write buffer from the processor to the external memory.

FIG. 11 shows an example of a memory access amount required for each process. Of the input / output data, the data enclosed by the dotted line is held in the local memory, so that no access is made to the external memory. In the case of this example, the processing with the largest memory access amount in the series of processing is block switching. As shown in FIG. 12, the first half of the time series data of 2048 samples obtained by the filter bank in the previous stage is
4 samples and 10 stored in the delay buffer
This is a process of performing weighted addition on time-series data of 24 samples according to a curve called a window and outputting the result to an audio output buffer. This process requires about 50 MB of memory bandwidth per second,
For other processing, a memory bandwidth of about 1 MB per second is sufficient.

Generally, the operations of the audio decoder and the video decoder are asynchronous. Even in the combination of the NTSC video decoding and the MPEG2 AAC decoding described above, a period occurs in which the required memory bandwidth decreases every 16.7 ms on the video side as shown in FIG. A high memory bandwidth is required every 0.5 ms to 4.2 ms.

For this reason, the memory bandwidth of the video / audio decoder employing the unified memory architecture has a peak value of the required memory bandwidth of the video decoder and a peak value of the required memory bandwidth of the audio decoder as shown in FIG. It is necessary to prepare a bandwidth just by adding the values. In the previous example, this rate would be 173 MB per second. Generally, in addition to the audio decoder and the video decoder, there is a system demultiplexer that supplies a bit stream to each of them, and the bit stream is written to the external memory. When the required memory bandwidth is added, the memory bandwidth required for the external memory is about 180 MB per second.

[0012]

However, in order to configure an external memory having a high data rate, it is necessary to increase the operating frequency of the memory device or to arrange a plurality of memory devices in parallel. Either method increases the circuit amount of the decoder integrated circuit or the memory device, and causes an increase in cost.

A commonly used synchronous dynamic memory (synchronous DRA) having a bus width of 16 bits
When M) is driven at a clock frequency of 100 MHz, assuming that there is about 15% overhead associated with memory access, the memory bandwidth is 174 MB per second.
It is. Therefore, in order to realize the aforementioned memory bandwidth of 180 MB per second, the memory bus width must be 32 bits. If the bus width can be reduced to 16 bits, the number of pins of the integrated circuit can be reduced, and there are significant advantages such as reduction in assembly cost and improvement in reliability.

[0014] The present invention has been made in view of the above point, and, of two processes which were originally asynchronous, a period in which one requested data rate is low and a period in which the other requested data rate is high are concentrated. Accordingly, it is an object of the present invention to provide a multimedia decoding apparatus which realizes video audio decoding processing in one integrated circuit without unnecessarily increasing a data rate even in a unified memory architecture.

[0015]

According to the present invention, there is provided a first decoder for decoding at least an encoded video signal, and a first decoder which operates asynchronously with the first video signal and outputs a signal different from the video signal. A second decoder for decoding, an external storage memory used as a data memory by the first decoder and the second decoder, a memory controller for transmitting and receiving data to and from the external storage memory, and a first decoding A data bus connecting the decoder, the second decoder, and the memory controller, and among a series of decoding processes performed by the second decoder, a process having a large access to the external storage memory is made independent, During the period when the memory bandwidth required by the decoder is large, the second decoder intensively performs processing with less memory access, and during the period when the memory bandwidth required by the first decoder is small, the second decoder Vessel is memory Provided is a multimedia decoding device that performs processing with large access at once.

According to the present invention, the multimedia decoding apparatus determines whether the video decoder is currently outputting video.
A means for notifying the audio decoder of the blanking period, a means for making the processing with the highest required memory bandwidth among the processings of the audio decoder independent, and the audio decoder being notified that the video decoder has entered the blanking period. Then, means for collectively performing processing having the highest required memory bandwidth over several blocks is provided.

According to the present invention, of the decoding processes of the video decoder and the audio decoder which operate asynchronously, the blanking period in which the required memory bandwidth of the video decoder is low and the required memory bandwidth of the audio decoder are high. The period can be synchronized. For this reason, the bandwidth of a single memory conventionally expressed by the sum of the maximum required memory bandwidths of both is reduced to about the sum of one relatively low memory bandwidth and the other maximum required memory bandwidth, It becomes possible to realize a video audio decoder based on a unified memory architecture in which a video decoder and an audio decoder are integrated at low cost.

[0018]

Embodiments of the present invention will be described below with reference to the drawings.

According to the multimedia decoding apparatus of the first embodiment of the present invention shown in FIG.
Performs decoding processing of MPEG2 video information. The maximum memory bandwidth required by the video decoder 13 is about 123 MB per second. In decoding MPEG2 video, the required memory bandwidth varies depending on the type of picture being decoded.

FIG. 2 shows the relationship between the type of each picture and the required memory bandwidth. According to this, I
Since a picture does not need to read a reference image, the required memory bandwidth is smaller than that of a B picture, and is approximately 80 per second.
MB. An I picture is generally transmitted at a frequency of about one picture every 0.5 seconds.

The still picture processor 12 performs decoding of a still picture that does not require strict real-time processing, such as EPG (Electronic Program Guide) in digital broadcasting. Memory access between the video decoder 13 and the still image processor 12 is performed via the memory controller 15. The memory controller 15 arbitrates when the processors 12 and 13 access the external memory 16 using the data bus 10. Now, when the still image processor 12 is a JPEG (Joint Picture Coding Exp
ert Group) when decoding a still image.

FIG. 3 shows an example of the JPEG decoding processing procedure. According to this, first, the encoded bit stream is Huffman decoded (S11). The DC coefficient of the Huffman-decoded bit stream is DPCM
The AC coefficient is run-length decoded together with the decoding (S12) (S13). The AC coefficient obtained by the run-length decoding is zigzag scanned (S1).
4). The DCT coefficients obtained by the DPCM decoding and the zigzag scan are inversely quantized (S15), and are inversely DCTed to be reconstructed into image information.

In the above-described JPEG decoding processing, an access to the external memory 16 occurs at least for reading the bit stream and writing the decoded image.

The video decoder 13 sends out a signal indicating permission of memory access to the still picture processor 12 using the communication means 14 when it is determined that there is no reference image read out during the blanking period or the decoding of the I picture. I do. Upon receiving this signal, the still image processor 12 starts a memory access associated with JPEG decoding.

The video decoder 13 sends a signal indicating that memory access is not permitted to the still picture processor 12 by using the communication means 14 when the blanking period or the decoding of the I picture is completed. Still image processor 12
, Upon receiving this signal, stops the memory access accompanying JPEG decoding.

At the time of decoding an I picture, the memory bandwidth (45 MB / s) for reading the reference image is released, and the memory bandwidth for displaying the decoded images (54 MB / s) is released during the blanking period. , JP
An average of about 6.9 MB per second of memory bandwidth is allocated for EG decoding. Still image processor 1
If there are enough write buffers and read buffers in 2, the still image processor 12 can continue to operate without waiting for memory access.

FIG. 4 is a block diagram of a multimedia decoding apparatus according to a second embodiment of the present invention. When the bit stream is input to the system demultiplexer 22,
The audio bit stream and the video bit stream are classified and stored in separate areas of the external memory 26, respectively. The stored bit streams are input to the video decoder 24 and the audio decoder 23, respectively. In the video decoder, as in the first embodiment,
Video information is decoded, and the decoded video information is output as video. At the time of decoding, a total bandwidth of about 123 MB per second is required to temporarily store decoded image information, read out previously decoded images, and send image information to video output. . Also,
In the audio decoder, decoding processing of audio information is performed, and an audio signal is output. The maximum value of this request rate is 50 MB per second.

When the decoding process is started, the video bit stream is decoded by the video decoder 24, and the decoding of the MPEG2 AAC bit stream is started by the audio decoder 23 composed of the MPEG2 AAC decoder shown in FIG. However, in this audio decoder, only the Huffman decoding, inverse quantization, TNS, and filter bank processing in the processing flow of the MPEG2 AAC decoder shown in FIG. 9 are performed, and the last block switching is not performed. However, intermediate results generated up to the filter bank are sequentially stored in the external memory 26. FIG. 5 shows the bandwidth of the memory access with the external memory at this time. Since the 8 KB local RAM of the MPEG2 AAC decoder shown in FIG. 10 operates as a write buffer, the 8 KB time domain sample information output from the filter bank is:
What is necessary is just to write to the external memory 26 during the processing time required for the next Huffman decoding. Therefore, the memory bandwidth required during a series of processing is at most about 7.3 MB per second.

When the video decoder 24 enters a blanking period during which no video output is performed, the video decoder 24 notifies the audio decoder 23 that the current time is during the blanking period by using the communication means 27. Upon receiving the notification from the video decoder, the audio decoder 23 temporarily suspends the current decoding processing, and collects the accumulated block switching processing using the processing result of the filter bank that has been decoded so far. Do. This processing is based on the coefficient of window 2 words (wl, w2) and two filter bank output words (F1, F2), and out =
This is a process for obtaining an audio output by the expression wlu + w2F2. Therefore, these wl, w2, F1, F
By performing the processing while sequentially reading data 2, the consumption of the data memory can be extremely reduced. Therefore, it is not necessary to temporarily save the data memory associated with the processing performed before blanking in the external memory 26.
Also, the processing time of block switching is 0.2 m.
In order to read 8 KB of filter bank output data during s and output 2 KB of audio data, 50 KB per second is required.
A memory bandwidth of about MB is required. When all the block switching processes are completed, the audio decoder 3 restarts the paused process.

When the video decoder 24 decodes an NTSC signal, a blanking period occurs once every 16.7 ms. Since the audio decoder 3 must finish decoding one block within a time of about 3.5 ms, the number of block switching processes that must be performed in one blanking period is a maximum of five times. The time required for block switching of one block is 0.2 ms.
The block switching process performed during the blanking period of the video decoder 24 is about Ims at the maximum. This is shorter than the blanking period length of 1.3 ms. That is, this means that all the block switching processing in audio decoding can be completed within the blanking period of the video decoder 24.

During the blanking period of the video decoder 24, the upper limit of the memory bandwidth required by the video decoder is 68 MB per second. At this time, the upper limit of the memory bandwidth of the audio decoder is 50 MB per second. Therefore, the memory bandwidth required by both decoders during this period is about 118 MB per second.

On the other hand, the upper limit of the memory bandwidth required by the video decoder during the non-blanking period is approximately 123 M / s.
B. At this time, since the upper limit of the memory bandwidth of the audio decoder is 7.3 MB / sec, the total of the memory bandwidth required by both is about 130.3 MB / sec.
That is, as shown in FIG. 6, the required bandwidth of the external memory according to the present embodiment can be reduced by about 40 MB per second, compared with the conventional memory bandwidth shown in FIG.

[0033]

As described above, according to the present invention,
In a multimedia decoding device composed of a unified memory architecture, two processes, which operate asynchronously, such as video and audio decoding processes, are focused on. One is a period during which the memory bandwidth is not required the most, and the other is a memory bandwidth. In this case, the memory bandwidth required for the unified memory, which has been conventionally obtained by the sum of the maximum memory bandwidths of both, can be reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram showing the configuration of a second embodiment of the present invention.

FIG. 2 is a diagram illustrating a relationship between a picture type and a required memory bandwidth in MPEG-2 video decoding.

FIG. 3 is a diagram showing a decoding procedure of JPEG.

FIG. 4 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 5 shows the processing order of the AAC decoder and the required memory bandwidth in the embodiment of the present invention.

FIG. 6 shows a temporal change of a memory bandwidth in the case of an embodiment of the present invention.

FIG. 7 is a block diagram showing a configuration of a video decoder.

FIG. 8 is a diagram showing a change over time of the bandwidth of the external memory required by the video decoder;

FIG. 9 is a diagram showing a processing flow of an MPEG2 AAC decoder.

FIG. 10 is a block diagram showing a configuration of an MPEG2 AAC decoder.

FIG. 11 shows the required memory bandwidth for each AAC decoder process.

FIG. 12 is a block diagram showing processing of block switching in the MPEG2 AAC decoder.

FIG. 13: Required memory bandwidth of an MPEG2 AAC decoder

FIG. 14 is a diagram showing a conventional memory bandwidth setting method.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Data bus 12 ... Still picture processor 13 ... Video decoder 14 ... Communication means 15 ... Memory controller 16 ... Video frame buffer 21 ... Data bus 22 ... System demultiplexer 23 ... Audio decoder 24 ... Video decoder 25 ... Memory controller 26 ... External Memory 27: Communication means

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B060 CA00 DA08 5C059 KK11 MA00 MA04 MA23 MC01 MC11 ME02 ME05 PP01 PP05 PP06 PP07 SS26 SS30 UA05 UA34 UA38 5J064 AA04 BC01 BC02 BD07 CA05 CB12 CC07

Claims (5)

[Claims] 1. A multimedia decoding device having a single memory for decoding a video signal, decoding an audio signal, decoding a still image, a system control function, etc., comprising: A first decoder that decodes, and a second decoder that operates asynchronously with the first decoder and decodes a signal different from the video signal; and a second decoder that decodes a signal different from the video signal. An external storage memory used as a data memory by the device, a memory controller for transmitting and receiving data to and from the external storage memory, and a data bus connecting the first decoder, the second decoder, and the memory controller. In the series of decoding processes performed by the second decoder, the process in which access to the external storage memory is large is made independent, and during a period when the memory bandwidth required by the first decoder is large, The second decoder intensively performs processing with less memory access, and the second decoder collectively performs processing with large memory access while the memory bandwidth required by the first decoder is small. A multimedia decoding device characterized by the above-mentioned. 2. The multimedia decoding apparatus according to claim 1, wherein the second decoder is an audio decoder that performs block switching processing in a blanking period. 3. The second decoder responds to the end of the blanking period with Huffman decoding, inverse quantization and T
3. The multimedia decoding apparatus according to claim 2, wherein the audio decoder performs only NS and filter bank processing. 4. A multimedia decoding device comprising a single memory for decoding video signals, decoding audio signals, and system control functions, wherein the video decoder decodes at least an encoded video signal. An audio decoder that operates asynchronously with the video decoder and decodes an encoded audio signal; an external storage memory used as a data memory by the video decoder and the audio decoder; A memory controller for transmitting and receiving data to and from a storage memory; a data bus for connecting the video decoder, the audio decoder, and the memory controller; and a unit for transmitting information from the video decoder to the audio decoder; Out of a series of decoding processes performed by the While the decoder is outputting the video signal, the audio decoder intensively performs processing with less memory access, and during the blanking period when the video decoder does not output the video signal, the audio decoder A multimedia decoding device, which performs a large access process at a time. 5. The multimedia decoding apparatus according to claim 4, wherein said audio decoder performs block switching processing collectively in said blanking period.