JP2000348168A - Method, device, and medium for picture processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To minimize the occurrence of a page miss at the time of reading out picture data handled in the unit of macro blocks. SOLUTION: When one macro block consists of 16×16 pixels (bytes) and macro blocks are stored in a frame memory consisting of a DRAM(dynamic random access memory) or the like, addresses are successively assigned in the ascending order, for example, addresses 0000 to 0255 are assigned to a first macro block and addresses 0256 to 0512 are assigned to a second macro block. Macro blocks stored in this matter are read out in the ascending order of address.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus, method, and medium, and more particularly to an image processing apparatus, method, and medium suitable for writing and reading image data in macroblock units.

[0002]

2. Description of the Related Art Recently, MPEG (Moving Picture Experts G)
The image compression technology represented by Roup) 2 has advanced and is used in various fields. When decoding the image data encoded by MPEG2, the decoding is performed for each macroblock. When storing the image data decoded for each macro block in the frame memory, the memory addresses of the image data are allocated in the order of screen display. That is, as shown in FIG.
A Y macroblock consisting of 6 pixels is 720
In the case of a screen composed of × 480 pixels, as shown in FIG. 1B, the frame memory has 45 pixels horizontally and 3 pixels vertically.
It is stored in a state where 0 pieces are arranged.

[0003] When memory addresses are allocated and stored in this way, when displaying on a screen based on image data stored in the frame memory, appropriate addresses are allocated. For example, in the frame memory shown in FIG. 1A, when one line at the top of the screen is displayed, the addresses 0000 to 0719 (7
Since data can be read continuously up to 20 pixels of data), the occurrence of page misses can be minimized. Here, a page error is a DR
The time required to precharge a sense amplifier provided in a memory such as an AM (Dynamic Random Access Memory) (therefore, during precharging,
Processing such as data reading cannot be performed).

[0004] By the way, DR is generally used in a frame memory.
AM is used. DRAM consists of rows and columns.
n), and one row is composed of a storage element composed of 256 (sometimes 512, etc.) × 8 columns. Among these storage elements, the 8-bit element has an external terminal of 8 bits.
16 bits have 16 external terminals. This represents the number of bits that can be output in one clock. For an 8-bit element, eight bits are output for one clock, and for a 16-bit element, one bit is output.
16 bits are output by the clock. Then, within the same row, data can be continuously taken out. That is, in the 8-bit element, data of 256 × 8 bits is stored in one row, so that data for 256 clocks (data for 256 bytes) can be continuously read.

[0005] In a video decoder or the like, image data stored in a frame memory is transmitted in units of macroblocks, and the order of decoding is also in the order of macroblocks. One Y macro block is composed of 16 lines, one line being 16 pixels (accordingly, 16 bytes). Therefore, for example, in the case of the first Y macro block, the first line has addresses 0000 to 001.
5, the second line is divided into addresses (addresses 0720 to 0735) and the third line is divided into addresses 1440 to 1455 (not a series of addresses).

[0006]

After taking out the data stored in one row, it moves to the next row, and it takes about six clocks for precharging until the data stored in that row is read out. I have to wait. Thus, the precharge required to read data stored in another row is, as described above,
Called a page miss. In the 6-bit device described above, 25
In every 16 clocks (every 256 bytes of data are extracted) every 16 clocks, a page miss occurs every 128 clocks. As described above, when the amount of data that can be extracted in one clock increases, the number of occurrences of page misses (occurrence cycle) also decreases.

As shown in FIG. 1B, image data is stored and read out from addresses 0000 to 0719 to display the first horizontal line.
When reading up to 439 and displaying the second horizontal line, and so on, reading and displaying one line at a time, 8
In the bit element, a page miss occurs every 256 bytes.
Since a page miss results in a loss time, minimizing the occurrence of a page miss will maximize the capacity of the memory (DRAM).

On the other hand, in the case of the above-mentioned video recorder, data is stored in macroblock units and read out in macroblock units. Further, as described above, one Y macro block is composed of 16 lines, so that a page miss occurs every time one line is read. That is, every time one Y macro block is read, 16
Page errors will occur twice. In the case of chroma (Cb, Cr), one macroblock has 1
Since the line is 8 bytes and consists of 8 lines,
Each time one Cb (Cr) macroblock is read, 8
Page errors will occur twice.

When a 16-bit element is used for the frame memory, 16-bit (2 bytes) data can be output in one clock.
A page miss occurs once every clock, and a page miss occurs once every four clocks in the Cb macroblock and the Cr macroblock. Therefore, to read one line of the Y macro block, eight clocks (one line of the Y macro block takes 16
8 × 16 clocks (one Y macro block is composed of 16 lines) in order to read one Y macro block.
is necessary.

On the other hand, it is assumed that six clock times are spent for one page miss, and the loss time due to the page miss is 1 when reading one Y macro block.
Since 6 page errors occur, 6 × 16
It turns out that it is a clock. This indicates that the ratio of page misses is considerably higher than the time for reading one Y macro block, that is, 8 × 16 clocks, in other words, the loss time is large. Similarly, for a chroma macroblock, the ratio of the loss time to the time for reading data is large.

A page miss also occurs when decoding by motion compensation. That is, one macroblock of image data is extracted from an arbitrary position in the frame memory according to the motion vector attached to the macroblock to be decoded.
Each time a line is read, the address moves by one line, so a page miss occurs. In order to maximize the capabilities of DRAM and the like, there is a problem that it is necessary to minimize the loss time such as a page error as much as possible.

SUMMARY OF THE INVENTION The present invention has been made in view of such a situation, and addresses are stored in an ascending order with respect to image data of a macroblock, thereby suppressing occurrence of a page miss. Aim.

[0013]

According to a first aspect of the present invention, there is provided an image processing apparatus comprising: input means for inputting image data in units of macroblocks; and image data input by the input means.
It is characterized by including a storage means for allocating and storing addresses in ascending order, and a reading means for reading out image data stored in the storage means in ascending order of address.

According to a fourth aspect of the present invention, there is provided an image processing method comprising the steps of: inputting image data in units of macroblocks;
It is characterized by including a storage step of assigning and storing addresses to the image data input in the input step in ascending order, and a reading step of reading out the image data stored in the storage step in ascending order of the addresses.

According to a fifth aspect of the present invention, there is provided a storage medium storing an input step of inputting image data in units of macroblocks, allocating addresses to the image data input in the input step in ascending order, and storing the same. A reading step of reading out the image data stored in the step in ascending order of the address.

An image processing apparatus according to claim 1, wherein
In the image processing method described in the item (1) and the medium described in the item (5), addresses are assigned to the input image data in units of macroblocks in ascending order and stored, and the stored image data is sorted in ascending order of the address Is read.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 2 is a block diagram showing the configuration of an image decoding device that decodes data encoded according to the MPEG2 system. The image data received by the tuner 1 is demodulated by the demodulation processing unit 2 and error-corrected by the error correction processing unit 3. Further, the image data subjected to the error correction processing is sent to the software processing unit 4. The software processing unit 4 includes a CPU (Central Processing Uni
It consists of a software program executed by t). The demultiplexer unit 11 of the software processing unit 4 demultiplexes the input data into image data and audio data for each channel. The demultiplexed image data is decoded by the decoding unit 12 according to the MPEG method.

The output of the decoding unit 12 is converted by an image frame conversion processing unit 13 from an image frame of a high-definition television signal represented by, for example, HDTV into an NTSC (National Telev.
The image frame is converted to an image frame of a television signal of an ision System Committee) system. Further, the controller 14 controls each unit of the software processing unit 4 in addition to the decoding unit 12, and uses the cache memory 7 as needed for processing of image data. DMAC (Direct Memory Access Con
The controller 6 controls DMA transfer from the cache memory 7 to the frame memory 5. The frame memory 5 is composed of, for example, a DRAM, and the stored image data is output to an image display device (not shown).

FIG. 3 is a block diagram showing a more detailed configuration of the decoding section 12 shown in FIG. The image data output from the demultiplexer unit 11 is
2 is input to the variable length decoding unit 31. Variable length decoding unit 31
Performs variable length decoding on the input image data, and outputs the quantization step and the image data to the inverse quantization unit 32 and the motion vector to the motion compensation prediction unit 35. The inverse quantization unit 32 inversely quantizes the image data subjected to the variable length code processing based on the quantization step supplied from the variable length encoding unit 31. The dequantized image data is stored in the cache memory 7 via the controller 14.

In the case of an intra macro block, the image data subjected to the inverse DCT transform processing by the inverse DCT transform unit 33 is transferred to the cache memory 7 as it is and stored. In the case of a non-intra macro block, the motion compensation prediction unit 35
Is a motion vector supplied from the variable length decoding unit 31,
Using the reference image stored in the frame memory 5,
A prediction macro block is calculated, and the image data is read from the cache memory 7 and stored. Arithmetic unit 34
Adds the image data (difference data) supplied from the inverse DCT transform unit 33 and the predicted macroblock supplied from the cache memory 7 to obtain decoded image data, which is transferred to the cache memory 7. Then, the image data stored in the cache memory 7 is DMA-transferred to the frame memory 5 under the control of the DMAC 6.

Next, the operation of the image decoding apparatus shown in FIG. 2 will be described. Tuner 1 receives the image data,
The image data is output to the demodulation processing unit 2. The demodulation processing unit 2 that has received the image data demodulates the input image data and outputs the demodulated image data to the error correction processing unit 3. The error correction processing unit 3 performs an error correction process on the input demodulated image data and outputs it to the software processing unit 4. The software processing unit 4 first demultiplexes the input image data into image data and audio data for each channel by the demultiplexer unit 11 and outputs the data to the decoding unit 12.

The decoding unit 12 is controlled by the controller 14, decodes the image data by using the cache memory 7 and the frame memory 5 as needed, and outputs the decoded image data to the image frame conversion processing unit 13. The image frame conversion processing unit 13 performs an image frame conversion process on the decoded image data, and stores the image data in the frame memory 5. Then, the image data stored in the frame memory 5 is output to the image display device or the like (not shown) in which the image data subjected to the image frame conversion processing is output.

The writing of image data to the frame memory 5 of the decoding unit 12 shown in FIG. 3 will be described. Writing of data to the frame memory 5 and reading of data from the frame memory 5 are performed in macroblock units. Since one Y macro block is composed of 16 × 16 pixels, it is composed of 256 bytes of data. Also, the Cb macroblock and the Cr macroblock are 64 (= 8 × 8), respectively.
It is composed of bytes. Therefore, as shown in FIG. 4, the Y macro block located at the top left position on the screen to be read first is read to the addresses 0000 to 0255 of the frame memory 5 and then to the Y macro block. Consecutive addresses are sequentially allocated and stored in ascending order, such as Y macroblocks at addresses 0256 to 0511 and Y macroblocks to be read next at addresses 0512 to 0768.

Similarly, the chroma macroblocks of the Cb macroblock and the Cr macroblock are, as shown in FIG. 5, the chroma macroblock to be read first at addresses 0000 to 0063, and the chroma macroblock to be read next. Blocks have addresses 0064 to 01
As shown at 28, successive addresses are sequentially allocated and stored in ascending order.

By storing image data in units of macroblocks in this way, a page miss can be caused by reading out one macroblock,
Neither of the Cb (Cr) macroblocks occurs at least once. That is, a page miss occurs when a row to be read is switched, and one row corresponds to 256 rows.
byte (hereinafter, appropriately, these 256 bytes are referred to as one page,
Since this unit is referred to as a page break), it occurs at least every reading of 256 bytes. As described above, if data is stored in the frame memory 5, Y A macroblock has a page break every page, and a Cb (Cr) macroblock has a page break every four pages.

In order to minimize the page miss (loss time), it is only necessary to reduce the number of row switching. Therefore, the loss time is reduced by storing the data in the frame memory 5 as described above. Things become possible.

Next, a case where the predicted image data is extracted from the frame memory 5 in which the image data is stored by using the motion compensation vector will be described. Here, a case will be described as an example where a Y macro block is targeted. One Y macroblock, as described above,
It is composed of 16 × 16 pixels (16 × 16 bytes).

FIG. 6 assumes a screen displayed on the screen of the screen display device, and predicts a predicted macro block MA_n (decoded macro block) from a macro block MA_m.
FIG. 4 is a diagram showing a positional relationship of FIG. Of the motion compensation vectors of MA_n, a horizontal (horizontal) motion compensation vector in the drawing is represented as a vector x, and a vertical (vertical) motion compensation vector is represented as a vector y. The vector x and the vector y are used to determine the horizontal and vertical address offsets for extracting data from the upper left corner of the macroblock MA_n.

The macroblock MA_m whose position is uniquely determined by the vector x and the vector y is, as shown in FIG.
It is assumed that the macroblock spans up to four macroblocks. If these four macroblocks are described as macroblock MA_0, macroblock MA_1, macroblock MA_2, and macroblock MA_3 in order from upper left to lower right, the relationship between macroblock MA_n to be decoded and vectors x, y is , As shown in the following equation.

[0030] Here, x and y indicate the magnitudes of the vectors x and y, respectively, and MA_w indicates the number of macroblocks arranged on a horizontal line on the screen.

Further, each macro block MA_0, MA_1, MA
_2 and MA_3, the address (top address) at the upper left corner of the portion overlapping macroblock MA_m is calculated according to the following equation. The head addresses of the macro blocks MA_0, MA_1, MA_2, and MA_3 are assumed to be addresses ad_0, ad_1, ad_2, and ad_3, respectively.

[0032] Here, ad_MA_0, ad_MA_1, ad_MA_2, and ad_MA_3 are the start addresses of the macroblocks MA_0, MA_1, MA_2, and MA_3, respectively.
Similarly, y% 16 indicates the remainder when the magnitude of the vector y is divided by 16.

Each macro block MA_0, MA_1, MA_2, MA_3
The number of horizontal data and the number of vertical lines of the part overlapping with the macro block MA_m are calculated according to the following equations. In the following equation, MA_0_h indicates the number of horizontal data portions where the macro block MA_m and the macro block MA_0 overlap, and MA_0_v is
It indicates the number of vertical lines where the macro block MA_m and the macro block MA_0 overlap. Other descriptions have the same meaning.

MA_0_h = 16− (x% 16), MA_0_v = 16− (y% 16) MA_1_h = 16−MA_0_h, MA_1_v = MA_0_v MA_2_h = MA_0_h, MA_2_v = 16−MA_0_v MA_3_h = MA_1_v = MA_2_v = MA_2_v = (3) Thus, the four macro blocks MA_0, MA_1, MA_2,
When extracting data from MA_3, four page errors occur. Same macroblock (one macroblock)
When data is retrieved from the, no page miss occurs.
That is, a page miss may occur at least 0 times and at most 4 times.

However, in a DRAM or the like, two banks are provided.
This is performed by switching the bank. In the macro block, writing is performed alternately on different banks for each horizontal line. For this reason, when writing is performed using two banks 0 and 1 as shown in FIG.
The data of the horizontal line is read, then the data of one horizontal line of the macro block MA_2 is read from the bank 1, the data of one horizontal line of the macro block MA_1 is read from the bank 0, and then the bank 1 is read. The process of reading data of one horizontal line of the macro block MA_4 from is repeated.

If data is read by switching between bank 0 and bank 1, no page miss will occur. That is, bank 0 and bank 1
Since they have independent sense amplifiers, they can be independently activated (precharged), so that macro blocks stored in different banks can be read continuously. (Even when reading is performed by switching banks), a page miss (loss time) due to the switching can be ignored. Therefore, only the page miss that occurs when the data of the macro block MA_0 is read first becomes the loss time.

In the above description, in the equations (1) to (3), the case of reading out the Y macro block has been described. However, in the case of the chroma (Cb, Cr) macro block, 16 bits in each equation are used. Should be set to 8.

As described above, by writing and reading image data, of the time required to read one Y macro block, the time for data transfer is 8 × 16 clocks, and If the time of one page miss is 6 clocks, the loss time is 6 × 1 clocks, so that the ratio of the loss time due to the page miss to the time for data transfer can be reduced. Similarly, even when one Cb (Cr) macroblock is read, the ratio of the loss time due to a page miss to the time required for data transfer can be reduced.

Next, for an image display device (not shown),
Processing for outputting image data will be described. FIG.
As shown in (B), when image data in units of macroblocks is stored in the same way as the data arrangement on the screen and the data arrangement in the frame memory, the data is read out in ascending address order, and Although the display was performed, as shown in FIG.
In the case of storing data in lines or two lines, an image cannot be displayed by reading data in ascending address order.

In other words, when the image data is stored as shown in FIG. 1B, in order to display one line on the screen, 16 pixels (16 pixels) from each macro block are displayed.
byte) of image data is read out.
Even when image data is stored as shown in FIG. 5, similarly, image data for every 16 bytes must be read from each macroblock. In this case, every time 16 bytes are read, the row must be switched, and a page miss occurs each time. In order to eliminate such inconvenience, address conversion may be performed as described below, and image data may be output to an image display device.

Using the cache memory 7 shown in FIG. 3 as a temporary buffer, one slice of image data is loaded from the frame memory 5 to the cache memory 7, and the memory structure shown in FIG. Address). When an SRAM (Static RAM) is used for the cache memory 7, a page miss does not occur because a page does not exist unlike a DRAM.

More specifically, a case where the address is converted from a state in which the Y macroblock is stored as shown in FIG. 4 to a state in which the Y macroblock is stored as shown in FIG. explain. As shown in FIG. 4, the image data of the addresses 0000 to 0015 is read from the frame memory 7 in which the image data is stored as the image data of the first line, and stored in the addresses 0000 to 0015 of the cache memory 7. Next, addresses 0016 to 0031 are used as image data of the second line.
Are read and stored in the cache memory 7 at addresses 0720 to 0735. Subsequently, addresses 0032 to 00 are used as image data of the third line.
Reads image data up to 47
At addresses 1440 to 1455. Such processing is repeated.

As described above, when the image data is read from the frame memory 7, the reading itself is performed in the ascending order of the address, so that a page miss occurs only every 256 bytes. The image data stored in the cache memory 7 is re-transferred to the frame memory 5 and stored. Then, the image data stored in the frame memory 5 as shown in FIG. 1B is output to an image display device (not shown).

From the cache memory 7 to the frame memory 5
No re-transfer is performed in the order of one-to-one addresses, and no page miss occurs when the cache memory 7 is composed of SRAM, so that no loss time is caused by this processing.

In the above description, the image data is re-transferred from the cache memory 7 to the frame memory 5. However, from the cache memory 7, a display or scaling frame memory different from the frame memory 5 is used. (DRAM).

As described above, by writing and reading the image data of the macro block, the decoded macro blocks can be sequentially stored in the same page of the frame memory composed of the DRAM or the like.
By utilizing the burst transfer function of AM, it is possible to reduce the number of occurrences of page misses and improve the memory bandwidth. Also, since the macroblock is stored in the same page in the frame memory, even when the reference macroblock is extracted by the motion compensation vector,
By utilizing the burst transfer of the DRAM, the number of occurrences of page misses can be reduced and the memory bandwidth can be improved.

When the macroblocks are stored in the frame memory, the data is alternately stored in another bank of the DRAM for each horizontal width of the screen, so that prediction data is extracted from a plurality of (up to four) macroblocks. In addition, since page errors can be minimized, and since the configuration can be made up of two banks, the structure of the memory system can be simplified, and a DRAM with a small (small) capacity can be realized.
We are flexible.

Next, with reference to FIG. 7, a description will be given of a medium used to install a program for executing the above-described series of processing in a computer so that the program can be executed by the computer.

The program is, as shown in FIG.
It can be provided to the user in a state where it is installed in a hard disk 52 or a semiconductor memory 53 as a recording medium incorporated in a personal computer 51 (corresponding to an image decoding device) in advance.

Alternatively, the program is as shown in FIG.
As shown in the figure, a floppy disk 61,
It can be temporarily or permanently stored in a recording medium such as the CD-ROM 62, the MO disk 63, the DVD 64, the magnetic disk 65, and the semiconductor memory 66, and can be provided as package software.

Further, as shown in FIG. 7C, the program is transmitted from a download site 71 to a satellite 72 wirelessly.
To the personal computer 73 via
The data can be transferred to the personal computer 73 in a wired or wireless manner via a network 81 such as a local area network or the Internet, and the personal computer 73 can download the data to a built-in hard disk or the like.

The medium in the present specification means a broad concept including all these media.

[0053]

According to the image processing apparatus of the first aspect, the image processing method of the fourth aspect, and the medium of the fifth aspect, the input image data in units of macroblocks are addressed in ascending order. Is assigned and stored, and the stored image data is read out in ascending order of addresses, so that it is possible to suppress the number of times a page error occurs.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a conventional method of storing image data.

FIG. 2 is a block diagram showing a configuration of a mobile phone according to an embodiment of the image processing apparatus to which the present invention is applied;

FIG. 3 is a block diagram illustrating a detailed configuration of a decoding unit 12 in FIG. 2;

FIG. 4 is a diagram illustrating a method of storing image data according to the present invention.

FIG. 5 is a diagram illustrating a method of storing image data according to the present invention.

FIG. 6 is a diagram illustrating reading of a predicted macroblock.

FIG. 7 is a diagram illustrating a medium.

[Explanation of symbols]

5 frame memory, 6 DMAC, 7 cache memory, 12 decoding unit, 31 variable length decoding unit,
32 inverse quantization unit, 33 inverse DCT transform unit, 34 arithmetic unit, 35 motion compensation prediction unit

Continued on front page F term (reference) 5B047 EA05 EB11 5B060 AC14 CA04 DA09 GA11 5C059 KK15 MA00 MA07 MA23 MC11 ME01 NN01 RF04 SS26 UA33 UA34 UA35 UA36 UA38

Claims (5)

[Claims] An input unit for inputting image data in units of macroblocks; a storage unit for assigning and storing addresses in ascending order to the image data input by the input unit; An image processing apparatus comprising: a reading unit that reads image data in ascending order of addresses. 2. The image processing apparatus according to claim 1, further comprising a conversion storage unit configured to convert the image data read by the reading unit into an address arrangement equivalent to an image displayed based on the image data, and store the converted data. The image processing device according to claim 1. 3. The storage unit includes two or more banks, and switches the bank for each horizontal width of an image displayed based on the image data and stores the image data. The image processing device according to claim 1. 4. An input step of inputting image data in units of macroblocks; a storage step of assigning and storing addresses to the image data input in the input step in ascending order; and storing the image data stored in the storage step. A reading step of reading image data in ascending order of addresses. 5. An input step of inputting image data in units of macroblocks; a storage step of assigning and storing addresses to the image data input in the input step in ascending order; and storing the image data stored in the storage step. A medium for causing a computer to execute a program, comprising: a reading step of reading image data in ascending order of addresses.