JP2006311589A - Moving image decoding method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform fast forwarding reproduction with high encoding efficiency and a higher degree of freedom at a decoding side when decoding a moving image using motion compensated prediction inter-frame encoding. <P>SOLUTION: Identification information indicating a group of a layer to which an encoding target frame included in encoded data is distributed, and a reference frame used for motion compensated prediction inter-frame encoding and side information including information indicating a picture type of the encoding target frame are decoded (S21), a reference frame belonging to a group of layers lower than the layer to which the encoding target frame is distributed, is selected according to the decoded identification information (S22), and the selected reference frame is used to decode a result of motion compensated prediction inter-frame encoding included in the encoded data (S23). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a moving picture decoding method and apparatus using motion compensated prediction interframe coding.

  MPEG1 (ISO / IEC11172-2), MPEG2 (ISO / IEC13818-2), MPEG4 (ISO / IEC14496-2), and the like have been widely put into practical use as moving image compression coding techniques. In these moving image encoding systems, encoding is performed by a combination of intra-frame encoding (intra encoding), forward prediction inter-frame encoding, and bidirectional prediction inter-frame encoding, and encoding is performed in these encoding modes. Each frame is called an I picture, a P picture, and a B picture. The P picture is encoded using the immediately preceding P or I picture as a reference frame, and the B picture is encoded using the immediately preceding and immediately following P or I picture as a reference frame. Forward prediction interframe coding and bidirectional prediction interframe coding are called motion compensated prediction interframe coding.

  When fast-forwarding reproduction of encoded data of a moving picture according to the MPEG system, only an I picture that does not require a reference frame is reproduced, or a B picture is skipped by utilizing the property that a B picture is not used as a reference frame. In general, a method of decoding only the I picture and the P picture is used. However, when reproducing only an I picture, if the period of the I picture is long, high-speed fast-forwarding can be realized, but smooth fast-forwarding reproduction cannot be performed. In fast-forward using I and P pictures, inter-frame predictive coding is used for P pictures, so it is necessary to decode all I and P pictures, and the fast-forward speed can be freely changed. It becomes difficult.

  In addition, in the conventional MPEG video coding, B pictures are not used as reference frames. Therefore, in the case of a prediction structure in which a plurality of B pictures are continuous, P pictures that are separated in time are encoded in the B picture coding. There is a problem that the encoding efficiency of the B picture decreases because it must be a reference frame. On the other hand, when the decoded B picture is used as a reference frame in the P picture, it is necessary to decode all the frames including the B picture even during the fast-forward playback with the B picture skipped. It becomes difficult to perform fast forward playback.

  As described above, when fast-forward playback is performed on moving image encoded data obtained by encoding including motion-compensated prediction inter-frame encoding as in MPEG, smooth playback can be freely performed when only I pictures are played back. In the case of performing fast forward playback skipped without decoding the B picture, it is difficult to use the decoded B picture as a reference frame. In a prediction structure in which the B picture is continuous, There was a problem that the encoding efficiency was lowered.

  An object of the present invention is to enable high-speed playback with high coding efficiency and higher degree of freedom on the decoding side in coding and decoding of moving images using motion compensated prediction interframe coding. is there.

  In order to solve the above-described problem, the present invention provides a moving image in which a coding process including a motion compensated prediction inter-frame coding using at least one decoded frame as a reference frame is performed on a coding target frame of a moving image. In encoding, the inter-frame prediction structure is set to a plurality of hierarchical prediction group structures, and the encoding target frame is allocated to a prediction group of any one of a plurality of prediction groups to which a plurality of reference frames belong, Motion compensation prediction interframe coding is performed using reference frames belonging to a prediction group of at least one layer below the layer to which the encoding target frame is distributed.

  Also, the first identification information indicating the prediction group of the layer to which the encoding target frame is distributed and the second identification information indicating the reference frame used for the motion compensated prediction interframe encoding are set for the encoding target frame. The result is output as encoded data together with the result of the motion compensation prediction interframe encoding.

  On the other hand, in the moving picture decoding in which the encoded data obtained by performing the encoding process including the motion compensated prediction inter-frame encoding on the encoding target frame of the moving picture is decoded to reproduce the moving picture, Decoding the first identification information indicating the prediction group of the layer to which the encoding target frame included in the encoded data is distributed and the second identification information indicating the reference frame used for the motion compensated prediction interframe encoding In accordance with the first identification information and the second identification information, the at least one reference frame belonging to the prediction group of the layer to which the encoding target frame is distributed or the prediction group of the lower layer is selected and selected. The result of the motion compensation prediction interframe coding included in the coded data is decoded using the reference frame.

  In this way, the encoding target frames allocated to the prediction groups of each layer are subjected to motion compensation prediction interframe encoding using reference frames belonging to the prediction groups below the layer, so that an upper layer is used for decoding. It is possible to normally decode without decoding the encoding result of the encoding target frame belonging to the prediction group. Therefore, it is possible to change the number of frames to be decoded by changing the highest layer of the prediction group to be decoded. The frame rate to be reproduced can be made variable according to the layer or the decoded frame can be changed. By changing the display frame rate, fast-forward playback with variable speed can be easily realized. In addition, by selecting and transmitting only encoded data below a specific layer, it is possible to perform streaming with variable bit rate according to the transmission band.

  In this case, the maximum number of reference frames belonging to each of the prediction groups of a plurality of hierarchies is determined separately for each prediction group in advance, and memory management of the reference frames of the prediction groups of each hierarchy is performed according to the maximum reference frame number. You may make it perform.

  In this way, the maximum number of reference frames required for decoding can be uniquely determined from the sum of the maximum number of reference frames in the prediction group below the prediction group of the highest hierarchy to be decoded. Accordingly, the highest hierarchy of a predictable prediction group is uniquely determined in a limited memory resource in moving picture decoding, and the highest hierarchy of the prediction group to be decoded is changed as described above. When changing fast-forward playback or transmission bit rate, the minimum amount of reference memory used for decoding can be uniquely determined, and it is possible to easily secure the minimum memory required for decoding. It becomes.

  Furthermore, the total sum of the maximum number of reference frames belonging to each of the prediction layers of a plurality of hierarchies is made constant, and information indicating the maximum number of frames is used for the first identification information, the second identification information, and the motion compensated prediction interframe coding. It may be output as encoded data together with the result, and memory management of reference frames of prediction groups in each layer may be performed.

  In this way, the reference memory having a fixed total capacity can be dynamically allocated to the prediction groups in each hierarchy, so the number of reference memories in the prediction group in each hierarchy according to the change in the nature of the video signal By dynamically allocating the frame memory so that the ratio is optimal, it is possible to improve the coding efficiency. In addition, since the total number of reference frames is constant, the number of reference frames necessary for encoding and decoding only needs to be secured at all times, and memory management can be facilitated. Furthermore, by encoding information indicating the number of reference frames belonging to the prediction group of each layer as header information, it is possible to match the number of reference frames of the prediction group of each layer on the encoding side and the decoding side, Even if a dynamic change in the number of reference frames in the prediction group of each layer occurs, decoding can be performed without failure.

  Further, as the encoding process, intra-frame encoding, forward prediction inter-frame encoding, and bidirectional prediction inter-frame encoding are switched for each frame of the encoding target frame, and intra-frame encoding and forward prediction inter-frame encoding are performed. An encoding target frame and a reference frame corresponding to the encoding target frame are allocated to a prediction group of the first layer, and an encoding target frame for bi-directional prediction interframe encoding and a reference frame corresponding to the encoding target frame are You may make it distribute to the prediction group of the 2nd hierarchy higher than the 1st hierarchy.

  In this way, as with conventional encoding such as MPEG, it is possible to perform fast-forward playback for decoding only I pictures or only I pictures and P pictures without decoding B pictures. . Further, in addition to the decoded I picture or P picture, in addition to the decoded I picture or P picture, one or more decoded B pictures can be used as a reference frame. It becomes possible to improve.

  Furthermore, according to the present invention, it is possible to provide a program for causing a computer to perform the above-described video encoding and video decoding processes.

  According to the present invention, the inter-frame prediction structure is formed into a plurality of hierarchized group structures, inter-frame prediction from reference frames of a higher-level prediction group is prohibited, and each layer has a fixed total number of reference frames. By dynamically changing the number of reference frames in the prediction group, it is possible to improve the encoding efficiency as compared with the prior art and enable fast-forward playback with a higher degree of freedom.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(About encoding side)
FIG. 1 is a block diagram showing a configuration of a video encoding apparatus according to the present embodiment. FIG. 2 is a flowchart showing a schematic procedure for motion compensated prediction interframe coding. The moving picture encoding apparatus shown in FIG. 1 may be realized by hardware, or may be executed by software using a computer. Some processing may be realized by hardware, and other processing may be performed by software.

  The present embodiment is based on video coding that combines motion compensated prediction, orthogonal transform, and variable length coding, as represented by the conventional MPEG system. In the following description, a case where the prediction group has two layers will be described.

  The moving image signal 100 (encoding target frame) input for each frame is first distributed to a prediction group of any one of the two layers of prediction groups by the motion compensation predictor 111 (step S11). Next, motion compensated prediction interframe coding is performed as at least one reference frame belonging to a prediction group of at least one layer below the layer to which the encoding target frame is distributed (step S12).

  The allocation of the encoding target frame to the prediction group of each layer is performed so as to change in the time direction, for example, an even frame is a first layer prediction group and an odd frame is a second layer prediction group. The reference frames that belong to the prediction group of each layer are also determined according to the prediction group to which the encoding target frame that is the source of the encoded frame that becomes the reference frame belongs. That is, if a certain encoding target frame is allocated to a prediction group in a certain hierarchy, an already encoded frame obtained by encoding and local decoding the encoding target frame also belongs to a prediction group in the same hierarchy. Specifically, the processes of steps S11 to S12 are performed as follows.

  As described above, a plurality of encoded frames belong to the first layer and second layer prediction groups in advance as reference frames. In order to temporarily store the encoded frame as a reference frame, two sets of reference memory sets 118 and 119 are prepared. The first reference memory set 118 stores a plurality of encoded frames belonging to the prediction group of the first hierarchy among a plurality of already encoded and decoded moving image frames (these are referred to as encoded frames). Is temporarily stored as a reference frame. In the second reference memory set 119, among the plurality of encoded frames, a plurality of encoded frames belonging to the second layer prediction group are temporarily stored as reference frames.

  The frame to be encoded distributed to the prediction group of the first layer is subjected to motion compensation prediction interframe encoding using the reference frame belonging to the prediction group of the first layer, which is stored in the first reference memory set 118. Done. On the other hand, the encoding target frame allocated to the second layer prediction group is a reference frame that is stored in the first and second reference memory sets 118 and 119 and belongs to both the first and second layer prediction groups. Is used to perform motion-compensated prediction interframe coding.

  The motion compensated prediction frame encoding will be described in detail. First, when the encoding target frame that is the input moving image signal 100 belongs to the prediction group of the first hierarchy, the first temporarily stored in the first reference memory set 118 is stored. One or more reference frames are read out and input to the motion compensated predictor 111. At this time, the switch 120 is in an OFF state, and the reference frame from the first reference memory set 119 is not input to the motion compensation predictor 111. The motion compensated predictor 111 generates a predicted image signal 104 by performing motion compensation prediction using one or a plurality of reference frames read from the first reference memory set 118. The predicted image signal 104 is input to the subtractor 110, where a predicted error signal 101 that is an error signal of the predicted image signal 104 with respect to the input moving image signal 100 is generated.

  When the encoding target frame that is the input moving image signal 100 belongs to the prediction group of the second hierarchy, the switch 120 is in the on state, and is temporarily stored in the first reference memory set 118 and the second reference memory set 119. One or a plurality of reference frames are read out and input to the motion compensated predictor 111, so that the motion compensated predictor 111 generates the predicted image signal 104 in the same manner as described above, and the subtractor 110 further generates a prediction error. A signal 101 is generated.

  The prediction error signal 101 is subjected to discrete cosine transform by the DCT transformer 112, and the DCT coefficient obtained thereby is quantized by the quantizer 113. The quantized DCT coefficient data 102 is bifurcated and is encoded by the variable length encoder 114 on one side. On the other hand, the quantized and bifurcated DCT coefficient data 102 is reproduced as a prediction error signal through an inverse quantizer 115 and an inverse DCT transformer 116. The reproduced prediction error signal is added to the predicted image signal 104, whereby a locally decoded image signal 103 is generated.

  The encoded frame that is the locally decoded image signal 103 is the first and the first according to the prediction group of the hierarchy allocated to the encoding target frame that is the input moving image signal 100 that is the source of the encoded frame. Temporarily stored in one of the two reference memory sets 118 and 119 (step S13). That is, the encoded frame is temporarily stored in the first reference memory set 118 when the encoding target frame that is the source of the encoded frame belongs to the first layer prediction group, and the encoding target If the frame belongs to the second layer prediction group, it is temporarily stored in the second reference memory set 119.

  From the motion compensation predictor 111, a motion vector used for motion compensation prediction, an index (first identification information) for identifying a prediction group to which an encoding target frame belongs, and motion compensation prediction interframe coding are used. So-called side information 105 including an index (second identification information) specifying the used reference frame is output and encoded by the variable length encoder 114 (step S14). In this case, an index for identifying a prediction group is encoded as, for example, a picture type representing a prediction structure, and an index for specifying a reference frame is encoded for each macroblock.

  The side information is output as encoded data 106 of a variable length code together with quantized DCT coefficient data that is a result of motion compensation prediction interframe coding (step S15). For example, the side information is encoded in the encoded data 106 as header information. Further, when a second reference frame number setting method (to be described later) that predefines the total number of reference frames belonging to the prediction group of each layer and sets the maximum number of reference frames to be allocated to the prediction group of each layer is adopted In addition, information indicating the maximum number of frames is also encoded as header information of the encoded data 106. The encoded data 106 is sent to a storage medium or transmission medium (not shown).

  In the reference memory sets 118 and 119, a so-called FIFO (First-In First-Out) type control in which new decoded frames are sequentially written as reference frames and sequentially deleted from the oldest reference frame in time is used as a frame. Done in units. However, when a reference frame is read, random access to any reference frame in each reference memory set 118, 119 is performed.

  The number of reference frames temporarily stored in the reference memory sets 118 and 119 (in other words, the number of reference memories included in the reference memory sets 118 and 119, respectively) is set by one of the following two methods. .

  In the first reference frame number setting method, the maximum number of reference frames belonging to the prediction group of each layer is individually determined in advance according to the encoding method or encoding specifications such as profile and level. In the moving picture coding apparatus and the moving picture decoding apparatus, the reference frame having the maximum number of frames determined in this way is ensured for each prediction group to perform coding and decoding. In this case, it is possible to automatically secure the necessary number of reference frames by matching the encoding specifications between the video encoding device and the video decoding device.

  In the second reference frame number setting method, the total number of reference frames belonging to the prediction group of each layer is defined in advance according to the encoding method or encoding specifications such as profile and level, Information relating to the distribution of whether to allocate only the number of reference frames, that is, information indicating the maximum number of frames is encoded as header information of the encoded data 106.

  As described above, in the second reference frame number setting method, the encoding side dynamically assigns the optimum maximum number of reference frames to the prediction group of each layer, and encodes information indicating the assigned maximum number of frames. By doing so, it is possible to dynamically match the maximum number of reference frames belonging to the prediction group of each layer on the encoding side and the decoding side. Therefore, it is possible to improve the encoding efficiency by optimally changing the ratio of the maximum number of reference frames belonging to the prediction group of each layer according to the change in the image properties of the input moving image signal 100. .

(About decryption side)
FIG. 3 is a block diagram showing a configuration of a moving picture decoding apparatus corresponding to the moving picture encoding apparatus shown in FIG. 1 according to the present embodiment. FIG. 4 is a flowchart showing a schematic procedure for decoding corresponding to motion compensated prediction interframe coding. The moving picture decoding apparatus shown in FIG. 3 may also be realized by hardware or may be executed by software, or some processes may be realized by hardware and other processes may be performed by software. Good.

  The encoded data 106 output from the moving image encoding apparatus illustrated in FIG. 1 is input to the moving image decoding apparatus illustrated in FIG. 3 via a storage medium or a transmission medium (not illustrated). The input encoded data 200 is subjected to variable length code decoding by a variable length decoder 214, and quantized DCT coefficient data 201 and side information 202 are output. The quantized DCT coefficient data 201 is decoded through an inverse quantizer 215 and an inverse DCT transformer 216, thereby reproducing a prediction error signal.

  On the other hand, a motion vector encoded for each macroblock as the side information 202, an index for identifying a prediction group belonging to each encoding target frame (first identification information), and an index for identifying a reference frame (second Are identified (step 21). According to these pieces of side information, the prediction image signal 203 is generated by selecting a reference frame and performing motion compensation in the same manner as in encoding.

That is, a reference frame is selected according to an index (first identification information) for identifying a prediction group belonging to each encoding target frame and an index (second identification information) for identifying a reference frame (step S22). The result of the motion compensation prediction interframe coding is decoded using the reference frame thus set (step S23). Further, the decoded image signal 204 is generated by adding the prediction image signal 203 and the prediction error signal from the inverse DCT converter 216. The decoded frame, which is the decoded image signal 204, is decoded. Depending on the prediction group to which the encoded frame that is the source of the frame belongs, it is temporarily stored in one of the first and second reference memory sets 219 and used as a reference frame (step S24). These reference memory sets 218 and 219 are subjected to FIFO type control in the same manner as the moving picture coding apparatus. Here, the number of reference frames belonging to the prediction group of each layer is set according to the first or second reference frame number setting method described in the previous video encoding apparatus.

  That is, when the maximum number of reference frames belonging to the prediction group of each layer is determined in advance according to the encoding specification according to the first reference frame number setting method, the reference frame belonging to the prediction group of each layer Is a fixed value for each encoding specification. Further, according to the second reference frame number setting method, the total number of reference frames belonging to the prediction group of each layer is defined in advance according to the coding specification, and the maximum number of reference frames is assigned to the prediction group of each layer. In this case, only the sum of reference frames is fixed, and the number of reference frames belonging to the prediction group of each layer is dynamically controlled based on information indicating the maximum number of reference frames decoded from header information of encoded data. Is done.

  FIG. 5 shows a detailed configuration of the motion compensated predictor 111 used as the motion compensated predictor 111 in the moving picture encoding apparatus shown in FIG. 1 and the motion compensated predictor 211 in the moving picture decoding apparatus shown in FIG. It is shown. As described above, usable reference frames differ depending on the prediction group of the hierarchy to which the frame to be encoded or decoded belongs. Assume that frame memories 302 to 304 in FIG. 5 store reference frames that can be used as reference frames for encoded frames belonging to a prediction group of one layer.

  In this motion compensation predictor, either one of the reference frames that can be used for each macroblock is selected, or a linear sum of the reference frames that can be used by the linear predictor 301 is taken and prediction based on the linear sum is performed. Is selected and motion compensation is performed to generate a prediction macroblock.

  In the moving picture coding apparatus, the reference frame and the motion vector are selected for each macroblock so that the prediction macroblock with the smallest prediction error and the highest coding efficiency is selected. The selected reference frame information and motion vector information are encoded for each macroblock.

  In the moving picture decoding apparatus, a prediction macroblock is generated by a motion compensator in accordance with the received motion vector and reference frame information, and decoding is performed. In the case of performing prediction by linear sum, encoding is performed using information on the linear prediction coefficient as header information of the encoded data, and the linear prediction coefficient is matched between encoding and decoding.

6 and 7 are diagrams showing an example of an inter-frame prediction structure and reference memory control in conventional MPEG moving image coding. The horizontal axis indicates time, and I0, P1, P2, etc. indicate frames in display order. For example, I0 indicates an I picture and the frame number is 0, P1 indicates a P picture and the frame number is 1, and B2 indicates a B picture and the frame number is 2, respectively. The arrows in the figure indicate the prediction structure between frames, and indicate the direction from the reference frame to the encoding target frame. For example, FIG. 6 shows that I0 is a reference frame for the encoding target frame P1. Hereinafter, each example of FIGS. 6 and 7 will be described in detail.
First, FIG. 6 shows a prediction structure composed of only an I picture and a P picture. In MPEG coding, only the I picture or P picture coded immediately before is used as a reference frame for the P picture. FM1 and FM2 in the figure indicate how the reference memory (frame memory) is used in decoding. Here, each frame is decoded over one frame period.

  It is assumed that the decoding of I0 is performed in the period from the I0 frame to the P1 frame in the figure, and the decoded image signal is sequentially written in the reference memory FM1, and then the reference memory FM1 until the decoding of P1 is completed. Saved in. Decoding of P1 is performed using the I0 frame stored in the reference memory FM1 as a reference frame, and the decoded P1 frame is sequentially written in the reference frame FM2, and then stored in FM2 until the decoding of P2 is completed. Is done. P2 is decoded using the P1 frame stored in the reference memory FM2 as a reference frame, and the decoded P2 frame is written while overwriting the I0 frame already stored in the reference memory FM1. As described above, decoding of encoded data composed of an I picture and a P picture is performed using a reference memory for two frames.

  FIG. 7 is an example in the case of including a B picture in the conventional MPEG moving image coding. Since the B picture is also used for prediction from the rear frame, the frames are rearranged in an order different from the display order, and encoded and decoded. The upper part of FIG. 7 shows the frames in the display order and the inter-frame prediction structure, and the lower part shows the frame order in encoding and decoding. FM1 and FM2 indicate the use of the reference memory for decoding.

  When decoding the encoded data having the inter-frame prediction structure shown in FIG. 7, the I0 frame is first decoded and written to the reference memory F1. Subsequently, the P3 frame is decoded using the decoded I0 frame stored in the reference memory FM1 as a reference frame, and the decoded P3 frame is written in the reference memory FM2. Next, the decoded I0 frame stored in the reference memory FM1 is used as a reference frame for forward prediction, and the decoded P3 frame stored in the reference memory FM2 is used as a reference frame for backward prediction. Decoding of B2 is sequentially performed. Next, the P6 frame is decoded using the P3 frame stored in the reference memory FM2 as a reference frame, and the decoded P6 frame is written while overwriting the I0 frame already stored in the reference memory FM1. Since the B picture is not used as a reference frame, the decoded picture of the B picture is not stored in the reference memory, but is sequentially output and displayed.

  When decoding an I picture and a P picture, the previous I picture or P picture stored in the reference memory is output and displayed. For example, I0 stored in the reference memory FM1 is displayed when P3 is decoded, and P3 stored in the reference memory FM2 is displayed when P6 is decoded. Thus, by delaying the display of the I picture and P picture by one cycle, the decoding order is rearranged in the correct display order. As described above, even when a B picture is included, decoding is performed with two reference frames.

  8 to 13 are diagrams illustrating examples of the inter-frame prediction structure and the reference memory control in the present embodiment. Each example will be described below. FIG. 8 is composed of an I picture and a P picture as in FIG. 6, but is an example in which each frame is alternately switched between the prediction group a and the prediction group b. The prediction group b is an upper layer of the prediction group a. To do. Further, it is assumed that the number of reference memories of the prediction group a and the prediction group b is 1 frame.

  As shown by Ia0, Pa2, and Pa4 in the figure, the picture to which the suffix a is added indicates the prediction group a, and the pictures to which the suffix b is added such as Pb1, Pb3, and Pb5 are prediction groups. b. The attributes of these prediction groups are encoded as header information of the encoding target frame as an extension of the picture type or as an independent index. For the encoding target frame belonging to the prediction group a, it is possible to use only the already decoded frame belonging to the prediction frame a as a reference frame, and in the upper layer prediction frame b, the already decoded prediction is used. It is possible to generate a predicted image using a linear sum of one frame or both decoded frames belonging to either group a or prediction group b.

  Since each prediction group in each layer has one frame of reference memory, the reference frame number of the encoding target frame of the prediction group a is 1 frame at maximum, and the reference frame number of the encoding target frame of the prediction group b is 2 at maximum. The frame is usable. For example, the Pa2 frame belonging to the prediction group a uses only the decoded Ia0 frame as the reference frame, but the Pb3 belonging to the prediction group b is assigned to the decoded Pa2 frame belonging to the prediction group a and the prediction group b. Two of the decoded Pb1 frames to which it belongs belong to the reference frame.

  In FIG. 8, FM1, FM2, and FM3 indicate the use of physical reference memory as in FIG. 6 or FIG. In addition, DEC, REFa, and REFb indicate the usage of the logical reference memory. In other words, DEC, REFa, and REFb are frame memories expressed by virtual addresses, and FM1, FM2, and FM3 are the frame memories expressed by physical addresses. In the virtual address expression, DEC is a frame memory for temporarily storing a frame currently being decoded, and REFa and REFb indicate reference memories of the prediction group a and the prediction group b, respectively. Therefore, the frames belonging to the decoded prediction group a are temporarily stored in REFa, and the frames belonging to the decoded prediction group b are temporarily stored in REBb.

  In the example of FIG. 8, for example, it is possible to discard only the frames belonging to the prediction group a by discarding the encoding target frame belonging to the upper layer prediction group b. In this case, the required number of reference memories can be decoded if there are two frames of the frame memory DEC for temporarily storing the currently decoded frame and the reference memory REFa of the prediction group a.

  By decoding only the frames belonging to the prediction group a, decoding with half the frame period can be performed without causing a failure in the prediction structure. For example, it is possible to perform smooth fast-forward playback by playing back decoded frames belonging to the prediction group a at twice the frame rate. In addition, when the bandwidth of the transmission path fluctuates with time in video streaming or the like, normally all encoded data is sent out, and when the effective bandwidth of the transmission path decreases, the encoded data belonging to the prediction group b is Even if it is discarded and only the encoded data belonging to the prediction group a in the lower layer is transmitted, the reception side can reproduce it without failure.

  FIG. 9 is a modified example of the example of FIG. 8, and is a prediction structure in which two frames belonging to the prediction group b are inserted between frames belonging to the prediction group a, and the reference group reference memory of each layer Each number is one frame. Also in this case, it is possible to perform the same decoding as in FIG. 6 by using the frame memory for three frames as in FIG. In the example of FIG. 9, for example, only the frame of the prediction group “a” is decoded, and the encoded frame is reproduced at the original frame rate, whereby smooth 3 × speed reproduction can be performed.

  In FIG. 10, similarly to FIG. 6, the prediction group is composed of an I picture and a P picture, and the prediction group has three hierarchies a, b, and c. The prediction structure is such that one frame of the prediction group b and two frames of the prediction group c are arranged between the frames of a.

  It is assumed that the number of reference frames of the prediction groups a, b, and c in each hierarchy is one frame, and the hierarchy goes up in the order of a, b, and c. That is, the frame belonging to the prediction group a uses only one frame of the decoded prediction group a as a reference frame, and the frame belonging to the prediction group b includes two frames of the decoded prediction group a and prediction group b. Can be used as the reference frame, and the three frames of the decoded prediction groups a, b and c can be used as the reference frame.

  In FIG. 10, DEC, REFa, REFb and REFc are logical frame memories indicating temporarily stored frame memories of decoded frames, reference frames of the prediction group a, reference frames of the prediction group b, and reference frames of the prediction group c. Reuse is indicated, and FM1, FM2, FM3, and FM4 indicate physical reuse of the frame memory for the four frames. For each prediction group in each layer, one frame decoded immediately before is temporarily stored in the reference memories REFa, REFb, and REFc, and the frame currently being decoded is written in the decoded frame memory DEC.

  In the configuration of FIG. 10, since the prediction group is composed of three layers, normal decoding is performed when all the encoded frames below the prediction group c are decoded, and the encoded frames below the prediction group b are decoded. Then, a normal half frame is decoded, and if only a coded frame of the prediction group a is decoded, a normal quarter frame is decoded. In any of the above decoding, a decoded image can be normally generated without causing a failure of the prediction structure. It is possible to realize smooth variable-speed fast-forward playback by dynamically controlling the decoding layer or dynamically changing the sending bit rate by dynamically controlling the sending layer. It becomes.

  In FIG. 11, similar to FIG. 7, the picture is composed of an I picture, a P picture, and a B picture. The I picture and the P picture are set as a prediction group a, and the B picture is set as a prediction group b. The prediction group b is a higher hierarchy than the prediction group a. The number of reference memories in the prediction group a is 2 frames, and the number of reference memories in the prediction group b is 1 frame. In the example of FIG. 11, the number of reference memories of the I picture and P picture of the prediction group a is two frames. Therefore, in the P picture, the I picture or P picture coded or decoded immediately before, and further before that Two frames of an encoded or decoded I picture or P picture can be used as a reference frame. In addition, in the B picture, since the prediction group b has one reference frame, the B picture 1 frame that has been encoded or decoded immediately before is used as a reference frame, and the I of the past two frames that are prediction groups in the lower layers. It is possible to use a reference frame of 3 frames together with a picture and a P picture.

  As in FIGS. 8 to 10, FM1, FM2, FM3, and FM4 indicate physical frame memory usage, and DEC, REFa1, REFa2, and REFb indicate logical frame memory usage. . DEC is a temporarily stored frame memory of a frame being decoded, REFa1 and REFa2 are reference memories for two frames of the prediction group a, and REFb is a reference memory for one frame of the prediction group b.

  Idx0, Idx1, and Idx2 in FIG. 11 indicate indexes for specifying the reference frame of the frame being decoded. For example, in the decoding of the Pa6 frame, the immediately preceding two frames Pa3 and Ia0 belonging to the prediction group a are reference frame candidates, and the reference frame index is assigned sequentially from the one closest in time to the encoding target frame. . The index indicating the reference frame is encoded for each macro block, and the reference frame is selected for each macro block. The macroblock with index 0 generates a predicted image from the immediately preceding I picture or P picture, and the macroblock with index 1 generates a predicted image from the previous I picture or P picture. In addition, when a predicted image is generated by a linear sum of the immediately preceding I picture or P picture and the immediately preceding I picture or P picture, an index for identifying a combination of index 0 and index 1 is a macrobook header. It is encoded as information.

  Further, BWref in FIG. 11 indicates a reference frame for backward prediction in a B picture. In the example of FIG. 11, for example, the backward reference frames of Bb1 and Bb2 are Pa3, and the backward reference frames of Bb4 and Bb5 are Pa6. The reference frame for backward prediction is limited to the I picture or the P picture that has been encoded or decoded immediately before due to the restriction of the frame rearrangement, so that the reference frame is uniquely determined. Therefore, it is not necessary to encode the backward prediction reference frame BWref as header information.

  Further, forward prediction in a B picture can be selected from a maximum of two frames in the example of FIG. For example, in encoding and decoding of Bb4, Pa3 that is the previous frame in time and belonging to the prediction group a and Bb2 that is temporally previous and that belongs to the prediction group b are used as reference frames. This is possible, and an index indicating which reference frame is selected for each macroblock or whether a prediction is performed on the linear sum of both is encoded. Similarly for Bb5, two types Bb4 and Pa3 are used as reference frames.

  For reference frame indexes, numbers are assigned sequentially from the closest to the reference frame that can be used as a reference frame for forward prediction for each encoding target frame. In the example of FIG. 11, in encoding and decoding of a P picture, I or P pictures stored in the reference memory are arranged in time order and numbered. In the encoding and decoding of a B picture, all reference frames stored in the reference memory except for an I picture or a P picture encoded or decoded immediately before being used as a reference frame for backward prediction are sorted in time order. Number them side by side. Idx0 and Idx1 in FIG. 11 are indexes generated according to the above rules.

  FIG. 12 is an extension of the example of FIG. 11 and shows a case where the reference frame number is set to 2 frames and the total frame memory number is 5 frames for the prediction group b, that is, the B picture. FM1 to FM5 indicate the reuse of physical reference frames, DEC is a temporary storage buffer at the time of decoding, REFa1 and REFa2 are prediction groups a, that is, reference memories of I and P pictures, and REFb1 and REFb2 are The logical use of the reference memory of the prediction group b, that is, the B picture is shown. Idx0, Idx1, and Idx2 are reference frame index assignments in forward prediction, and BWref is a reference frame for backward prediction in a B picture. Similar to the example of FIG. 11, the reference frame index in the forward prediction is encoded as header information for each macro clock.

  In the examples of FIGS. 8 to 12, the number of reference memories in the prediction group in each layer is fixed. However, the distribution of the reference memory numbers in the prediction group in each layer is dynamically distributed under a fixed total number of reference frames. It is good also as a structure to change. For example, at the timing shown in FIG. 8, the redistribution is performed by setting the number of reference memories of the prediction group b to 0 and simultaneously the number of reference memories of the prediction group a to 2, and the distribution change is encoded with the header information of the encoded data. The configuration may be such that notification is made from the encryption side to the decoding side. At this time, on the encoding side, prediction from the past two frames of the prediction group a can be used for the frame of the prediction group a, and prediction from the past frames of the prediction group b for the frame of the prediction group b. Is prohibited and the selection of motion compensation prediction is controlled so that prediction from the past two frames of the prediction group a is performed.

  FIG. 13 shows the prediction structure and the reuse of the frame memory when the distribution of the number of reference memories is changed as described above with respect to the example of FIG. In this way, it becomes possible to dynamically set an optimal prediction structure suitable for an input moving image within a limited number of reference frames, and to improve prediction efficiency and perform highly efficient encoding. It becomes possible.

The block diagram which shows the structure of the moving image encoder which concerns on one Embodiment of this invention. The figure which shows the flow of the main processes regarding the motion compensation prediction inter-frame coding in moving image coding. The block diagram which shows the structure of the moving image decoding apparatus which concerns on one Embodiment of this invention. The figure which shows the flow of the main processes regarding decoding of the motion compensation prediction inter-frame encoding result in moving image decoding. FIG. 3 is a block diagram showing a configuration example of a motion compensated predictor used in the video encoding device and the video decoding device according to the embodiment. The figure which shows the example of the prediction structure between frames in the conventional MPEG moving image encoding, and reference memory control The figure which shows the example of the prediction structure between frames in the conventional MPEG moving image encoding, and reference memory control The figure which shows the example of the inter-frame prediction structure and reference memory control which concern on one Embodiment of this invention The figure which shows the example of the inter-frame prediction structure and reference memory control which concern on one Embodiment of this invention The figure which shows the example of the inter-frame prediction structure and reference memory control which concern on one Embodiment of this invention The figure which shows the example of the inter-frame prediction structure and reference memory control which concern on one Embodiment of this invention The figure which shows the example of the inter-frame prediction structure and reference memory control which concern on one Embodiment of this invention The figure which shows the example of the inter-frame prediction structure and reference memory control which concern on one Embodiment of this invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Input moving image signal 101 ... Prediction error signal 102 ... Quantized DCT coefficient data 103 ... Local decoded image signal 104 ... Predicted image signal 105 ... Side information 106 ... Encoded data 111 ... Motion compensation predictor 112 ... DCT converter DESCRIPTION OF SYMBOLS 113 ... Quantizer 114 ... Variable length encoder 115 ... Inverse quantizer 116 ... Inverse DCT device 117 ... Reference frame write control switch 118, 119 ... Reference memory set 120 ... Reference frame read controller 200 ... Encoded data 201 Quantized DCT coefficient data 202 ... Side information 203 ... Predicted image signal 204 ... Decoded image signal 211 ... Motion compensation predictor 214 ... Variable length decoder 215 ... Inverse quantizer 216 ... Inverse DCT converter 217 ... Reference frame Write control switch 218, 219 ... Reference memory set 22 ... reference frame read control switch 300 ... prediction macroblock selector 301 ... linear predictor 302,303,204 ... reference memory

Claims (3)

A moving picture decoding method for reproducing a moving picture by decoding coded data obtained by performing a coding process including motion compensated prediction interframe coding on a coding target frame of a moving picture,
First identification information indicating a group of layers to which the encoding target frame included in the encoded data is distributed, second identification information indicating a reference frame used for the motion compensation prediction interframe encoding, and Decoding information indicating a picture type of the encoding target frame;
In accordance with the decoded first identification information and second identification information, select at least one reference frame belonging to a group of layers below the layer to which the encoding target frame is distributed,
A moving picture decoding method for decoding a result of the motion compensated prediction interframe coding included in the coded data using a selected reference frame. A moving image decoding apparatus that decodes encoded data obtained by performing an encoding process including motion compensated prediction inter-frame encoding on a moving image encoding target frame and reproduces a moving image,
First identification information indicating a group of layers to which the encoding target frame included in the encoded data is distributed, second identification information indicating a reference frame used for the motion compensation prediction interframe encoding, and Means for decoding information indicating a picture type of the encoding target frame;
Means for selecting at least one reference frame belonging to a group of layers below the layer to which the encoding target frame is distributed according to the decoded first identification information and second identification information;
A moving picture decoding apparatus comprising: means for decoding a result of the motion compensated prediction interframe coding included in the coded data using a selected reference frame. A program for causing a computer to perform a process of reproducing a moving image by decoding encoded data obtained by performing an encoding process including motion compensated prediction inter-frame encoding on a moving image encoding target frame. There,
First identification information indicating a group of layers to which the encoding target frame included in the encoded data is distributed, second identification information indicating a reference frame used for the motion compensation prediction interframe encoding, and A process of decoding information indicating a picture type of the encoding target frame;
A process of selecting at least one reference frame belonging to a group in a layer to which the encoding target frame is distributed or a group in a lower layer according to the decoded first identification information and second identification information; ,
A program for causing the computer to perform a moving picture decoding process including a process of decoding a result of the motion compensated prediction interframe encoding included in the encoded data using a selected reference frame.

Images