Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/577—Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/103—Selection of coding mode or of prediction mode
  • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/157—Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
  • H04N19/159—Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/187—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding

Definitions

• Methods and apparatuses consistent with the present invention relate to encoding a multiview video sequence, and more particularly, to encoding a multiview video photographed by a multiview camera using a minimum amount of information regarding the multiview video.
• Realism is an important factor in realizing high-quality information and telecommunication services. This realism can be achieved with video communication based on three-dimensional (3D) images. 3D imaging systems have many potential applications in education, entertainment, medical surgery, videoconferencing, and the like. To provide many viewers with more vivid and accurate information of a remote scene, three or more cameras are placed at slightly different viewpoints to produce a multiview sequence.
• the MVP defines the usage of a temporal scalability mode for multi-camera sequences and acquisition camera parameters in an MPEG-2 syntax.
• a base-layer stream which represents a multiview video signal can be encoded at a reduced frame rate, and an enhancement-layer stream, which can be used to insert additional frames in between, can be defined to allow reproduction at a full frame rate when both streams are available.
• a very efficient way to encode the enhancement layer is to determine the optimal method of performing motion-compensated estimation on each macroblock in an enhancement layer frame based on either a base layer frame or a recently reconstructed enhancement layer frame.
• FIG. 1 is a block diagram of a conventional encoder and decoder of the MPEG-2
• the scalability provided by the MPEG-2 is used to simultaneously decode images having different resolutions or formats with an image-processing device.
• temporal scalability is used to improve visual quality by increasing a frame rate.
• the MVP is applied to stereo sequences in consideration of temporal scalability.
• the encoder and decoder illustrated in FIG. 1 are a stereo video encoder and decoder with temporal scalability. Left images in a stereo video are input to a base view encoder, and right images are input to a temporal auxiliary view encoder.
• the temporal auxiliary view encoder provides temporal scalability, and is an interlayer encoder interleaving images between images of the base layer.
• a two-dimensional (2D) video can be obtained.
• a stereoscopic video can be obtained.
• a system multiplexer and a system demultiplexer are needed to combine or separate sequences of the two images.
• FIG. 2 is a block diagram of a conventional stereo- video encoder and decoder using the MPEG-2 MVP.
• An image of the base layer is encoded through motion compensation and discrete cosine transform (DCT).
• the encoded image is decoded in a reverse process.
• a temporal auxiliary view encoder functions as a temporal interlayer encoder which performs prediction based on the decoded image of the base layer.
• disparity compensated estimation may be performed twice, or disparity estimation and motion compensated estimation may each be performed once.
• the temporal auxiliary view encoder includes a disparity and motion compensated DCT encoder and decoder.
• a disparity compensated encoding process requires a disparity estimator and a compensator as a motion estimation/compensation encoding process requires a motion estimator and compensator.
• the encoding process includes performing DCT on a difference between an estimated image and an original image, quantization of DCT coefficients, and variable length encoding.
• a decoding process includes variable length decoding, inverse quantization and inverse DCT.
• MPEG-2 encoding is a very effective compression method because bi-directional motion estimation is performed for bi-directionally motion-compensated pictures (B pictures). Since the MPEG-2 encoding provides highly effective temporal scalability, B pictures can be used to encode a right image sequence. Consequently, a highly compressed right sequence can be generated.
• FIG. 3 illustrates disparity-based predictive encoding in which disparity estimation is used twice for bi-directional motion estimation.
• a left image is encoded using a non-scalable MPEG-2 encoder, and a right image is encoded using a MPEG-2 temporal auxiliary view encoder based on the decoded left image.
• a right image is predicted using two reference images, e.g., two left images, and encoded into a B picture.
• one of the two reference images is an isochronal left image to be simultaneously displayed with the right image, and the other is a left image that follows the isochronal left image.
• the two predictions have three prediction modes: a forward mode, a backward mode and an interpolated mode.
• the forward mode denotes disparity estimation based on the isochronal left image
• the backward mode denotes disparity estimation based on the left image that immediately follows the isochronal left image.
• a right image is predicted using disparity vectors of the two left images.
• Such an estimation method is called predictive encoding, considering only disparity vectors. Therefore, an encoder estimates two disparity vectors for each frame of a right image, and a decoder decodes the right image from the left image using the two disparity vectors.
• FIG. 4 illustrates predictive encoding using a disparity vector and a motion vector for the bi-directional estimation.
• B pictures obtained through the bi-directional estimation of FIG. 3 are used.
• disparity estimation and motion estimation are each used once in the bi-directional estimation. That is, the disparity estimation using an isochronal left image and the motion estimation using a previous right image are used.
• the bi-directional estimation also includes three estimation modes, i.e., a forward mode, a backward mode and an interpolated mode, as in the disparity-based predictive encoding of FlG. 3.
• the forward mode denotes motion estimation based on a decoded right image
• the backward mode denotes disparity estimation based on a decoded left image.
• the MPEG-2 MVP does not consider a multiview video encoder, it is not suitable for encoding a multiview video. Therefore, a multiview video encoder for simultaneously providing a multiview video, which is stereoscopic and realistic, to many people is required.
• the present invention provides a method and apparatus for efficiently encoding a multiview video which is realistic and simultaneously providing the encoded multiview video to many people.
• the present invention also provides a method and apparatus for encoding a multiview video using a prediction structure that uses a minimum amount of information regarding the multiview video.
• the present invention provides a method and apparatus for efficiently encoding a multiview video to simultaneously provide the multiview video which is realistic to many people.
• the present invention also provides a method and apparatus for encoding a multiview video using a B-frame prediction structure that uses a minimum amount of information regarding the multiview video.
• FIG. 1 is a block diagram of a related art encoder and decoder of a motion picture experts group 2 (MPEG-2) multiview profile (MVP);
• FIG. 2 is a block diagram of a related art stereo- video encoder and decoder using the MPEG-2 MVP;
• FIG. 3 illustrates a related art disparity-based predictive encoding in which disparity estimation is used twice for bi-directional motion estimation;
• FlG. 4 illustrates a related art predictive encoding using a disparity vector and a motion vector for the bi-directional estimation;
• FIG. 5 is a block diagram of an apparatus for encoding a multiview video according to an exemplary embodiment of the present invention;
• FlG. 6 illustrates a unit encoding structure of a multiview video according to an exemplary embodiment of the present invention;
• FIGS. 7 A through 7F illustrate three types of B pictures used in multiview video encoding according to an exemplary embodiment of the present invention
• FlG. 8 illustrates a horizontally extended unit encoding structure of a multiview video according to an exemplary embodiment of the present invention
• FlG. 9 illustrates a prediction sequence of the multiview image of FlG. 8;
• FlG. 10 illustrates a video encoding structure having an odd number of views for motion estimation and disparity estimation according to an exemplary embodiment of the present invention
• FlG. 11 illustrates a video encoding structure having an even number of views for motion estimation and disparity estimation according to an exemplary embodiment of the present invention.
• FlG. 12 is a flowchart illustrating a method of encoding a multiview video according to an exemplary embodiment of the present invention.
• a method of encoding a multiview video including: categorizing a plurality of B frames into at least two groups according to a predetermined standard; and sequentially encoding the categorized B frames.
• the predetermined standard may be the number of frames to which each B frame refers.
• the predetermined standard may be the number of reference frames to which each B frame refers and positions of the reference frames.
• the B frames may be categorized into a first group of B frames which are predicted with reference to two horizontally adjacent frames, two vertically adjacent frames or one horizontally adjacent frame and one vertically adjacent frame, a second group of B frames which are predicted with reference to two horizontally adjacent frames and one vertically adjacent frame or one horizontally adjacent frame and two vertically adjacent frames, and a third group of B frames which are predicted with reference to two horizontally adjacent frames and two vertically adjacent frames, wherein the one or two horizontally adjacent frames are a frame or frames obtained from the multiview video at a same temporal level as a referring B frame, and the one or two vertically adjacent frames are a frame or frames obtained from the multiview video at a same view position as a referring B frame .
• the sequential encoding of the categorized B frames may include sequentially encoding the first group of B frames, the second group of B frames, and the third group of B frames.
• the sequential encoding may be performed based on a video encoding structure which includes the B frames, and may further include performing disparity estimation between frames disposed horizontally according to a plurality of views and performing motion estimation between frames disposed vertically according to the passage of time, and the video encoding structure can be horizontally and vertically extended.
• a video encoding structure having n views can be configured into a video encoding structure having n-1 views by disabling an (n-l)th column of frames, wherein n is an odd natural number.
• an apparatus for encoding a multiview video including: a prediction unit which predicts a disparity vector and a motion vector of an input multiview video; a disparity and motion compensation unit which compensates an image using the predicted disparity vector and motion vector; a residual image encoding unit which receives an original image and the compensated image generated by the disparity and motion compensation unit, subtracts the compensated image from the original image, and encodes a residual image obtained from the subtraction; and an entropy-encoding unit which generates a bit stream for the multiview video using the disparity vector, the motion vector, and the encoded residual image, wherein the prediction unit categorizes a plurality of B frames into at least two groups according to a predetermined standard and sequentially predicts the categorized B frames.
• a computer- readable recording medium on which a program for executing a program for implementing the method is recorded.
• FlG. 5 is a block diagram of an apparatus for encoding a multiview video according to an exemplary embodiment of the present invention.
• the apparatus includes a multiview image buffer 510, a prediction unit 520, a disparity/motion compensation unit 530, a residual image encoding unit 540, and an entropy-encoding unit 550.
• the apparatus can receive a multiview video source from a plurality of camera systems or through another method.
• the received multiview video is stored in the multiview image buffer 510.
• the multiview image buffer 510 provides the multiview video to the prediction unit 520 and the residual image encoding unit 540.
• the prediction unit 520 includes a disparity estimation unit 522 and a motion estimation unit 524.
• the prediction unit 520 performs motion estimation and disparity estimation on the multiview video.
• the prediction unit 520 estimates a disparity vector and a motion vector in directions indicated by arrows illustrated in FIGS. 6 through 11, and provides the estimated disparity vector and the motion vector to the disparity/ motion compensation unit 530.
• the prediction unit 520 may set directions for performing motion estimation and disparity estimation by efficiently using a multiview disparity vector and a motion vector which are generated when the multiview video source is extended based on a time axis.
• an MPEG-2 encoding structure can be extended based on a view axis to use spatial/temporal correlation of the multiview video.
• the disparity/motion compensation unit 530 performs the disparity estimation and the motion estimation using the motion vector and the disparity vector estimated by the disparity estimation unit 522 and the motion estimation unit 524.
• the disparity/motion compensation unit 530 reconstructs an image using the estimated motion vector and disparity vector and provides the reconstructed image to the residual image encoding unit 540.
• the residual image encoding unit [55] To provide better visual quality and stereoscopy, the residual image encoding unit
• the entropy-encoding unit 540 encodes a residual image obtained by subtracting the image compensated and reconstructed by the disparity/motion compensation unit 530 from the original image provided by the multiview image buffer 510 and provides the encoded residual image to the entropy-encoding unit 550.
• the entropy-encoding unit 550 receives the estimated disparity vector and motion vector from the prediction unit 520 and the encoded residual image from the residual image encoding unit 540 and generates a bit stream for the multiview video source.
• FlG. 6 illustrates a unit encoding structure of a multiview video according to an exemplary embodiment of the present invention.
• a core-prediction structure or a unit- prediction structure illustrated in FlG. 6 is based on the assumption that there are three views.
• a square block indicates an image frame in a multiview video.
• a horizontal arrow indicates a sequence of frames according to views or positions of cameras, and a vertical arrow indicates a sequence of the frames according to time.
• An I picture indicates an 'intra picture', identical to an I frame in MPEG-2/4 or H. 264.
• P and B pictures respectively indicate a 'predictive picture' and a 'bi-directional prediction picture', similar to P and B frames in MPEG-2.4 or H. 264.
• the P and B pictures are estimated by the motion estimation and the disparity estimation together in the multiview video coding.
• arrows between picture- frames indicate prediction directions.
• Horizontal arrows indicate disparity estimation, and vertical arrows indicate motion estimation.
• FIGS. 7 A through 7F illustrate three types of B pictures used in multiview video encoding according to an exemplary embodiment of the present invention.
• B pictures there are three types of B pictures: B, Bl, and B2 pictures.
• the B, Bl, and B2 pictures denote picture-frames predicted using two or more horizontally or vertically adjacent frames.
• FlG. 7 A two vertically adjacent frames as illustrated in FlG. 7B, or a horizontally adjacent frame and a vertically adjacent frame as illustrated in FlG. 7C.
• B 1 pictures are predicted using two horizontally adjacent frames and one vertically adjacent frame as illustrated in FlG. 7D or a horizontally adjacent frame and two vertically adjacent frames as illustrated in FlG. 7E.
• B 2 pictures are predicted using four horizontally or vertically adjacent frames as illustrated in FlG. 7F.
• an I frame 601 is intra-predicted.
• a P frame 603 is predicted by referring to the I frame 601, and a P frame 610 is predicted by referring to the I frame 601.
• a B frame 602 is predicted using the I frame 601 and the P frame 603 horizontally adjacent to the B frame 602.
• a B frame 604 and a B frame 607 are predicted using the I frame 601 and the P frame 610 vertically adjacent to the B frame 604 and the B frame 607.
• a B frame 612 is predicted using the P frame 610 horizontally adjacent to the B frame 612 and the P frame 603 vertically adjacent to the B frame 612.
• B 1 frames are predicted. Specifically, a B 1 frame 606 is predicted using the
• a Bl frame 609 is predicted using the B frame 607 horizontally adjacent to the Bl frame 609 and the P frame 603 and the B frame 612 vertically adjacent to the Bl frame 609.
• a Bl frame 611 is predicted using the P frame 610 and the B frame 612 horizontally adjacent to the B 1 frame 611 and the B frame 602 vertically adjacent to the B 1 frame 611.
• B2 frames are predicted. Specifically, a B2 frame 605 is predicted using the B frame 604 and the Bl frame 606 horizontally adjacent to the B 2 frame 605 and the B frame 602 and the Bl frame 611 vertically adjacent to the B2 frame 605. In addition, a B2 frame 608 is predicted using the B frame 607 and the Bl frame 609 hor- izontally adjacent to the B2 frame 608 and the B frame 602 and the B 1 frame 611 vertically adjacent to the B2 frame 608.
• bi-directional prediction is performed with reference not only to B frames, but also to Bl and B2 frames. Since the number of B type frames can be increased, the amount of information required for encoding a multiview image can be minimized. Therefore, according to an exemplary embodiment of the present invention, to efficiently encode a multiview image, B frames are grouped according to the types of frame illustrated in FIGS. 7 A through 7F and encoded in the prediction sequence B frame -> B 1 frame -> B 2 frame as described above.
• FlG. 8 illustrates a horizontally extended unit encoding structure of a multiview video according to an exemplary embodiment of the present invention.
• FlG. 8 illustrates a prediction block structure which has a 5-view of an input image source.
• FlG. 9 illustrates a prediction sequence of the multiview image of FlG. 8. In FlG.
• frames in the same column are predicted at the same time.
• an I frame 801 is intra-predicted.
• a P frame 803 and a P frame 816 in a second column are predicted, and B frames 802, 806, 811 and 818 and a P frame 805 in a third column are predicted.
• Bl frames 817, 808 and 813, and B frames 804 and 820 are predicted.
• B2 frames 807 and 821 and Bl frames 810, 819 and 815 in a fifth column are then predicted.
• B2 frames 809 and 814 are predicted. Therefore, the prediction sequence according to the present exemplary embodiment is I, P, B, Bl, B2, P, B, Bl and B2 pictures in order.
• FlG. 10 illustrates a video encoding structure having an odd number of views for motion estimation and disparity estimation according to an exemplary embodiment of the present invention.
• FlG. 11 illustrates a video encoding structure having an even number of views for motion estimation and disparity estimation according to an exemplary embodiment of the present invention.
• the video encoding structure of FlG. 11 can be obtained by disabling a fourth column of prediction frames in the five- view video encoding structure of FlG. 10.
• the video encoding structure according to the present exemplary embodiment can be horizontally and vertically extended.
• an n- view (n is an odd number) video encoding structure can be reconfigured into an (n- 1)- view video encoding structure by disabling an (n-1) column of prediction frames.
• FlG. 12 is a flowchart illustrating a method of encoding a multiview video according to an exemplary embodiment of the present invention. The method has been described with reference to FIGS. 6 through 11. In particular, B frames are encoded in the method as follows.
• a plurality of B frames are divided into at least two groups according to a predetermined standard (S 1210).
• the predetermined standard may be the number of frames that each B frame refers to or may be the number of frames that each B frame refers to and the position of the reference frames.
• the B frames may be categorized into a first group of B frames which are predicted with reference to two horizontally adjacent frames, two vertically adjacent frames or one horizontally adjacent frame and one vertically adjacent frame, a second group of B frames which are predicted with reference to two horizontally adjacent frames and one vertically adjacent frame or one horizontally adjacent frame and two vertically adjacent frames, and a third group of B frames which are predicted with reference to two horizontally adjacent frames and two vertically adjacent frames.
• the B frames grouped as described above are sequentially encoded (S 1220).
• the B frames may be encoded in the order of the first group, the second group, and the third group.
• the present invention provides a method and apparatus for efficiently encoding a multiview video to simultaneously provide the multiview video which is realistic to many people.
• the present invention also provides a method and apparatus for encoding a multiview video using a B-frame prediction structure that uses a minimum amount of information regarding the multiview video.
• the present invention can also be implemented as computer-readable code on a computer-readable recording medium.
• the computer-readable recording medium is any data storage device that can store data which can be thereafter read by a computer system. Examples of the computer-readable recording medium include read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, and carrier waves (such as data transmission through the Internet).
• the computer-readable recording medium can also be distributed over network- coupled computer systems so that the computer-readable code is stored and executed in a distributed fashion.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Compression, Expansion, Code Conversion, And Decoders (AREA)

Applications Claiming Priority (3)

Publications (2)

Family

ID=37771787

Family Applications (1)

Country Status (5)

Families Citing this family (14)

Family Cites Families (6)

• 2006
  • 2006-08-19 CN CN200680030315.4A patent/CN101243692B/zh not_active Expired - Fee Related
  • 2006-08-19 MX MX2008002391A patent/MX2008002391A/es active IP Right Grant
  • 2006-08-19 WO PCT/KR2006/003268 patent/WO2007024072A1/en active Application Filing
  • 2006-08-19 EP EP06783669A patent/EP1917814A4/de not_active Withdrawn
  • 2006-08-19 JP JP2008527838A patent/JP2009505604A/ja active Pending

Also Published As

Similar Documents

Legal Events