JP2005236719A - Device and method for multiplexing code - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve transmission efficiency at the time of transmitting a multiplexed stream and to efficiently utilize the resources of a reproducing device when encoded data such as sound and images are multiplexed. <P>SOLUTION: When the reproducing device side generates a multiplexed stream for synchronizing and reproducing each encoded data with respect to a plurality of pieces of encoded data, the block size of each encoded data is specified through a decoder buffer size inputting part 109 and the maximum number of frames that does not exceed the block size is read from one of the plurality of pieces of encoded data, and when a frame having simultaneousness with the one of the encoded data is read from the other of the plurality of pieces of encoded data, a frame is read within a range where the other encoded data does not exceed the block size. When the frames having simultaneousness are multiplexed, it is possible to read a frame whose time stamp value is smaller than the terminating time of the last frame of the one of the encoded data. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a code multiplexing apparatus and a multiplexing method, and more particularly to an apparatus and method for efficiently multiplexing code information such as voice and images.

  In recent years, streaming distribution services using the Internet have become widespread. Streaming refers to sequentially playing video and music recorded on a server while downloading them. By using this, it is possible to play without waiting for completion of the download.

  As an important technology in the streaming service, there is a multiplexing technology. When multiplexing, as shown in FIG. 2, a multiplexed stream in which a plurality of media data to be played back at the same time is generated so that the playback device can play back multimedia data such as moving images and sounds in synchronization. There is a need to. Usually, the multimedia data is compressed and encoded. If the encoded media data is a fixed bit rate, multiplexing is relatively easy. However, in MPEG, which is a widely used encoding method, the bit rate is variable, so multiplexing is not possible. It cannot be done easily.

  As one of the techniques for solving this, in the code recording device and the code multiplexing method described in Patent Document 1, a time stamp indicating the reproduction time of video data is compared with a time stamp indicating the reproduction time of audio data, Multiplexing so that each data has the same time property supports a variable bit rate.

JP-A-10-320914

  However, in the conventional code multiplexing device described in Patent Document 1, since encoded data is divided as much as possible and multiplexed, data other than encoded data such as packet headers increases, When transmitting a multiplexed stream that can reproduce media (voice or image) encoded data synchronously, there has been a problem that transmission efficiency decreases. This is a particular problem in low-bandwidth communication terminals such as mobile phones because it is important to improve the transmission efficiency as much as possible.

  Further, as described above, since the encoded data is divided and multiplexed as much as possible, if the capacity of the decoder input buffer, which is a temporary storage area arranged in the preceding stage of the decoder, is too large, the apparatus is wasted. There was a problem of becoming.

  Therefore, the present invention has been made in view of the above problems in the conventional code multiplexing device and the like, and in multiplexing encoded data such as voice and image, the transmission efficiency at the time of multiplexed stream transmission is improved. It is an object of the present invention to provide a code multiplexing apparatus and a multiplexing method that can improve and utilize the resources of a reproducing apparatus without waste.

  In order to achieve the above object, the present invention provides a code multiplexing device, and in generating a multiplexed stream for reproducing a plurality of encoded data in synchronization with each encoded data on the reproducing device side, A decoder buffer size input unit for specifying a block size of encoded data is provided.

  Then, according to the present invention, when generating a multiplexed stream in which encoded data such as audio and images can be reproduced in synchronization with each encoded data on the playback device side, the block size of each encoded data is specified. Since the decoder buffer size input unit can be provided, overhead other than the encoded data to be multiplexed can be reduced, transmission efficiency can be improved, and the input buffer size on the playback device side is considered. Therefore, it is possible to generate a multiplexed stream that is optimized so as to conform to the decoder buffer model, and it is possible to use the resources of the playback device without waste. The plurality of encoded data may include audio data and image data.

  Further, the present invention is a code multiplexing method, wherein a block of each encoded data is generated when generating a multiplexed stream for reproducing a plurality of encoded data in synchronization with each encoded data on the reproducing apparatus side. Specify the size, read the maximum number of frames that does not exceed the block size from one of the plurality of encoded data, and have the same time property as the one encoded data from the other of the plurality of encoded data In reading a frame, the frame is read in a range in which the other encoded data does not exceed the block size. As a result, as described above, overhead other than the encoded data to be multiplexed can be reduced to improve transmission efficiency, and resources of the playback device can be utilized without waste.

  In the code multiplexing method, when reading a frame having the same time property as the one encoded data from the other of the plurality of encoded data, the other encoded data does not exceed the block size, A frame having a time stamp value smaller than the end time of the last frame of the one encoded data can be read. The plurality of encoded data may be data including audio data and image data.

  According to the present invention, it is possible to improve the transmission efficiency at the time of multiplexed stream transmission and multiplex the resources of the playback apparatus without waste when multiplexing encoded data such as audio and images.

  FIG. 1 shows an embodiment of a code multiplexing apparatus according to the present invention. This code multiplexing apparatus includes an audio encoding file 101 storing audio code data and a video code storing video code data. Multiplex file 102, application 103, multiplex processing section 104 that multiplexes each encoded data, and multiplex file 105 that stores the multiplexed stream. The multiplex processing section 104 includes an audio output buffer 106. A video output buffer 107 and a system information buffer 108, and each buffer is used as a temporary storage area when a multiplexed stream is output. The application 103 includes a decoder buffer size input unit 109.

  Next, the operation of the code multiplexing apparatus having the above configuration will be described.

  FIG. 3 shows a schematic configuration of an MPEG-4 file format (hereinafter referred to as “MP4”) which is a format for multiplexing a plurality of media data according to the present invention. MP4 is a file format standardized by MPEG-4 (ISO / IEC 14496), and is a general-purpose format for storing encoded data such as MPEG-4. The MP4 structure is expressed by a set of basic data units called hierarchical boxes (or atoms). The encoded data of audio and image is stored in a unit called “chunk” in the “media data” box 301.

  A “track” box 302/303 is prepared for storing information relating to media data, and is required for each type of media. A “track” box 302/303 includes a “chunk offset” box 304 that indicates where the corresponding medium exists in the file, a “sample to chunk” box 305 that indicates how many samples are included in each chunk, and each sample. “Sample size” box 306 indicating the size of each sample, “time to sample” box 307 indicating the time length of each sample, and the like.

  In the present invention, multiplexing is performed in consideration of the decoding input buffer size on the playback apparatus side. Therefore, the decode input buffer size is designated in advance in the application 103. The application 103 reads the maximum number of frames that do not exceed the decoding input buffer size from the audio encoding file 101 (or the video encoding file 102). Subsequently, all frames having the same time property are read from the other encoded file. At this time, if the decoding input buffer size is not exceeded, the previously read encoded data is multiplexed together, but if the decoding input buffer size is exceeded, the maximum number of frames and Of the previously read encoded data, only frames having the same time property are multiplexed. The actual multiplexing process is performed by the multiplexing processing unit 104, and all samples included in each output buffer are added to the multiplexed file 105 so as to become one chunk.

  Next, a specific example of the operation of the code multiplexing apparatus according to the present invention will be described.

  Assume that the video sequence 401 shown in FIG. 4 is stored in the video encoded file 102 (see FIG. 1), and the audio sequence 402 is stored in the audio encoded file 101, respectively. Here, a stream of about 1 second is shown. Each frame period (time length) included in the audio sequence 402 is fixed, but may be variable.

  The decode input buffer size is designated using a user interface as shown in FIG. 9 and processed and stored in the decoder buffer size input unit 109. Here, it is assumed that the decode input buffer size is video input buffer = 3000 bytes and audio input buffer = 1500 bytes.

  Next, a processing flow by the code multiplexing apparatus according to the present invention will be described with reference to FIGS. 1, 4 and 7.

First, the maximum number of frames that do not exceed the video input buffer size, that is, video frame 1 to video frame 6 are read from the video encoded file 102 (701), and at the same time, the size and time stamp of each frame are stored (702 / 703). Subsequently, audio frames are read frame by frame from the audio encoding file 101 (704), but reading is continued while the following two conditions are satisfied.
(A) The time stamp (TS) value is smaller than the end time of the last video frame read earlier (here, video frame 6) (705).
(B) The sum of the read audio frame sizes is smaller than the audio decoding input buffer size (706).

  Here, since the time stamp (384 ms) of the audio frame 7 becomes larger than the end time (370 ms) of the video frame 6 (the condition (a) is not satisfied), the reading is stopped.

  Subsequently, the video frames 1 to 6 and the audio frames 1 to 6 are written into the video output buffer 107 and the audio output buffer 106 (707/708), and a multiplexing instruction is given to the multiplexing processing unit 104. Upon receiving the multiplexing instruction, the multiplexing processing unit 104 similarly converts the data accumulated in the audio output buffer 106 into one chunk so that the data accumulated in the video output buffer 107 becomes one chunk. (709/710). Since it is a “track” box at this time, the header information is held in the system information buffer 108 (711), and after all the encoded data is multiplexed and output to a file, the file is output to generate a final multiplexed file. (712).

  Subsequently, the process returns to the encoded data reading process. The maximum number of frames that do not exceed the video input buffer size, that is, video frame 7 to video frame 13 are read from the video encoded file 102. Subsequently, audio frames are read frame by frame from the audio encoding file 101 while satisfying the above-described conditions. Here, since the total size (1650 bytes) of the audio frames 7 to 13 exceeds the audio input buffer size (the condition (b) is not satisfied), reading is stopped. However, here, since the end time (900 ms) of the video frame 13 is larger than the end time (768 ms) of the audio frame 12, the video frames 7 to 12 and the audio frames 7 to 12 are written to the output buffer, and the multiplexing processing unit A multiplexing instruction is issued to 104, and the video frame 13 is held in the application 103. Upon receiving the multiplexing instruction, the multiplexing processing unit 104 performs the same multiplexing process as described above.

  The video frames 13 to 15 and the audio frames 13 to 16 are read from the respective encoded files in the same procedure as described above, and the multiplexing processing unit 104 performs multiplexing processing. In this way, the multiplexed stream shown in FIG. 5 is obtained.

  On the other hand, when the video sequence and the audio sequence shown in FIG. 4 are multiplexed by applying the conventional technique, the multiplexed stream shown in FIG. 6 is obtained.

The process of generating the multiplexed stream of FIG. 6 by applying the prior art will be supplementarily described.
First, video frame 1 is output to the multiplexed stream. Subsequently, an audio frame having the same time property as the video frame 1 is output to the multiplexed stream. The frame having the same time property refers to a frame in which the time stamp (start time) of the other encoded data is included in one frame period (time length) of one encoded data. Audio frames having the same time as video frame 1 (0 to 80 ms) are audio frame 1 (0 ms) and audio frame 2 (64 ms). Subsequently, the video frame 2 (80 ms) and the video frame 3 (120 ms) having the same time as the audio frame 2 (64 ms to 128 ms) are output to the multiplexed stream. By repeating similar determination processing and multiplexing processing, the multiplexed stream shown in FIG. 6 is generated.

  Here, the header information amounts of the two are compared with each other with reference to FIG. Here, the box of the difference in the data amount is omitted, and the comparison is performed using the total size of the “sample to chunk” box and the “chunk offset” box.

  The total size of the “sample to chunk” box and the “chunk offset” box when multiplexed by applying the conventional technology is 320 bytes (chunk No, number of samples, idx and offset address are 4 bytes each). The total size of “sample to chunk” box and “chunk offset” box when multiplexed according to the invention is 72 bytes, which is reduced by 248 bytes per one second of encoded data. When this is converted into transmission time, when the communication speed is 64000 bps (bit / sec), it becomes about 30 milliseconds. For example, when data for 60 seconds is multiplexed, transfer is about 1.8 seconds. Time can be shortened. In streaming playback, the smaller the system (header) information transmitted prior to the encoded data, the shorter the waiting time until playback starts.

1 is a system configuration diagram of a code multiplexing apparatus according to the present invention. It is a figure which shows the multiplexed stream input into a decoding apparatus. It is the schematic of the format of an MPEG-4 file. It is a figure which shows a video and an audio encoding sequence. FIG. 4 shows a multiplexed stream according to the present invention. It is a figure which shows the multiplexed stream by a prior art. It is a multiplexing process flowchart by this invention. It is the figure which compared the header information amount with this invention and the prior art. FIG. 6 is a decoder buffer size input diagram according to the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Audio encoding file 102 Video encoding file 103 Application 104 Multiplexing process part 105 Multiplexing file 106 Audio output buffer 107 Video output buffer 108 System information buffer 109 Decoder buffer size input part 301 Media Data Box
302 Track Box (Video)
303 Track Box (Audio)
304 Chunk Offset Box
305 Sample to Chunk Box
306 Sample Size Box
307 Time to Sample Box
401 Video sequence 402 Audio sequence 701 Video frame read processing 702 Video frame size storage processing 703 Video frame / time stamp storage processing 704 Audio frame read processing 705 Time stamp comparison processing 706 Audio data size comparison processing 707 Video output buffer write processing 708 Audio output Buffer write processing 709 Multiplex file write processing (from video output buffer)
710 Multiplex file write processing (from audio output buffer)
711 System information output buffer write processing 712 Multiplex file write processing (from system information output buffer)

Claims (5)

  A feature is provided with a decoder buffer size input unit for designating a block size of each encoded data when generating a multiplexed stream for reproducing the encoded data in synchronization with each other on the playback device side. A code multiplexing apparatus.   2. The code multiplexing apparatus according to claim 1, wherein the plurality of encoded data includes audio data and image data. In generating a multiplexed stream for reproducing a plurality of encoded data on the playback device side in synchronization with each encoded data,
Specify the block size of each encoded data,
Read the maximum number of frames not exceeding the block size from one of the plurality of encoded data,
When reading a frame having the same time property as the one encoded data from the other of the plurality of encoded data, the frame is read within a range in which the other encoded data does not exceed the block size. Code multiplexing method.   When reading a frame having the same time property as the one encoded data from the other of the plurality of encoded data, the other encoded data is within a range that does not exceed the block size. 4. The code multiplexing method according to claim 3, wherein a frame having a time stamp value smaller than an end time of the last frame is read.   5. The code multiplexing method according to claim 3, wherein the plurality of encoded data includes audio data and image data.