JP3496714B2 - Multiplexing device and multiplexing method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture code having a variable bit rate or a fixed bit rate, in which the allocation of the number of bits of each frame is different, and a voice code having a variable bit rate or a fixed bit. The present invention relates to a video / audio multiplexer that multiplexes in frame units and records it in a storage medium so that it can be played back in synchronization with a voice code of a rate. The present invention relates to a moving picture voice multiplexing apparatus capable of multiplexing a moving picture code and a voice code by using a memory of a size.

[0002]

2. Description of the Related Art Normally, an image is digitized into a CD-RO.
When recording on a recording medium such as an M or a hard disk, the amount of data is enormous, so the data is compressed and encoded. As a compression encoding method, a DCT that performs compression by utilizing the property that the spatial frequency of an image is concentrated in a low frequency
Many encoding methods based on (Discrete cosine Transformation) are used. This compression encoding method is based on JPEG (Joint Photograp
hic Experts Group: International standard for color still image compression)
And MPEG1 (Moving Pictures Experts Group 1:
It has been adopted as an international standard for moving image compression) or an international standard such as MPEG2.

The conventional multiplexing operation in the case of MPEG will be described below. FIG. 1 shows the configuration of a system coding control device 51 that multiplexes a video code and an audio code.
7 shows. The system coding control device 51 includes an audio pack generation unit 52 that packs audio codes.
And a video pack generation unit 53 that packs the video code.
Composed of and.

In the system coding control device 51 configured as described above, the audio pack generation unit 52 cuts out audio codes for packets from the audio code file 54, and adds a packet header and a pack header to the cut out audio codes. Add and generate an audio pack. Further, the video pack generation unit 53 cuts out a video code for packets from the video code file 55, adds a packet header and a pack header to the cut out video code, and generates a video pack.
Then, these packs are stored for each frame to generate a system code.

In the above technique, when the video code and the audio code are multiplexed, usually, the video code and the audio code to be reproduced at the same time so that the moving picture and the audio can be reproduced in synchronization on the decoder side. Are arranged alternately.

[0006]

By the way, when a video code and an audio code are multiplexed, if both codes are compression-encoded at a fixed bit rate, the bit rate of each code is relatively easy. Can be processed. For example, a video code encoded at 150000 bps and 24
When multiplexing with an audio code encoded at 000 bps, the bit rate of the video code is 6.
25 times (150000/24000) times. Therefore, the length of the video code packet and the length of the audio code packet may be the same, and one audio packet may be inserted after six to seven video packets.

However, even at a constant bit rate, I usually
The allocation of the number of bits is different for each PB picture. Therefore, it is difficult to accurately synchronize the frame and the video code and the audio code for multiplexing. In particular, when the video code is compressed at a variable bit rate in order to keep the image quality uniform, the multiplexing is more difficult because the ratio of the information amount of both codes is not constant. In addition, since the bit rate of the audio code may change in the middle of the process, the ratio of the information amount of both codes is not constant, and thus the multiplexing is difficult.

As a technique for solving such a problem, in the case of a variable bit rate, as a method for realizing multiplexing of moving image / sound, Japanese Patent Laid-Open Nos. 6-343158, 10-199140 and 10-199140, 10-199141
No. 10, JP-A-10-234010, JP-A-10-320
There is a technique described in No. 914. JP-A-6-343158
The technique of No. stores the video code and the audio code in the respective buffers, and controls the transfer / waiting of the coded code to the multiplexing circuit according to the storage amount of the buffer. Therefore, even if the generation amount of code data is variable, overflow and underflow of the buffer can be suppressed. Further, by switching the input data to the multiplexing circuit at constant time intervals, both codes are multiplexed at constant time intervals.

In the technique of Japanese Patent Laid-Open No. 10-199140, a delay circuit that absorbs a difference in processing time of each encoder and a timing circuit that outputs timing data are provided in the subsequent stage of an audio encoder or a video encoder. Then, by these circuits, the codes at substantially the same time are multiplexed so that the information amount becomes substantially equal to the bit rate ratio of each code.

In the technique disclosed in Japanese Patent Laid-Open No. 10-199141, a delay circuit is provided between the audio encoder and the buffer memory to monitor the total amount of data stored in each code encoder. Then, each time the total amount of monitored data reaches a predetermined amount, the data is multiplexed by the controller that reads each data. In this way, the codes at approximately the same time are multiplexed so that the information amount becomes approximately equal to the bit rate ratio.

The technique disclosed in Japanese Patent Laid-Open No. 10-234010 stores a video code and an audio code in respective buffers and compares the sum of the stored amounts with a predetermined reference value. Then, when the accumulated amount became equal to the reference value, each data was packetized and multiplexed. JP-A-10-32091
The technique of No. 4 compares a time stamp indicating a reproduction time and / or a decoding time of a video frame with a time stamp indicating a reproduction time and / or a decoding time of an audio frame. Then, by comparing the time stamps,
Both frames were multiplexed so as to have the same time property.

However, none of the above techniques considers an application that does not have a function of multiplexing a moving image and audio, and such an application cannot control the multiplexing function. . Further, in any of the above techniques, it is necessary to store the video code and the audio code at substantially the same time in the memory,
Saving memory was not considered. Therefore, there is a problem that the required memory increases.

Therefore, the first problem to be solved by the present invention is to easily multiplex even an application having no multiplex function by integrating the multiplex function in the multiplex means of the multiplex device. It is to provide the technology that can be done. A second problem is to provide a technique capable of easily multiplexing a moving image code and a voice code by using a fixed size buffer.

A third object is to provide a technique capable of saving the buffer size required by the multiplexer.

[0015]

The above-mentioned problem is a multiplexing device for multiplexing two or more types of codes, which comprises a plurality of storage means provided for each type of code and a plurality of storage means. When codes of each type are read out and multiplexed, and when one type of code is insufficient, the missing type of code is stored in the corresponding storage means, and the missing code is stored in the storage means. Then, it is achieved by a multiplexing device having a multiplexing means for reading the stored code and continuing the multiplexing.

Further, the above problem is a multiplexing device for multiplexing two or more types of codes, and a plurality of storage means are provided for each type of code and store each type of code in a predetermined order. And, when the codes are read from the plurality of storage means for each type and multiplexed, and when one type of code is insufficient,
When the code of the missing type is stored in the corresponding storage means by changing the predetermined order and the missing code is stored in the storage means, the stored code is read and the multiplexing is continued. It is achieved by a multiplexing device comprising:

The multiplexing device further has a second storage means for temporarily storing the code to be multiplexed, and the multiplexing means is capable of multiplexing when one type of code is insufficient. Another type of code that has not been encoded is read from the storage means and stored in the second storage means, and when the multiplexing of the insufficient type of code is completed, it is stored in the second storage means. It is preferable to further have means for reading the code that is present and multiplexing the code.

Further, the multiplexing device further reads the code from the file in which the code to be multiplexed is stored, stores it in the storage means, and receives the instruction from the multiplexing means. It is preferable to have a reading means for reading the code and storing it in the storage means. Further, the reading means is configured to read the codes to be multiplexed from the file in a predetermined order, and when receiving an instruction from the multiplexing means, the predetermined order is changed to change the specified type. It is preferably configured to read the code.

Further, the multiplexing device further calculates the storage size of the storage means and / or the second storage means based on the information of the code to be multiplexed, and the storage means and / or the storage size of the second storage means. It is preferable to further have means for securing the second storage means. Further, when storing the code in the second storage means, the multiplexing means, when the code amount to be stored is less than a predetermined storage size of the second storage means, orders the read means. It is also possible to configure so as not to issue an instruction to change.

Further, when the one type of code is insufficient, the multiplexing means is configured so as not to overwrite the storage means in which the unmultiplexed code is stored, but to perform the multiplexing. It is possible to continue. Also,
The above-mentioned problem is a multiplexing method for multiplexing two or more types of codes, a step of setting a storage area for each type of code, and reading and multiplexing the codes from the storage area,
When one type of code is insufficient, a step of storing the insufficient type of code in the corresponding storage area; and when the insufficient code is stored in the storage area, the insufficient code is read and multiplexed. And a step of continuing the multiplexing.

Further, the above-mentioned problem is a multiplexing device for multiplexing two or more types of codes, wherein a plurality of first storage means provided for each type of code and a code to be multiplexed are temporarily stored. Code stored in the first storage means and the plurality of first storage means are read out for multiplexing, and the codes are multiplexed for each type. If one type of code is insufficient, another type of code that is not multiplexed is used. A code is read from the first storage means and stored in the second storage means, and the code of the missing type is stored in the corresponding first storage means and stored in the first storage means. The second code is read out and multiplexed,
And a multiplexing unit for reading and multiplexing the code stored in the storage unit of the above.

Further, the above-mentioned problem is a multiplexing device for multiplexing two or more types of codes, and a plurality of storage means provided for each type of code and a code of the notified type in a predetermined file. If there is a shortage of one type of code, the reading means for reading and storing in the storage means corresponding to the notified type and the code for each type are read out from the plurality of storage means and multiplexed, and this is insufficient. After notifying the reading means of the type of code and storing the code of the type notified by the reading means in the corresponding storage means by the notification,
It is also achieved by a multiplexing device characterized by comprising a multiplexing means for reading and multiplexing the stored code.

Further, the above-mentioned problem is a multiplexing device for multiplexing two or more types of codes, wherein a plurality of first storage means provided for each type of code and a code to be multiplexed are temporarily stored. Second storage means that is stored temporarily, a reading means that reads a code of the notified type from a predetermined file, and stores the code in the first storage means corresponding to the notified type, and the plurality of first storage means. When the codes for each type are read out from the storage means and multiplexed and one type of code is insufficient, the first
Other types of codes that are not multiplexed are read from the storage means of the above, stored in the second storage means, the types of the insufficient codes are notified to the reading means, and the reading means causes the reading means to After storing the code in the first storage means corresponding to the notified type, the code stored in the first storage means is read out and multiplexed, and after this multiplexing, the second code is stored.
And a multiplexing means for reading and multiplexing the code stored in the storage means.

[0024]

BEST MODE FOR CARRYING OUT THE INVENTION A first embodiment will be described below. In the present invention, a case where a video code and an audio code are used as a code to be multiplexed will be described.
A first embodiment of the present invention will be described with reference to FIGS. Here, the drawings used will be briefly described.

The structure of the multiplexer according to the present invention is shown in FIG. Also, regarding MPEG which is an application of the present invention, FIG.
~ Respectively shown in FIG. FIG. 2 shows the structure of a code format compliant with MPEG video, and FIG. 3 shows the structure of a code format compliant with MPEG audio. And
Regarding the multiplexing code when these are multiplexed, MPEG
FIG. 4 shows the structure of a multiplexing format of a video code and an audio code that conforms to the system. Furthermore, MPE
FIG. 5 shows the configuration of the moving image compression means used for G-type moving image compression, and FIG. 6 shows the configuration of the audio compression means.
A series of flow of the multiplexing operation of the present invention is shown in FIGS. 7 and 8 as a sequence chart. 9 to 11 are operation flowcharts of the system encoding control unit that actually multiplexes both codes. FIG. 12 shows an example of storing a video code using the code storing means (system buffer), and FIG. 13 shows an example of storing an audio code. 14 to 16 show a series of storage examples of both codes by the multiplexing process. FIG. 14 shows the operation when the video code is insufficient in the first multiplexing, and FIG. 15 shows the operation when the audio code is insufficient in the second multiplexing. Further, FIG. 16 shows an operation in the case where the video code becomes insufficient due to the third multiplexing.

First, the structure of the multiplexer of the present invention is shown in FIG. 1 and will be described below. In FIG. 1, reference numeral 1 is a multiplexing device. The multiplexing device 1 includes a system encoding control unit 2 and
Audio pack generation unit 3 and video pack generation unit 4
, An interface control unit 5, an audio buffer 6, a video buffer 7, a system buffer 8 and a temporal buffer 9.

The multiplexer 1 synchronizes a video code and an audio code and multiplexes them. The system coding control unit 2 includes an audio pack generation unit 3 that packs audio codes to generate an audio pack, and a video pack generation unit 4 that packs video codes to generate a video pack. Then, the system encoding control unit 2 multiplexes the audio code and the video code by storing the generated audio pack or video pack in a system buffer 8 which will be described later in synchronization with one frame unit. .

The interface control unit 5 has an audio buffer 6, a video buffer 7, and an audio buffer 6 on the basis of information such as the bit rate of the code from the system encoding control unit 2.
The sizes of the system buffer 8 and the temporal buffer 9 are respectively calculated to secure each buffer. The audio buffer 6 stores audio codes to be multiplexed. Video buffer 7
Is for storing the video code to be multiplexed.
The system buffer 8 stores a system code which is a multiplexing code. In the temporal buffer 9, when the number of codes to be multiplexed at the time of multiplexing becomes insufficient due to one frame or less and the multiplexing cannot be continued, an extra code of a different type from the insufficient code To
It is to be stored temporarily. The interface control unit 5 also mediates notification between the system encoding control unit 2 and an application 10 described later.

An application 10 and a video code file 11 are provided outside the multiplexer 1 configured as described above.
, An audio code file 12 and a system code file 13 are provided. Application 10
Receives the notification from the system encoding control unit 2 and reads the video code and the audio code from the video code file 11 and the audio code file 12, respectively. Here, the video code file 11 is a file in which code strings of video codes are stored. The audio code file 12 is a file in which a code string of audio codes is stored.

The system code file 13 is a file in which a code string of the system code formed by the system coding control unit 2 is stored. Next, MPEG (Moving Pi)
The cture Experts Group) compression method will be described below with reference to FIGS. In FIG. 2, the video data consists of one or more video sequences and ends with a video sequence end code. Here, the video sequence includes a video sequence header and one or more GOs.
P (Group Of Picture). At this time,
A GOP is composed of one or more pictures, and one picture represents one image. A picture header containing information such as the picture type is placed at the beginning of the picture.

The picture includes an I picture composed only of intra-frame codes, a P picture composed of inter-frame codes only in the forward direction, and a B picture composed of bidirectional inter-frame codes before and after. There are three types. These various pictures are composed of a plurality of slices divided into arbitrary areas. The plurality of slices are composed of a plurality of macroblocks arranged in order from left to right or from top to bottom. Here, in the macro block,
Broadly speaking, an intra block, which is an intraframe code,
There are two types, interblocks which are forward or bidirectional interframe codes. The above-mentioned I picture is composed of only intra blocks, but P pictures and B pictures may include not only inter blocks but also intra blocks. And the macroblock is 16
Luminance components (Y1, Y2, Y3, Y4) obtained by further dividing a block of × 16 dots into blocks of 8 × 8 dots;
It is composed of 6 blocks including the color difference components (Cb, Cr) of the 8 × 8 dot block in the area that matches the luminance component. At this time, in the macroblock, a block of 8 × 8 dots is the minimum unit of encoding.

Next, the audio data will be described. In FIG. 3, an audio sequence is composed of one or more AAUs (Audio Access Units). One AAU is composed of an AAU header containing information such as a sync word, a bit rate, a sampling frequency, and a compressed audio code. In audio data, one AAU is the minimum unit for encoding. Speech code is usually a fixed bit rate,
The bit rate can be changed in units of AAU. In that case, the bit rate value of the AAU header also changes.

Next, a multiplexing format of a video code and an audio code based on the MPEG system will be described. In FIG. 4, the video code and the audio code are multiplexed to form one system stream. This system stream is composed of a plurality of packs, and this pack is composed of a pack header and one or more packets. Each packet has a video packet and an audio packet. A video packet is composed of a packet header and a video code string. If you take out only the video code string in this video packet and continue,
A series of video sequences is constructed. The audio packet is composed of a packet header and an audio code string. When only the audio code string in this audio packet is extracted and continued, a series of audio sequences is formed.

Next, the compression means for compressing both codes will be described respectively. First, the structure of the MPEG image compression means 30 is shown in FIG. In FIG. 5, the image compression means 30 is a signal format (Y,
A signal conversion unit 31 for converting to a Cr, Cb format, and a block (MP
In the EG, a motion search unit 32 that searches for each region of 16 pixels × 16 pixels), a frequency conversion unit 33 that converts a block image into a spatial frequency, a quantization unit 34 that performs quantization, and a variable length encoding The variable-length coding unit 35 that performs the motion compensation, the motion compensation unit 36 that compensates the motion of the image on a block-by-block basis to generate a new reference frame, the inverse frequency conversion unit 37 that returns from the frequency conversion to the original, and the quantization based on the original It comprises an inverse quantizer 38 for performing inverse quantization.

The image compression means 30 having the above configuration converts the original image into Y, Cr, Cb data by the signal conversion section 31 and performs compression according to each picture type. In the case of I picture, Y, Cr, Cb data is divided into blocks,
The frequency conversion unit 33 converts each block into a spatial frequency. Then, the converted data is quantized by the quantizer 34, variable-length coded by the variable-length coding unit 35, and the compressed code is output. Further, the quantized block is inversely quantized by the inverse quantization unit 38. Then, the block is inverse-frequency-converted by the inverse-frequency converter 37 to create a reference frame.

In the case of a P picture, Y, Cr, Cb data is divided into blocks, and the block having the highest correlation among the blocks of the previous frame stored as the reference frame by the motion search unit 32 is obtained for each block. . At this time, the frequency converter 33 converts the difference between the Y, Cr, Cb data divided into blocks and the block of the previous frame having the highest correlation into a spatial frequency. Then, the converted difference is quantized by the quantizer 34, variable-length coded by the variable-length coding unit 35, and the compressed code is output.
Further, the quantized block is inversely quantized by the inverse quantization unit 38, the inverse frequency conversion unit 37 performs inverse frequency conversion, and the motion compensation unit 36 adds the motion-compensated block to create a reference frame.

In the case of a B picture, the Y, Cr, Cb data is divided into blocks, and the block having the highest correlation among the blocks of the previous and subsequent frames stored as the reference frame by the motion search unit 32 for each block. Ask for.
The frequency conversion unit 33 converts the difference between the Y, Cr, Cb data divided into blocks and the block of the front / rear frame having the highest correlation into a spatial frequency. Subsequently, the converted difference is quantized by the quantizer 34, variable-length coded by the variable-length coding unit 35, and the compressed code is output. In the case of a B picture, it is not necessary to create a reference frame.

Next, the MPEG-compliant voice compression means 4
The configuration of 0 is shown in FIG. In FIG. 6, the voice compression means 4
Reference numeral 0 denotes an original sound cutout unit 41 that cuts out original sound data of 1 AUU from an original sound, 32 frequency band mapping units 42 that perform a frequency band mapping process in units of 1 AAU, and a quantization coding unit 43 that performs linear quantization and coding. And a frame forming unit 44 that adds compressed data to the encoded data to create compressed data for 1 AAU, and a psychoacoustic unit 45 that performs a psychoacoustic process.

The audio compression means 40 having the above-mentioned structure cuts out the original sound by 1 AAU (1152 samples in the case of the MPEG audio layer 2) by the original sound cutting section 41 and compresses it in AAU units. Then, the 32 frequency band mapping unit 42 decomposes the input signal into 32 band sub-band signals by the sub-band analysis filter, calculates the scale factor for each sub-band signal, and aligns the dynamic range. Subsequently, the psycho-acoustic unit 45 performs fast Fourier transform on the input signal, calculates psycho-acoustic masking from the result, and calculates bit allocation for each sub-band.

The quantization coding unit 43 also performs quantization and coding according to the determined bit allocation. further,
The frame forming unit 44 adds a header and auxiliary information to the quantized and encoded subband signal, shapes the bitstream, and outputs the bitstream as a compression code. Next, the multiplexer 1
The operation of the application 10 and the application 10 will be described with reference to the sequence charts of FIGS. 7 and 8. In the present description, as an order of multiplexing codes, an example in which video codes are first multiplexed in the order of audio codes will be described, but the invention is not limited to this order, and audio codes and video codes are not limited thereto. May be multiplexed in the order of.

First, before explaining the specific operation, the normalization of the multiplexing process and the error notification transmitted from the system coding controller 2 of the multiplexer 1 to the application 10 via the interface controller 5 are notified. Will be described. The multiplexing process according to the present invention is performed by the temporal buffer 9
And the audio buffer 6 or the video buffer 7 in which a code different from this code is stored.
The code stored in the above is alternately multiplexed in frame units. In such multiplexing, when all the codes stored in the audio buffer 6 or the video buffer 7 as well as the temporal buffer 9 are multiplexed, the system encoding control unit 2 causes the interface control unit 5 to execute Application 10 (direct application 10 when the interface control unit 5 is not used)
Notify that the multiplexing process has completed normally. However, when the system coding control unit 2 takes out the code for the first time, only when the taken-out code is stored in the temporal buffer 9, the notification of the normal end is transmitted as an exception.

Next, the error notification will be described. As described above, both codes to be multiplexed are alternately stored in the system buffer 8 by the system coding control unit 2. However, as the multiplexing progresses, the code amount stored in various buffers runs out, and it may not be possible to continue the multiplexing any more. Therefore, in such a case, the system encoding control unit 2 sends a notification requesting to add a code of the missing type via the interface control unit 5 to the application 10 (directly when the interface control unit 5 is not used. Application 10)
Send to. This notification is an error notification.

Next, the operation of the multiplexing process will be described. First, the operation until the first call is made at the start of the actual multiplexing process and the notification of the normal termination of the multiplexing process is transmitted will be shown. The external application 10 notifies the interface control unit 5 of information (for example, an address and a file name) of the video code file 11 and the audio code file 12 storing the code to be multiplexed (S1). The interface control unit 5, which has received the information of both code files, notifies the system coding control unit 2 of the information of the video code file 11 (S2). System coding control unit 2 notified of information of video code file 11
Reads the information such as the size and bit rate of the video code from the video code file 11 and informs the interface controller 5 of the information (S3).

Then, the interface control section 5 informs the system encoding control section 2 of information (for example, address and file name) of the audio code file 12 (S).
4). The system encoding control unit 2, which has been notified of the information of the audio code file 12, reads the information such as the number of channels and the bit rate of the audio code from the audio code file 12 and informs the interface control unit 5 of the information (S5). .

Further, the interface control unit 5 calculates the size of the buffer for storing each code, based on the notified information such as the bit rate of both codes. The calculated buffer includes an audio buffer 6 storing an audio code, a video buffer 7 storing a video code,
The system buffer 8 stores a system code in which both codes are multiplexed.

The formulas (1) to (3) for calculating the sizes of the audio buffer 6, the video buffer 7, and the system buffer 8 are shown below. Audio buffer size: (Ar x T) / 8 (bytes) (1) Video buffer size: (Vr x T) / 8 (bytes) (2) System buffer size: ((Vr + Ar) x α x T) / 8 (byte) (3) In the formula, Ar is the bit rate (bit / bit) of the audio code.
Second) and indicates the bit rate of the AAU at the beginning of the code when the code bit rate changes in AAU units. Vr represents the bit rate (bits / second) of the video code, and if the code has a variable bit rate, any value less than 15 Mbps,
For example, 6 Mbps. α indicates a correction value in consideration of the header part of the system code. For example, if 1 pack = 1 packet and the packet size is 2048 bytes, and the header size is 34 bytes, then 2048 / (2048-34) = 1.1
7 T indicates the time (second) of the information amount to be secured.

When the above equation is specifically calculated, for example, T
= 0.1, Ar = 224000, Vr = 6000000, α = 1.17, the size is as follows. Audio buffer size: (224000 x 0.1) / 8 = 2800 (bytes) Video buffer size: (6000000 x 0.1) / 8 = 75000 (bytes) System buffer size: {(6000000 + 224000) x 1.17 x 0.1} / 8 = 91026 (bytes) In this way, the interface control unit 5 secures the audio buffer 6, the video buffer 7, and the system buffer 8 based on the calculation result. Then, the interface control unit 5 notifies the application 10 of the addresses and sizes of these buffers (S
7).

Further, the interface control unit 5 secures the temporal buffer 9 having the same size as the larger size of the calculated sizes of the audio buffer 6 and the video buffer 7. The formula for calculating the size of the temporal buffer 9 is shown below in formula (4). Temporal buffer size: (Max (Vr, Ar) × T) / 8 (bytes) (4) Here, Max (a, b) in Expression (4) is the larger value of a and b. Is shown. Calculating formula (4), the size of temporary buffer: (6000000 × 0.1) / 8 = 7
It will be 5000 (bytes).

As described above, the interface control section 5 secures the temporal buffer 9 and informs the system encoding control section 2 of its address and size (S
6). The application 10, which has received the notification of the addresses and sizes of the audio buffer 6, the video buffer 7, and the system buffer 8, reads the code corresponding to the size of the video buffer 7 from the video code file 11 and stores it in the video buffer 7 to store the video. The buffer 7 is passed to the interface controller 5. For example, the control right of the video buffer 7 is passed (S8). The interface control unit 5, which has received the video buffer 7, transfers the video buffer 7 to the system encoding control unit 2 (S9). Upon receiving the video buffer 7, the system coding control unit 2 reads the video code from the video buffer 7 and copies the video code to the temporal buffer 9. Then, the system coding control unit 2 transmits a notification of the normal end of the multiplexing process to the interface control unit 5 after copying the video code (S10). Interface controller 5 that received the notification
Sends this notification of normal termination to the application 10 (S11).

The application 10 which has been notified of the normal end reads the code of the size of the audio buffer 6 from the audio code file 12 and stores it in the audio buffer 6. The application 10 storing the audio code passes the audio buffer 6 to the interface control unit 5 (S12). Audio buffer 6
The interface control unit 5 that received the data passes this audio buffer 6 to the system encoding control unit 2 (S1).
3).

The system encoding control unit 2, which has received the audio buffer 6, reads the video code stored in the temporal buffer 9. Then, the system encoding control unit 2
Packs the read video code in the video pack generation unit 4, and repeats the packing, and stores it in the system buffer 8 until the code amount for one frame is reached. After storing the video pack, the system coding control unit 2 reads the audio code from the audio buffer 6 and causes the audio pack generation unit 3 to pack the audio code. Then, the system encoding control unit 2 stores this audio pack in the system buffer 8 until the code amount for one frame is reached.

As described above, the system coding control unit 2
In this case, since both codes are alternately multiplexed, when this multiplexing process progresses, one of the codes has a code amount of less than one frame, and the multiplexing cannot be continued. If both codes have a fixed bit rate, the same number of frames are extracted from each buffer and multiplexed, so the video code that is the first code to be extracted has a code amount of less than one frame first. Becomes Therefore, the system encoding control unit 2
Reads out the remaining audio code that has not been multiplexed yet from the audio buffer 6 and stores it in the temporal buffer 9. Immediately after the audio code is stored in the temporal buffer 9, the system coding control unit 2 transmits an error of insufficient video code to the interface control unit 5 (S1).
4). The interface control unit 5, which has received the video code shortage error, notifies the application 1 of this error.
0 (S15).

The application 10 which has received the video code shortage error reads the system code stored in the system buffer 8 and writes it in the system code file 13. Then, the application 10 reads the video code of the size of the video buffer 7 from the video code file 11 and stores it in the video buffer 7. Further, the application 10 passes the video buffer 7 to the interface control unit 5 (S16). The interface control unit 5, which has received the video buffer 7, transfers the video buffer 7 to the system encoding control unit 2 (S17).

The system coding control unit 2 which has received the video buffer 7 reads out the video code stored therein. Here, in the previous multiplexing (S14), since the video code has been multiplexed in the middle of one frame, the continuation is continued, and therefore the system encoding control unit 2 sets the remaining video code of this frame. Stored in the system buffer 8. Then, the system encoding control unit 2 reads the audio code from the temporal buffer 9, packs it, and stores it in the system buffer 8.

If the audio code stored in the temporal buffer 9 is less than one frame, the system coding control unit 2 cannot continue the multiplexing. Therefore, the system encoding control unit 2 stores the audio code of less than one frame in the system buffer 8, reads the excess video code that has not been multiplexed yet from the video buffer 7, and stores it in the temporal buffer 9. After storing the surplus video code in the temporal buffer 9, the system coding control unit 2 transmits an audio code shortage error to the interface control unit 5 (S18). The interface controller 5, which has received the audio code shortage error, transmits this error to the application 10 (S19).

The application 10 receiving the audio code shortage error writes the system code stored in the system buffer 8 to the system code file 13.
Further, the application 10 reads the audio code of the size of the audio buffer 6 from the audio code file 12 and stores it in the audio buffer 6.
Furthermore, the application 10 uses the audio buffer 6
Is passed to the interface control unit 5 (S20). The interface control unit 5, which has received the audio buffer 6,
The audio buffer 6 is passed to the system encoding control unit 2 (S21).

The system encoding control unit 2 which has received the audio buffer 6 reads the audio code from the audio buffer 6. Here, in the previous multiplexing (S18), since the multiplexing is completed in the middle of one frame of the audio code, the system encoding control unit 2 stores the remaining audio code of this frame in the system buffer 8. Then, the system coding control unit 2 reads the video code from the temporal buffer 9, packs it, and stores it in the system buffer 8.

In the case of the present multiplexing, the video code stored in the temporal buffer 9 becomes less than one frame before the audio code in which a new code of the size of the audio buffer 6 is stored. . For this reason,
The system encoding control unit 2 stores the video code of less than one frame in the system buffer 8. Then, the system encoding control unit 2 extracts the audio code remaining in the audio buffer 6 without being multiplexed and stores it in the temporal buffer 9. After storing such an audio code in the temporal buffer 9, the system coding control unit 2 sends an error of insufficient video code to the interface control unit 5 (S22). The interface control unit 5, which has received the error, transmits an error of insufficient video code to the application 10 (S23).

The above operation is repeated until the end of the multiplexing process. Next, the operation of the system coding control unit 2 of the multiplexing device 1 will be described with reference to FIGS. First, when there is a call from the interface control unit 5, the system encoding control unit 2 determines whether or not it is the first call (S100).

If it is the first call, the system coding control unit 2 confirms whether or not the stored code is a video code (S119). The system coding control unit 2 stores the video code in the temporal buffer 9 if it is a video code (S120), and stores the audio code in the temporal buffer 9 if it is not a video code (S121). After being stored in the temporal buffer 9, the interface controller 5 is notified of the normal end (S122).

If it is not the first call (S10
0), the system encoding control unit 2 takes out the audio code stored in the temporal buffer 9 or the audio buffer 6 (S101). If there is an audio code (S102), the system coding control unit 2 sets 1
It is confirmed whether the audio code for the packet can be taken out (S103). When the audio code for one packet can be extracted, the packet header and the extracted audio code for one packet are stored in the system buffer 8 (S104). If the audio code for one packet cannot be extracted, the process returns to S101. The system encoding control unit 2 confirms whether the stored code is one frame or more (S105). If the audio code for one frame cannot be stored, the process returns to S101. When one frame or more is stored (S105), the system coding control unit 2
Stores the video code, the video code is taken out from the temporal buffer 9 or the video buffer 7 (S106).

When there is a video code (S107), the system coding control unit 2 confirms whether or not the video code for one packet can be taken out (S108). If the video code for one packet cannot be extracted, the process returns to S106.
When the video code for one packet can be extracted, the packet header and the extracted video code for one packet are stored in the system buffer 8 (S109). Then, the system encoding control unit 2 confirms whether the picture header of the next frame of the video code is stored in the system buffer 8 or whether the next video code starts from the picture header (S110). If the picture header of the next frame of the video code is stored in the system buffer 8, or if the next video code confirms the start from the picture header,
Since the system coding control unit 2 has stored one frame of the video code in the system buffer 8, the process returns to S101. If the picture header of the next frame of the video code cannot be stored in the system buffer 8 and the start of the next video code from the picture header cannot be confirmed, the system coding control unit 2 returns to S106.

On the other hand, if there is no audio code in the judgment of S102, the system coding control unit 2 confirms whether or not the video code remains in the video buffer 7 (S111). When the video code remains (S11
1), the system encoding control unit 2 stores the remaining video code in the temporal buffer 9 (S113). Furthermore, the system coding control unit 2 transmits an error of audio code shortage to the interface control unit 5 (S11).
4).

When there is no video code (S111), the system coding control unit 2 sends a notification of normal termination to the interface control unit 5 (S112). If there is no video code in S107, the presence / absence of an audio code in the audio buffer 6 is confirmed (S115).
If the audio code remains, the system encoding control unit 2 stores the remaining audio code in the temporal buffer 9 (S117). Further, the system encoding control unit 2 transmits an error of insufficient video code to the interface control unit 5 (S118).

If no audio code remains, the system coding control section 2 sends a normal termination notification to the interface control section 5 (S116). Since the system encoding control unit 2 centralizes the multiplexing control in this way, the application does not need to be directly involved in the multiplexing process. Storage examples of the video code and the audio code using the system buffer are shown in FIGS. 12 and 13, respectively.

FIG. 12 shows an example of storing a video code using a system buffer. In the example of FIG. 12, the packet size is fixed at 1 pack = 1 packet. Since the picture header is inserted in the video code on a frame-by-frame basis, when the code immediately before the picture header is stored, it is regarded as the end of one frame. However, since the packet size is fixed, the picture header indicating the beginning of the frame of the video code as shown in FIG. 12A is not always stored at the beginning of the beginning packet of the frame. In this case,
2B and 12C, the code of the previous frame and the code of the next frame may be arranged together with the picture header interposed.

FIG. 13 shows an example of storing an audio code using a system buffer. In the example of FIG. 13, the packet size is fixed at 1 pack = 1 packet. The audio code does not have a header inserted in frame units, but the bit rate is fixed in AAU units.
When the time for one frame is calculated and the code for one frame is stored, it is regarded as the end of one frame. However, since the packet size is fixed, the head of the code for one frame is not always stored at the head of the packet in the audio code as shown in FIG. In this case, the head of the code for one frame may be arranged together with the code of the previous frame as shown in FIGS. 13 (B) and 13 (C).

In the example of FIGS. 12 and 13, one pack = 1 packet, but a plurality of packets arranged continuously for one frame may be collectively put into one pack. Also, the size of both packets is fixed,
The size of the packet storing the last code of one frame may be changed so that the code of the next frame is not stored.

The operation of both codes stored in each buffer will be described below with reference to FIGS. 14 to 16. First, FIG. 14 shows the operation when the system coding control unit 2 multiplexes the video code up to the middle of the third frame and the audio code up to the middle of the third frame and runs out of video codes. This is shown as an example of multiplexing. 14
As shown in, the system encoding control unit 2 packs the audio code of the first frame starting with the video code of the first frame, and then packs the video code and the audio code alternately up to the second frame. To do. And
The system encoding control unit 2 stores the video code in the middle of the third frame in the system buffer 8. However, since the code amount of the video code of the third frame is less than one frame, the system coding control unit 2 cannot continue the multiplexing process. Therefore, the system coding control unit 2 stores the audio code up to the middle of the third frame, which is left without being multiplexed, in the temporal buffer 9. Then, the system coding control unit 2 sends an error of video code shortage to the application 1 via the interface control unit 5.
Notify 0.

Next, after the first multiplexing, the system coding control unit 2 causes the video code from the middle of the third frame to the middle of the fourth frame and the audio code up to the middle of the remaining third frame last time. FIG. 15 shows an operation in the case where the audio codes become insufficient by multiplexing and. As shown in FIG. 15, the system coding control unit 2 reads out the video code from the middle of the third frame from the video buffer 7 continuously from the first multiplexing. Then, the system coding control unit 2 packs the read video code to the end of the third frame and stores it in the system buffer 8.
Subsequently, the system encoding control unit 2 stores in the system buffer 8 the audio code up to the middle of the third frame stored in the temporal buffer 9. However, since the code amount of the audio code of the third frame is less than one frame, the system coding control unit 2 cannot continue the multiplexing process. Therefore, the system encoding control unit 2
Stores the video code up to the middle of the fourth frame remaining in the video buffer 7 in the temporal buffer 9.
Then, the system coding control unit 2 notifies the application 10 of an audio code shortage error via the interface control unit 5.

Further, after the second multiplexing, the system coding control unit 2 outputs the audio code from the middle of the third frame to the middle of the fifth frame and the video code up to the middle of the remaining fourth frame last time. FIG. 16 shows the operation in the case where the video codes are not enough due to the multiplexing of. Figure 1
As shown in FIG. 6, the system encoding control unit 2 packs the audio code from the middle of the third frame to the end of the third frame and stores it in the system buffer 8 continuously from the second multiplexing. . Then, the system encoding control unit 2 outputs the video code up to the middle of the fourth frame stored in the temporal buffer to the system buffer 8
To store. However, since the code amount of the video code of the fourth frame is less than one frame, the system coding control unit 2 cannot continue the multiplexing process. Therefore, the system encoding control unit 2 stores in the temporal buffer 9 the audio codes from the fourth frame to the middle of the fifth frame that are left without being multiplexed. Then, the system encoding control unit 2 stores this audio code, and then reports an error of insufficient video code via the interface control unit 5.
Notify the application 10.

This is the end of the description of the first embodiment. In the first embodiment, when the video code or audio code that is left over during multiplexing is stored in the temporal buffer 9, the system coding control unit 2 generates an error that the video code or audio code is insufficient. However, when the size of the code stored in the temporal buffer 9 by the system encoding control unit 2 is a part of the temporal buffer 9, for example, half or less, it is not an error notification but the normalization of the multiplexing process. A notification of termination may be sent. In this case, the system coding control unit 2 multiplexes the video code or the audio code stored in the temporal buffer 9, and then multiplexes the code stored in the video buffer 7 or the audio buffer 6.

Next, a second embodiment will be described. still,
Regarding the configuration of the multiplexing device 14 in the second embodiment, description of the same components as those in the first embodiment will be omitted, and only different components will be described. The difference from the first embodiment is that (1) the temporal buffer 9
The system encoding control unit 2 prohibits overwriting of the buffer in which the excess code is stored, and (2) identifies the non-multiplexed code stored in the buffer in which overwriting is prohibited without using It is the point that is configured.

In order to prevent the buffer from being overwritten, the system encoding control unit 2 uses the application 10
In addition to notifying the error to, the notification that prohibits overwriting of the buffer storing the excess code is also performed. By this notification, the application 10 stores the code only in the buffer notified by the error, and does not store (overwrite) the code in the buffer in which overwriting is prohibited.

As a configuration for identifying a code that has not been multiplexed, the system coding control unit 2 stores the address of the boundary between the code that has been multiplexed and the code that has not been multiplexed, and the address stored at the time of multiplexing is used as the basis. A configuration is conceivable in which a code that is not multiplexed in the above is read. As another configuration, a method of filling the area of the multiplexed code with null data can be considered.

Next, the operation of the multiplexer 14 and the application 10 will be described. In order to prohibit overwriting the buffer in which the excess code is stored as described above, in the first embodiment, when multiplexing as in S14 of FIG. A notification indicating that the audio buffer 6 storing the excess code is prohibited from being overwritten is transmitted to the application 10 via the interface control unit 5 together with the notification of the insufficient error. The system coding control unit 2 also stores the address of the buffer at the boundary between the code that has already been multiplexed and the code that has not been multiplexed.

Upon receiving the notification, the application 10 reads only the video code instructed by the error and stores it in the video buffer 7 in which overwriting is not prohibited. Then, the application 10 passes the video buffer 7 to the system encoding control unit 2 via the interface control unit 5. In order to continue the multiplexing, the system coding controller 2 reads the video code from the newly stored video buffer 7 and multiplexes it. After the video code is multiplexed, the system coding control unit 2 reads the remaining audio code from the audio buffer 6 and multiplexes it based on the stored buffer address.

In this way, the multiplexing is continued. This is the end of the description of the second embodiment. In the first and second embodiments described above, the system encoding control unit 2 notifies the external application 10 of the code that has been compressed in advance to add and multiplex the necessary code. There is. However, when the application 10 or the interface control unit 5 operates the image compression unit 30 shown in FIG. 5 and the audio compression unit 40 shown in FIG. 6, each compression unit is compressed during the multiplexing process. Compress no code. Then, while compressing the code by each compression means, the application 10 or the interface control section 5 may receive the already compressed code from each compression means and multiplex it by the system encoding control section 2.

[0079]

As described above, according to the present invention,
Since the reading and writing of codes, the securing of memory, and the multiplexing are layered, the application itself does not need to have the multiplexing function. Therefore, the application itself does not need to know specific information such as the bit rate or code size of the video code or audio code. Therefore, according to the error from the system encoding control unit, the application can continue the multiplexing by considering only the type of the code to be passed next.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a multiplexing device according to an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of a code format based on MPEG video of the present embodiment according to the present invention.

FIG. 3 is a diagram showing a configuration of a code format based on MPEG audio according to the present embodiment of the present invention.

FIG. 4 is a diagram showing a configuration of a multiplexing format of a video code and an audio code based on the MPEG system of the present embodiment according to the present invention.

FIG. 5 is a diagram showing a configuration of an image compression unit used for image compression of the MPEG system of the present embodiment according to the present invention.

FIG. 6 is a diagram showing a configuration of a voice compression unit of the present embodiment according to the present invention.

FIG. 7 is a sequence chart showing a series of flow of a multiplexing operation of this embodiment according to the present invention.

FIG. 8 is a sequence chart showing a series of flow of a multiplexing operation according to the present embodiment of the present invention.

FIG. 9 is a flowchart showing the operation of the system coding control unit that actually performs the multiplexing of this embodiment according to the present invention.

FIG. 10 is a flowchart showing the operation of the system coding control unit for actually performing the multiplexing of this embodiment according to the present invention.

FIG. 11 is a flowchart showing the operation of the system coding control unit that actually performs the multiplexing of this embodiment according to the present invention.

FIG. 12 is a diagram showing a storage example of a video code using the system buffer of the present embodiment according to the present invention.

FIG. 13 is a diagram showing an example of storing an audio code using the system buffer of the present embodiment according to the present invention.

FIG. 14 is a diagram showing an operation in the case where a video code is insufficient due to the first multiplexing according to the present embodiment of the present invention.

FIG. 15 is a diagram showing an operation when an audio code is insufficient due to the second multiplexing of the present embodiment according to the present invention.

FIG. 16 is a diagram showing an operation in the case where the video code becomes insufficient due to the third multiplexing according to the present embodiment of the present invention.

FIG. 17 is a diagram showing a configuration of a system coding control device for multiplexing a video code and an audio code according to a conventional embodiment.

[Explanation of symbols]

1 Multiplexer 2 System coding control unit 3 Audio pack generator 4 Video pack generator 5 Interface control section 6 audio buffer 7 video buffer 8 system buffer 9 Temporal buffer 10 applications 11 video code files 12 audio code files 13 System code file

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ identification code FI H04N 7/24 H04N 7/13 Z (56) Reference JP-A-6-343158 (JP, A) JP-A-8-97795 ( JP, A) JP 9-139720 (JP, A) JP 10-199140 (JP, A) JP 10-199141 (JP, A) JP 10-234010 (JP, A) JP Flat 10-320914 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04J 3/04 G11B 20/10 301 H04L 12/56 H04N 5/92 H04N 7/24

Claims (12)

(57) [Claims] 1. A multiplexing device for multiplexing two or more types of codes, comprising: a plurality of storage means provided for each type of code; and reading and multiplexing the codes for each type from the plurality of storage means. , When one type of code is insufficient, the missing type of code is stored in the corresponding storage means,
A multiplexing device comprising: a multiplexing unit that reads out the stored code and continues multiplexing when the insufficient code is stored in the storage unit. 2. A multiplexing device for multiplexing two or more types of codes, comprising a plurality of storage means provided for each type of code and storing codes of each type in a predetermined order; When the codes are read from the storage means for each kind and multiplexed, and when one kind of codes is insufficient, the predetermined order is changed to store the missing kinds of codes in the corresponding storage means, and the shortage is caused. Multiplexing means for reading the stored code and continuing the multiplexing when the stored code is stored in the storage means. 3. A second storage means for temporarily storing the code to be multiplexed, wherein the multiplexing means is provided when one type of code is insufficient.
Another type of code that is not multiplexed is read from the storage means and stored in the second storage means, and stored in the second storage means when multiplexing of the insufficient type of code is completed. The multiplexing device according to claim 1 or 2, further comprising means for reading out the stored code and multiplexing the code. 4. A code is read from a file in which codes to be multiplexed are stored and stored in the storage means, and when a command is received from the multiplexing means, a code of the type specified is read and stored in the storage means. 4. The multiplexing device according to claim 1, further comprising read means for storing the data. 5. The reading means is configured to read the codes to be multiplexed from a file in a predetermined order, and when receiving an instruction from the multiplexing means, the predetermined order is changed and instructed. The multiplexing device according to claim 4, wherein the multiplexing device is configured to read different types of codes. 6. A storage size of the storage means or / and the second storage means is calculated based on information of codes to be multiplexed, and the storage means or / and the second storage means is secured. 2. The method according to claim 1, further comprising means.
6. The multiplexing device according to claim 5. 7. The reading means when the code is stored in the second storage means when the code amount to be stored is a predetermined value or less than the storage size of the second storage means. The multiplexing device according to claim 4, wherein the multiplexing device is configured not to issue an instruction to change the order. 8. The multiplexing means is configured such that, when one type of code is insufficient, the storage means that stores a code that is not multiplexed is not overwritten. The multiplexing device according to any one of claims 1 to 7. 9. A multiplexing method for multiplexing two or more types of codes, the method comprising: setting a storage area for each type of code; reading a code from the storage area and multiplexing; When the shortage of codes is stored in the corresponding storage area, when the shortage of codes is stored in the corresponding storage area, the step of reading the shortage of codes and continuing the multiplexing And a multiplexing method comprising: 10. A multiplexing device for multiplexing two or more types of codes, wherein a plurality of first storage means provided for each type of code and a code to be multiplexed are temporarily stored. Codes are read from the second storage means and the plurality of first storage means for each type and multiplexed, and when one type of code is insufficient, another type of code that is not multiplexed is used as the first type. Read out from the storage means and store in the second storage means, store the missing type code in the corresponding first storage means, and read the code stored in the first storage means. And a multiplexing unit for reading and multiplexing the code stored in the second storage unit after the multiplexing. 11. A multiplexing device for multiplexing two or more types of codes, comprising: a plurality of storage means provided for each type of code; and reading out the code of the notified type from a predetermined file,
A read means for storing in the storage means corresponding to the notified type, and a code for each type is read from the plurality of storage means and multiplexed, and when one type of code is insufficient, the type of the insufficient code To the reading means, store the code of the type notified by the reading means by the notification in the corresponding storage means, and then read the stored code and multiplex it. Multiplexer. 12. A multiplexing device for multiplexing two or more types of codes, wherein a plurality of first storage means provided for each type of code and a code to be multiplexed are temporarily stored. Second storage means, reading the code of the notified type from a predetermined file,
Read means for storing in the first storage means corresponding to the notified type; and code for each type is read out from the plurality of first storage means and multiplexed, and when one type of code is insufficient, Another type of code that is not multiplexed is read from one storage unit and stored in the second storage unit, the type of the missing code is notified to the reading unit, and the reading unit is notified by the notification. The first corresponding to the notified type
Storing the code in the storage means, reading the code stored in the first storage means for multiplexing, and reading the code stored in the second storage means for multiplexing and multiplexing the multiplexed code. A multiplexing device having.