CN101212679B - AV synchronization method and device for switching multi-channel audio stream during playing of AVI file - Google Patents

Abstract

An AV synchronization method for switching multi-channel audio stream when playing AVI file is applied to a system comprising a switching control unit, a data transmission unit, a data processing unit and a data decoding unit, wherein when playing AVI file such as user switching audio stream, the switching control unit sends switching instruction to the data processing unit and the data decoding unit; the data decoding unit returns the current decoded video frame number; the switching control unit calculates to obtain a retransmission position and a synchronous playing parameter according to the video frame number, the index information and the file header information, and respectively sends the retransmission position and the synchronous playing parameter to the data transmission unit and the data decoding unit; and after the data decoding unit completes the reinitialization of the audio and video decoding, the audio and video data which are retransmitted are decoded, and whether the decoded audio and video data are transmitted to the data display unit for synchronous playing is determined according to the synchronous playing parameters. The method can keep AV synchronization after switching the audio stream when playing the AVI file containing one path of Divx video coding data and multiple paths of audio data.

Description

AV synchronization method and device for switching multiple audio streams during AVI file playback

technical field

The present invention mainly relates to the field of audio and video playback, and in particular to a method for keeping audio and video (AV) synchronization when switching audio data played by an AVI file containing one Divx video encoding data and multiple audio data in an embedded system and device.

Background technique

First explain the relevant technical terms that appear in the text:

I frame: key video frame, lossless compressed video picture, all references to I frame in this file refer to video frame;

Divx: a video encoding format;

AV: Audio and Video, audio and video;

CBR: fixed code stream;

VBR: variable bit stream;

SPHE1001N: Sunplus Home Entertainment 1001N;

STB: The full name is Set-Top-Box, which is the set-top box;

AVI: Audio Video Interleaved, audio and video data interleaved access.

AVI (Audio Video Interleaved (interleaved access)) is the most commonly used media file format, mostly used in audio and video capture, editing, playback and other applications. Usually, an AVI file can contain multiple media streams of different types. Usually the media stream contained in an AVI file is a video stream and an audio stream, but an AVI file containing a video stream and several audio streams is also legal.

As shown in FIG. 1 , the AVI file structure mainly includes three parts: AVI file header, audio and video data, and index information.

Wherein from the AVI file header, the information that can be obtained includes: AVIMAINHEADER (AVI main file header) information, format information, audio stream number, video stream time scale and rate index, audio and video data start position Movie_Offset, audio and video data overall size Movie_Size, Information such as the sampling rate of the VBR audio stream or the byte rate of the CBR audio stream. Among them, the audio stream in the AVI file is usually two encoding formats, VBR audio or CBR audio, so the corresponding parameters are analyzed according to the situation.

Among them, the audio and video data part is to interleave and store the actual audio and video data of each channel sequentially in units of data blocks. as shown in picture 2.

Among them, INDEX is composed of a list of index information blocks. Each index information block corresponds to each audio or video data block in the AVI file. The information block includes: a data block mark (is it a video data block, an audio data block or a filled blank data block, and the audio data block also includes the mark of which audio stream it belongs to), whether it is a key frame mark (that is, the mark of the I frame) , is the combination data block flag (that is, multiple data blocks are combined into one data combination) or single data block (a data block, which can be a frame, or multiple frames), offset dwOffset (each The audio/video data block corresponds to the offset from the file header or the audio/video data header in the file), and the number of bytes corresponding to the audio/video data block.

For video data, one data block generally corresponds to one frame; for VBR audio data, one data block corresponds to one audio frame, and for CBR audio data, there is no absolute correspondence between data blocks and audio frames. There are a few cases in AVI files as follows: several audio data blocks are combined into one combined data block.

Because AVI video files do not contain accurate, evenly distributed time stamp information. For AV synchronization in the non-switching process, it is mainly to read the time scale parameter and rate parameter of each stream from the audio and video stream header information before initializing the decoder, and initialize the decoding rate of the audio and video decoder according to these two types of parameters , thus maintaining AV synchronization. And when playing the AVI file containing one Divx video encoding data and multiple audio data, when switching the audio data to be played, it is necessary to recalculate the precise position of audio and video playback to ensure AV synchronization.

The invention is based on the embedded system, the decoding work is completed by the hardware chip, and the data transmission control is completed by the software, so as to reduce the operating requirements of the embedded system and accelerate the decoding speed. However, when hardware decoding is used, the information required to maintain AV synchronization, such as the position information of data transmission and the position information of data decoding, needs to be transmitted between each unit, and the control and coordination of work is difficult to achieve. There is no such method at present. .

Contents of the invention

The technical problem to be solved by the present invention is to provide an AV synchronization method and device for switching multiple audio streams when an AVI file is played. Keep AV in sync.

In order to solve the above-mentioned technical problems, the present invention provides an AV synchronization method for switching multiple audio streams when an AVI file is played, which is applied to a system including a switching control unit, a data transmission unit, a data processing unit, and a data decoding unit. When the AVI file of multiple audio streams is used, the AVI file header information is saved first. After the user switches the audio stream, the following steps are performed:

(a) After receiving the switching request, the switching control unit sends switching instructions to the data processing unit and the data decoding unit;

(b) the data decoding unit returns the currently decoded video frame number to the switching control unit;

(c) The switching control unit calculates the retransmission position and synchronous playback parameters according to the number of currently decoded video frames and the saved index information and file header information, sends the retransmission position to the data transmission unit, and sends the synchronous playback parameters to the data decoder unit, the synchronous playback parameter is the calculated audio skip number Skip_audio and video skip number Skip_Frame;

(d) The data transmission unit retransmits the audio-video data of the AVI file by the specified retransmission position, and the data processing unit separates the video stream and the audio stream that should be switched to after receiving the audio-video data; the data decoding unit completes video decoding Re-initialize, and re-initialize the audio decoding according to the parameters of the audio stream to be switched to;

(e) After the data decoding unit receives the retransmitted audio and video data, the data decoding unit does not transmit to the Skip_audio bytes or frame numbers and the Skip_Frame video frames of the retransmitted audio and video stream beginning part after decoding The data display unit, after which the decoded audio and video data is transmitted to the data display unit for synchronous playback, wherein the number of bytes is used for the fixed code stream, and the number of frames is used for the variable code stream.

Further, the above audio and video synchronization method can also have the following characteristics:

The data decoding unit continues to decode the buffered video data after receiving the switching instruction, and clears the buffered video data after receiving the parameters Skip_audio and Skip_Frame to re-initialize the video decoding.

Further, the above audio and video synchronization method can also have the following characteristics:

The system is an embedded system, the data decoding unit is realized by hardware, and the switching control unit, data transmission unit and data processing unit are realized by CPU executing corresponding software.

Further, the above audio and video synchronization method can also have the following characteristics: the step (c) switching control unit further completes the calculation through the following steps:

(c1) Add the set frame number B to the frame number A being decoded when the audio data stream is switched to obtain the frame number of the video synchronization point V _C for this switch, and according to the starting position of the audio and video data analyzed and its total Size, determine the starting position of the index information, read the index information and save it;

(c2) from audio-video data index start position accumulative number of video frame, with the video frame number of accumulative closest and the I frame that is less than V _C frame sequence number is the nearest I frame _VI before V _C ;

(c3) find the audio data block A _I corresponding to V _I from the index information, read the offset position of A _I in the AVI file, and take this position as the retransmission position;

(c4) Calculate the frame number difference between _VC and V _I , that is, the video skip number _{Skip_Frame} , and the number of audio bytes or frames Audio_accum_size and A _I The difference between the previously accumulated audio bytes or frame numbers _{Audio_TotalN_AI} is the number of audio skips Skip_audio.

Further, the above-mentioned audio-video synchronization method can also have the following characteristics: in the step (c3), the audio data block _A1 corresponding to _V1 is searched in the following manner:

(c31) calculate the number of audio bytes or the number of frames Audio_TotalN_I that will be switched to that way audio stream before playing _VI , and the formula is:

To fixed stream: Audio_TotalN_I= _VI frame sequence number*(video stream time scale/video stream rate indicator)*CBR audio byte rate

To variable bit stream: Audio_TotalN_I= _VI frame sequence number*(video stream time scale/video stream rate index)*VBR audio frequency sampling rate/1152

(c32) From the starting position of the audio and video data, add up the byte number or frame number of the audio stream data block to be switched to one by one, compare the accumulated value with the Audio_TotalN_I obtained above, if it is less than Audio_TotalN_I, then proceed to the next step Once accumulated, if it is greater than or equal to Audio_TotalN_I, go to the next step;

(c33) If the accumulated value is equal to Audio_TotalN_I, then the next audio data block _of this road audio stream is the audio data block A _I corresponding to VI, and if the accumulated value is greater than Audio_TotalN_I, then this audio data block is the corresponding audio of _VI Data block A _I , while recording the number of audio bytes or frames Audio_TotalN_A _I accumulated before A _I ;

(c34) Find the offset position dwOffset from the index information block corresponding to the audio data block _A1 , which is the retransmission position.

When calculating the number of audio bytes or the frame number Audio_accum_size of that road audio stream that will switch to when playing _VC in the described step (c4), only need to change the frame sequence number of _VI in the formula of step (c31) It can be converted into the frame number of _VC .

Further, the above audio and video synchronization method can also have the following characteristics:

The value range of the number of frames B is: 6 to 12 frames.

Further, the above audio and video synchronization method can also have the following characteristics:

In step (a), after the switch control unit receives the switch request, it also sends a switch command to the data transmission unit, and after the data transmission unit receives the switch command, it immediately stops the transmission of the original audio and video stream; meanwhile, such as AVI If the file contains three or more audio streams, the switching control unit needs to indicate which audio stream to switch to in the instructions sent to the data processing unit and the data decoding unit.

The AV synchronization device for switching multiple audio streams during AVI file playback provided by the invention comprises:

The data transmission unit is used to read the audio and video data of the AVI file and transmit it to the data processing unit, and retransmit the audio and video data of the AVI file by the designated retransmission position after receiving the retransmission instruction;

The data processing unit is used to analyze and save the header information and index information of the AVI file, complete the separation of the audio stream and the video stream and transmit them to the data decoding unit, and separate the video stream from the audio and video data after receiving the switching instruction and the audio stream that should be switched to;

The switching control unit is used to send switching instructions to the data processing unit and the data decoding unit after receiving the switching request, and then calculate the retransmission position and synchronous playback according to the number of currently decoded video frames and the saved index information and file header information Parameter, the retransmission position is sent to the data transmission unit, and the synchronous playback parameter is sent to the data decoding unit, and the synchronous playback parameter is the calculated audio skipping number Skip_audio and video skipping number Skip_Frame;

The data decoding unit includes an audio decoding subunit and a video decoding subunit; after receiving the switching instruction, it returns the number of currently decoded video frames to the switching control unit, and completes the reinitialization of video decoding and the channel to be switched to The parameters of the audio stream are re-initialized for audio decoding; according to the received synchronous playback parameters, the data decoding unit does not decode the Skip_audio bytes or frames and the Skip_Frame video frames at the beginning of the retransmitted audio and video stream. It is transmitted to the data display unit, and then the decoded audio and video data is transmitted to the data display unit for synchronous playback, wherein the number of bytes is used for the fixed code stream, and the number of frames is used for the variable code stream.

Further, the above-mentioned audio-video synchronization device may also have the following characteristics: the switching control unit further includes:

The interface subunit is used to complete the communication between the switching control unit and other units;

The information extraction subunit is used to obtain the AVI file header information and index information from the data processing unit for use by the operation subunit.

The control subunit is used to send switching instructions to the data decoding unit, data processing unit and data decoding unit after receiving the switching request from the user interface unit, and the instructions sent to the data processing unit and decoding unit should carry the audio to be switched to Stream ID, obtain the current decoded video frame sequence number from the data decoding unit and send it to the operation subunit; and after receiving the parameters of the retransmission position, the number of audio skips and the number of video skips of the operation subunit, send a retransmission to the data transmission unit An instruction is transmitted, carrying a retransmission position parameter, and transmitting the parameters of the number of audio skips and the number of video skips to the data decoding unit.

The operation subunit is used to calculate the retransmission position, the number of audio skips Skip_audio and the number of video skips Skip_Frame according to the currently decoded video frame number and AVI file header information and index information, and transmit them to the control subunit.

Further, the above-mentioned audio-video synchronization device can also have the following characteristics:

The data decoding unit continues to decode the buffered video data after receiving the switching instruction, and clears the buffered video data after receiving the parameters Skip_audio and Skip_Frame to re-initialize the video decoding.

Further, the above-mentioned audio-video synchronization device may also have the following characteristics: the operation subunit further includes the following parts:

The synchronization point calculation and index information extraction part, according to the frame number A being decoded when switching the audio data stream plus the set frame number B, obtains the frame number of the video synchronization point V _C of this switch, and according to the analyzed audio and video The starting position of the data and its total size, determine the starting position of the index information, read and save the index information;

The I frame determines the part, from the audio and video data index start position to accumulate the number of video frames, the I frame closest to and less than the V _C frame sequence number is the nearest I frame V _I before the V _C with the accumulated video frame number;

The retransmission position calculation part finds the audio data block A _I corresponding to VI from the index information, reads the offset position _of A _I in the AVI file, and takes this position as the retransmission position;

Skip the number calculation part, calculate the frame number difference between _VC and _VI , that is, the video skip number Skip_Frame, and the number of audio bytes or frames Audio_accum_size that should be played by the audio stream to be switched to when playing _VC The difference between the number of audio bytes or the number of frames Audio_TotalN_A _I accumulated before _AI , that is, the number of audio skips Skip_audio.

Further, the above-mentioned audio-video synchronization device can also have the following characteristics:

When the retransmission position calculation part searches for the corresponding audio data block _{A1 of V1} , it first calculates the number of audio bytes or the number of frames Audio_TotalN_I that should be played by the audio stream that will be switched to before playing _V1 , and the formula is:

To fixed stream: Audio_TotalN_I= _VI frame sequence number*(video stream time scale/video stream rate indicator)*CBR audio byte rate

To variable bit stream: Audio_TotalN_I= _VI frame sequence number*(video stream time scale/video stream rate index)*VBR audio frequency sampling rate/1152

Then from the starting position of the audio and video data, add up the number of bytes or frames of the audio stream data block to be switched to one by one, compare the accumulated value with the Audio_TotalN_I obtained above, if it is less than Audio_TotalN_I, perform the next accumulation , if the accumulated value is _{equal to Audio_TotalN_I, then the next audio data block of this road audio stream is the corresponding audio data block A I of} VI, if the accumulated value is greater than Audio_TotalN_I, then this audio data block is the corresponding audio data block of _VI A _I , while recording the number of audio bytes or frames Audio_TotalN_A _I accumulated before A _I ;

Finally, the offset position dwOffset is found from the index information block corresponding to the audio data block _A1 , and this position is the retransmission position.

When the said skip number calculation part calculates the number of audio bytes or the number of frames Audio_accum_size that the audio stream to be switched to when playing _VC , only the frame sequence number of _VI in the above-mentioned calculation Audio_TotalN_I formula needs to be changed into The frame number of _VC is enough.

Further, the above-mentioned audio-video synchronization device can also have the following characteristics:

The switching control unit is also used to send a switching instruction to the data transmission unit after receiving the switching request; after the data transmission unit receives the switching instruction, immediately stop the transmission of the original audio and video stream; the switching control unit When the played AVI file contains three or more audio stream data, it is necessary to indicate which audio stream to switch to in the instructions sent to the data processing unit and the data decoding unit.

Further, the above-mentioned audio-video synchronization device can also have the following characteristics:

The device is applied to an embedded system, the data decoding unit is realized by hardware, and the switching control unit, data transmission unit and data processing unit are realized by CPU executing corresponding software.

The invention guarantees to play AVI files containing one channel of Divx video encoding data and multiple channels of audio data, and maintain AV synchronization when switching different audio stream data. Furthermore, the continuity of video playback can be maintained as much as possible. It can be used especially for embedded systems with low configuration conditions to realize AV synchronization during audio data switching of AVI files, thereby reducing the requirement for the main frequency of the system, and can realize original/accompaniment switching of Karaoke (Karaoke) and other functions.

Description of drawings

Figure 1 is a schematic diagram of the structure of an AVI file.

FIG. 2 is a schematic structural diagram of the audio and video data part in FIG. 1 .

Fig. 3 is a structural block diagram of the system of the embodiment of the present invention.

Fig. 4 is a flow chart of processing when the original accompaniment is switched in the method of the embodiment of the present invention.

FIG. 5 and FIG. 6 are schematic diagrams of calculating parameters of a CBR audio stream and a VBR audio stream, respectively.

Detailed ways

The present invention is to play in the AVI file process that contains one Divx video encoding data, multi-channel audio data, the method and the system that keep AV synchronism when switching audio data, the original accompaniment singing is switched when playing Karaoke system as example below, in conjunction with accompanying drawing and implementation Examples illustrate the present invention in detail.

Since it takes a period of time to calculate between getting the switching instruction and executing the multi-channel audio switching, in order to avoid the video image from stilling, the video will continue to play for a period of time. In the case of video playback, in order to ensure the synchronization of audio and video after switching, a point where audio and video are played simultaneously is selected, i.e. video synchronization point (i.e. video frame) C and its corresponding audio data block. This video sync point should be somewhere after the video frame decoded at the time of switching.

In addition, according to the decoding requirements, the decoding of the video must start from the I frame, and the audio can start from any position. If the video is decoded directly from the C point, mosaic phenomenon will appear. Therefore, when determining the start position of re-decoding after switching, it is necessary to find the audio data block corresponding to (referring to _{synchronous playback) of the nearest I frame (also represented by VI) before the video synchronization point C (also represented by V C )} (following Represented by _AI ), retransmit and decode the AVI file data from the offset position Goto_offset of the audio data block in the AVI file. For the video frame before _VC and the audio-video data before the audio data block (hereinafter referred to as A _C ) corresponding to VC, _only decode but not play.

The retransmission or retransmission in this article refers to breaking the order of the first transmission and starting a new second transmission. The starting position of this transmission is no longer the original file header.

Fig. 3 is a structural block diagram of the system of the embodiment of the present invention, as shown in the figure, including a data transmission unit, a data processing unit, a switching control unit, a data decoding unit, a data display unit and a user interface unit, wherein:

The data transmission unit is used to transmit the audio and video data of the read AVI file to the data processing unit, and the data transmission unit stops data transmission after receiving the switching instruction of the switching control unit, and after receiving the retransmission instruction of the unit, press The restart position indicated in the command restarts the transmission of audio and video data. The video data adopts encoding formats such as Divx3.11, Divx4.x, Divx5.x, etc., and the audio data adopts encoding formats such as MP1, MP2, and MP3.

The data processing unit is used to receive the audio and video data transmitted by the data transmission unit, save after parsing the AVI file header information, Index information, and audio and video format information, and separate the audio and video data to be played and transmit it to the data decoding unit. For the retransmitted data after the switching command, it is necessary to separate the video stream and the audio stream to which it should be switched from the audio and video data;

The switching control unit further includes:

The information extraction subunit is used to obtain the saved AVI file header information and Index information from the data processing unit for use by the operation subunit;

The interface subunit is used to complete the communication between the switching control unit and other units;

The control subunit is used to send switching instructions to the data transmission unit, data processing unit and data decoding unit after receiving the switching request from the user interface unit, and the instructions sent to the data processing unit and the decoding unit indicate which switch to switch to One audio stream (the indication can be realized by carrying the audio stream identifier to be switched to, of course, if there are only two audio streams, it can not be indicated), and the currently decoded video frame number is obtained from the data decoding unit and sent to the operation subunit; and After receiving the retransmission position, audio skip number and video skip number parameters of the operator unit, send a retransmission instruction to the data transmission unit, including the retransmission position parameter, and set the audio skip number and video skip number parameters transmitted to the data decoding unit;

The operation subunit is used to calculate the retransmission position, the number of audio skips and the number of video skips according to the currently decoded video frame number and the AVI file header information and Index information, and transmit them to the control subunit. This subunit can be divided into synchronization point calculation and index information extraction part, I frame determination part, retransmission position calculation part and skip number calculation part, and the functions of each part will be introduced later.

The data decoding unit includes an audio decoding subunit and a video decoding subunit, which are respectively used to decode the audio and video data transmitted by the data processing unit, wherein:

The video decoding subunit is also used to return the video frame number of the current video decoding to it after receiving the switching instruction from the switching control unit, and continue to decode the cached data; after receiving the number of video skips, clear the cache and re-initialize , the number of video frames at the beginning of the retransmitted video stream is only decoded, and not transmitted to the data display unit, and the subsequent video frames are decoded and output to the data display unit;

The audio decoding subunit stops decoding audio data after receiving the switching instruction from the switching control unit, and re-initializes the audio stream parameters to be switched to; after receiving the number of audio skips, it starts to retransmit the audio stream Part of the number of audio frames/bytes ("/" means "or" relationship, the audio data of CBR is the number of bytes, and the audio data of VBR is the audio frame) is only decoded, not transmitted to the data display unit, and then The audio data is decoded and output to the data display unit.

The data display unit is used to display and play the decoded audio and video data, the video can be output on TV/VGA equipment, and the audio can be output on TV equipment.

The user interface unit is an interface operated by the user, through which the user can send a switching request, wherein the switching request should include the audio stream identifier to be switched to when there are more than three audio streams, that is, indicate which audio stream to switch to flow.

The process performed when the original accompaniment is not switched includes the following steps:

Step 1, STB is powered on, automatically runs the Karaoke playback system, and the user controls the system through the playback interface;

Step 2, when the user selects to play the song of the AVI file format, start to transmit data from the file header in the AVI file, as the transmission data received is the AVI file header, perform step 3, if it is audio and video data, perform step 5;

Step 3, parsing the AVI file header, saving the information contained therein;

The AVI file header information parsed and saved includes: number of audio and video streams to confirm how many audio and video streams there are in this file; video stream time scale and video stream rate index; VBR data alignment value (fixed value is 1152,) , the sampling rate of the VBR audio stream or the byte rate of the CBR audio stream; the starting position of the audio and video data Movie_Offset and the overall size of the audio and video data Movie_Size; the width and height of the video image, that is, the image resolution; the video decoder type; Decoder palette; video decoder decoding rate (represented by the two parameters of video stream time scale and video stream rate index); each audio stream audio decoder type; each audio stream audio decoder code stream type (fixed code rate or non-fixed code rate); the decoding rate of the audio decoder for each audio stream; the decoding start time of the audio and video decoders.

Step 4, carry out the initial setting operation of decoding according to the information that obtains from AVI file header;

It mainly includes the following operations: set the width and height of the video image, that is, the image resolution; set the video decoder type; set the video decoder palette; set the video decoder decoding rate; set the audio decoder type; set the audio decoding Coder stream type (fixed code rate or non-fixed code rate); set the decoding rate of the audio decoder; set the decoding start time of the audio and video decoders.

Among them, the AVI file only contains one channel of video data, but can contain multiple channels of audio data, so when initializing the audio decoder, use the default audio stream parameters to perform the initialization operation.

Step 5, separating the audio and video data streams to be played;

Step 6: Decode the separated audio and video data, and output the decoded data to a display device and a speaker through the AV interface for playback.

The flow is just a normal processing flow.

As shown in Figure 4, in the process of playing the AVI file that contains multiple audio streams, the processing flow for the user to switch the original accompaniment includes the following steps:

Step 110, the user switches the original accompaniment;

Step 120, after the switching control unit receives the switching request, it sends a switching instruction to the data transmission unit, the data processing unit, and the data decoding unit, wherein the instructions sent to the data processing unit and the data decoding unit need to carry the audio stream identifier to be switched to;

Step 130, the data transmission unit stops the transmission of audio and video streams, and the data decoding unit re-performs the initialization setting of audio decoding according to the parameters of the audio stream to be switched to, returns the current decoded video frame number to the switching control unit, and at the same time Continue to decode the cached video data;

The audio decoding initialization setting in this step includes: setting the audio decoder type; setting the audio decoder code stream type (fixed frame rate or non-fixed frame rate); setting the audio decoder decoding rate; setting the audio and video decoder start decoding time . The parameters required for these settings are originally obtained from the AVI file header.

Step 140, the switching control unit calculates parameters such as the retransmission position, the number of audio skips Skip_audio and the number of video skips Skip_Frame according to the number of currently decoded video frames and the stored INDEX information and file header information, and sends a retransmission to the data transmission unit The instruction includes a retransmission position parameter, and transmits the parameters Skip_audio and Skip_Frame to the data decoding unit;

Step 150, the data transmission unit retransmits the audio and video data of the AVI file according to the specified retransmission position, and the video stream and the audio stream that should be switched to are separated from the audio and video data by the data processing unit; the data decoding unit first clears the buffered Video data, reinitialized;

Step 160, the data decoding unit does not transmit to the data display unit after decoding the Skip_audio audio bytes/frames and the Skip_Frame video frames at the beginning of the retransmitted audio and video stream, and then transmits the decoded audio and video data to the data display unit display units to play in sync.

So far the A/V synchronization process is completed. It should be noted that the data decoding unit (the audio decoding sub-unit in it) can also continue to decode the audio data after receiving the switching instruction, stop decoding after receiving the number of audio skips, and complete the initialization of the audio stream to be switched to .

How to perform the calculation in step 140 will be described in detail below, please refer to the schematic diagrams of parameter calculation of the CBR (Constant Bit Stream) audio stream and the VBR (Variable Bit Stream) audio stream shown in FIG. 5 and FIG. 6 . "a" or "v" in the figure corresponds to a data block. This step is further divided into the following steps:

Step 1401, according to the video frame number A being decoded when switching the audio data stream plus the set frame number B, obtain the frame sequence number of the video synchronization point V _C of this switch, and simultaneously according to the starting position of the audio and video data analyzed and its The total size determines the starting position of the INDEX information, reads the INDEX information and saves it;

The frame number of _VC is the number of video frames accumulated from the start position of audio and video data to this frame. The number of frames B is an empirical value or calculated. If B is too large, the playback will skip. If it is too small, there will be repetitions. The optimal value range is: 6 to 12 frames. This operation is completed by the synchronization point calculation and index information extraction part of the operation subunit.

Step 1402, first determine the frame sequence number of the latest I frame V _I before V _C ;

In this embodiment, from the starting position of the audio and video data index, respectively accumulate the number of video frames, all the audio data blocks of this channel before each audio data block of the audio data stream to be switched to, the number of bytes/frames, and the number of each channel The offset position of each audio data block of the audio stream in the AVI file; in this way, the frame number of each I frame (that is, the number of video frames accumulated to the I frame) can be obtained at one time, and each audio stream before the I frame Information such as the number of data blocks, the number of bytes/frames, the offset position of the audio data block corresponding to the I frame in the AVI file, etc. Then the I frame whose frame number is closest to and smaller than the V _C frame number is V _I .

Since there is no serial number of audio data block or video data block in the INDEX information, the number of video frames can be accumulated from the beginning, and when it is equal to the frame serial number of _VC , _VC is found, and then the latest I frame before it can be determined. Or, also can set up the collection of each I frame frame sequence number in the accumulation process, compare directly with the frame sequence number of _VC , wherein the I frame of the frame sequence number closest to and less than or equal to the frame sequence number of _VC is the I to be looked for frame. In addition, since the number of frames between each I frame is fixed, as long as the sequence number of the first I frame is known (such as reading INDEX information or a fixed value), the frame sequence number of each I frame can be directly calculated , by the same method can be compared to get the frame number of _VI .

This operation is completed by the I frame determination part of the operation subunit.

Step 1403, find the audio data block A _I corresponding to V _I from the INDEX information, read the offset position of A _I in the AVI file, and take this position as the retransmission position Goto_offset;

In INDEX, the audio data block A _I corresponding to V _I is searched in the following way:

1) Calculate the number of audio bytes or the number of frames Audio_TotalN_I that the audio stream that will be switched to before playing _VI should be played, and what calculates for the CBR (fixed code stream) audio stream is the number of audio data bytes, for VBR (can be Transcode stream) audio stream calculates the number of audio data frames, the formula is:

To CBR: Audio_TotalN_I= _VI 's frame sequence number*(video stream time scale/video stream rate indicator)*CBR audio byte rate

To VBR: Audio_TotalN_I= _VI 's frame sequence number*(video stream time scale/video stream rate index)*VBR audio frequency sampling rate/1152

The audio data block and video data block that should be played synchronously are not always adjacent, so the calculation above is based on the relevant parameters in the AVI file header.

2) From the starting position of the audio and video data, accumulate the number of bytes/frames of the audio stream data block to be switched to one by one, compare the accumulated value with the Audio_TotalN_I obtained above, if it is less than Audio_TotalN_I, proceed to the next time Accumulate, if it is greater than or equal to Audio_TotalN_I, go to the next step;

3) If the accumulated value is equal to Audio_TotalN_I, then the next audio data block of the audio stream is the corresponding audio data block A _I of _VI , and if the accumulated value is greater than Audio_TotalN_I, then the audio data block is the corresponding audio data of _VI Block A _I , record the number of audio bytes/frames Audio_TotalN_A _I accumulated before A _I at the same time;

4) Find the offset position dwOffset from the index information block corresponding to the audio data block _A1 , which is the retransmission position Goto_offset.

In another embodiment, after the INDEX information is read, the INDEX information can be traversed first, and the number of bytes or frames corresponding to each audio data block of the audio stream to be switched to and its offset position can be recorded. In this way, after calculating Audio_TotalN_I, the retransmission position Goto_offset can be directly compared.

This operation is completed by the retransmission position calculation part of the operation subunit.

Step 1404, calculate the frame number difference between _VC and _VI , that is, the number of video skips _{Skip_Frame} , and the number of audio bytes/frames Audio_accum_size and A The difference between the number of audio bytes/frames Audio_TotalN_A _I accumulated before _I , that is, the number of audio skips Skip_audio;

Skip_Frame is equal to the difference between the frame numbers of _VC and _VI . The calculation method of Audio_accum_size is the same as that of Audio_TotalN_I, the formula is as follows:

For CBR: Audio_accum_size=V _C frame number*(video stream time scale/video stream rate index)*CBR audio byte rate

For VBR: Audio_accum_size=V _C frame number*(video stream time scale/video stream rate index)*VBR audio sampling rate/1152

Since the CBR audio stream calculates the number of audio data bytes, the VBR audio stream calculates the number of audio data frames, so Audio_accum_size is in the combined data block or one data block of CBR (the audio data block of CBR does not correspond to the audio frame) In can be in the middle of a frame, in VBR combined data block is in between frames.

This operation is completed by the skip number calculation part of the operation subunit.

To sum up, the present invention can use a combination of software and hardware to play an AVI file containing one Divx video encoding data and multiple audio data, and keep AV synchronization when switching between different audio stream data. It ensures the AV synchronization when switching the audio data of the AVI file under the condition of low configuration in the embedded system, thereby reducing the requirement for the main frequency of the system.

In another embodiment of the present invention, if it is only necessary to achieve synchronization without considering the continuity of playback, operations in the process and units in the device related to the number of audio skips and video skips can also be omitted.

Claims (14)

1. an AV synchronous method for switching multiple audio streams during AVI file playback, applied to the system comprising switching control unit, data transmission unit, data processing unit and data decoding unit, when playing the AVI file containing multiple audio streams , first save the AVI file header information, when the user switches the audio stream, it is characterized in that the following steps are performed: (a) After receiving the switching request, the switching control unit sends switching instructions to the data processing unit and the data decoding unit; (b) the data decoding unit returns the currently decoded video frame number to the switching control unit; (c) The switching control unit calculates the retransmission position and synchronous playback parameters according to the number of currently decoded video frames and the saved index information and file header information, sends the retransmission position to the data transmission unit, and sends the synchronous playback parameters to the data decoder unit, the synchronous playback parameter is the calculated audio skip number Skip_audio and video skip number Skip_Frame; (d) The data transmission unit retransmits the audio-video data of the AVI file by the specified retransmission position, and the data processing unit separates the video stream and the audio stream that should be switched to after receiving the audio-video data; the data decoding unit completes video decoding Re-initialize, and re-initialize the audio decoding according to the parameters of the audio stream to be switched to; (e) After the data decoding unit receives the retransmitted audio and video data, the data decoding unit does not transmit to the Skip_audio bytes or frame numbers and the Skip_Frame video frames of the retransmitted audio and video stream beginning part after decoding The data display unit, after which the decoded audio and video data is transmitted to the data display unit for synchronous playback, wherein the number of bytes is used for the fixed code stream, and the number of frames is used for the variable code stream. 2. AV synchronization method as claimed in claim 1, characterized in that: The data decoding unit continues to decode the buffered video data after receiving the switching instruction, and clears the buffered video data after receiving the parameters Skip_audio and Skip_Frame to re-initialize the video decoding. 3. AV synchronization method as claimed in claim 1, characterized in that: The system is an embedded system, the data decoding unit is realized by hardware, and the switching control unit, data transmission unit and data processing unit are realized by CPU executing corresponding software. 4. AV synchronization method as claimed in claim 1, is characterized in that: described step (c) switching control unit further completes calculation by the following steps: (c1) Add the set frame number B to the frame number A being decoded when the audio data stream is switched to obtain the frame number of the video synchronization point V _C for this switch, and according to the starting position of the audio and video data analyzed and its total Size, determine the starting position of the index information, read the index information and save it; (c2) from audio-video data index start position accumulative number of video frame, with the video frame number of accumulative closest and the I frame that is less than V _C frame sequence number is the nearest I frame _VI before V _C ; (c3) find the audio data block A _I corresponding to V _I from the index information, read the offset position of A _I in the AVI file, and take this position as the retransmission position; (c4) Calculate the frame number difference between _VC and V _I , that is, the video skip number _{Skip_Frame} , and the number of audio bytes or frames Audio_accum_size and A _I The difference between the previously accumulated audio bytes or frame numbers _{Audio_TotalN_AI} is the number of audio skips Skip_audio. 5. AV synchronous method as claimed in claim 4, is characterized in that: in described step (c3), the audio data block _AI corresponding to _VI is searched in the following manner: (c31) calculate the number of audio bytes or the number of frames Audio_TotalN_I that will be switched to that way audio stream before playing _VI , and the formula is: To fixed stream: Audio_TotalN_I= _VI frame sequence number*(video stream time scale/video stream rate indicator)*CBR audio byte rate To variable bit stream: Audio_TotalN_I= _VI frame sequence number*(video stream time scale/video stream rate index)*VBR audio frequency sampling rate/1152 (c32) From the starting position of the audio and video data, add up the byte number or frame number of the audio stream data block to be switched to one by one, compare the accumulated value with the Audio_TotalN_I obtained above, if it is less than Audio_TotalN_I, then proceed to the next step Once accumulated, if it is greater than or equal to Audio_TotalN_I, go to the next step; (c33) If the accumulated value is equal to Audio_TotalN_I, then the next audio data block _of this road audio stream is the audio data block A _I corresponding to VI, and if the accumulated value is greater than Audio_TotalN_I, then this audio data block is the corresponding audio of _VI Data block A _I , while recording the number of audio bytes or frames Audio_TotalN_A _I accumulated before A _I ; (c34) find the offset position dwOffset from the index information block corresponding to the audio data block _A1 , this position is the retransmission position, When calculating the number of audio bytes or the frame number Audio_accum_size of that road audio stream that will switch to when playing _VC in the described step (c4), only need to change the frame sequence number of _VI in the formula of step (c31) It can be converted into the frame number of _VC . 6. AV synchronization method as claimed in claim 3, characterized in that: The value range of the number of frames B is: 6 to 12 frames. 7. AV synchronization method as claimed in claim 1, characterized in that: In step (a), after the switch control unit receives the switch request, it also sends a switch command to the data transmission unit, and after the data transmission unit receives the switch command, it immediately stops the transmission of the original audio and video stream; meanwhile, such as AVI If the file contains three or more audio streams, the switching control unit needs to indicate which audio stream to switch to in the instructions sent to the data processing unit and the data decoding unit. 8. an AV synchronization device for switching multiple audio streams during AVI file playback, characterized in that, comprising: The data transmission unit is used to read the audio and video data of the AVI file and transmit it to the data processing unit, and retransmit the audio and video data of the AVI file by the designated retransmission position after receiving the retransmission instruction; The data processing unit is used to analyze and save the AVI file header information and index information, complete the separation of the audio stream and the video stream and transmit them to the data decoding unit, and separate the video stream from the audio and video data after receiving the switching instruction and the audio stream that should be switched to; The switching control unit is used to send switching instructions to the data processing unit and the data decoding unit after receiving the switching request, and then calculate the retransmission position and synchronous playback according to the number of currently decoded video frames and the saved index information and file header information Parameter, the retransmission position is sent to the data transmission unit, and the synchronous playback parameter is sent to the data decoding unit, and the synchronous playback parameter is the calculated audio skipping number Skip_audio and video skipping number Skip_Frame; The data decoding unit includes an audio decoding subunit and a video decoding subunit; after receiving the switching instruction, it returns the number of currently decoded video frames to the switching control unit, and completes the reinitialization of video decoding and the channel to be switched to The parameters of the audio stream are re-initialized for audio decoding; according to the received synchronous playback parameters, the data decoding unit does not decode the Skip_audio bytes or frames and the Skip_Frame video frames at the beginning of the retransmitted audio and video stream. It is transmitted to the data display unit, and then the decoded audio and video data is transmitted to the data display unit for synchronous playback, wherein the number of bytes is used for the fixed code stream, and the number of frames is used for the variable code stream. 9. The AV synchronization device according to claim 8, characterized in that: the switching control unit further comprises: The interface subunit is used to complete the communication between the switching control unit and other units; The information extraction subunit is used to obtain the AVI file header information and index information from the data processing unit for use by the operation subunit The control subunit is used to send switching instructions to the data decoding unit, data processing unit and data decoding unit after receiving the switching request from the user interface unit, and the instructions sent to the data processing unit and decoding unit should carry the audio to be switched to Stream ID, obtain the current decoded video frame sequence number from the data decoding unit and send it to the operation subunit; and after receiving the parameters of the retransmission position, the number of audio skips and the number of video skips of the operation subunit, send a retransmission to the data transmission unit Passing instructions, carrying retransmission position parameters, and transmitting the parameters of the number of audio skips and the number of video skips to the data decoding unit; The operation subunit is used to calculate the retransmission position, the number of audio skips Skip_audio and the number of video skips Skip_Frame according to the currently decoded video frame number and AVI file header information and index information, and transmit them to the control subunit. 10. The AV synchronization device according to claim 8, characterized in that: The data decoding unit continues to decode the buffered video data after receiving the switching instruction, and clears the buffered video data after receiving the parameters Skip_audio and Skip_Frame to re-initialize the video decoding. 11. The AV synchronization device according to claim 9, characterized in that: the operation subunit further comprises the following parts: The synchronization point calculation and index information extraction part, according to the frame number A being decoded when switching the audio data stream plus the set frame number B, obtains the frame number of the video synchronization point V _C of this switch, and according to the analyzed audio and video The starting position of the data and its total size, determine the starting position of the index information, read and save the index information; The I frame determines the part, from the audio and video data index start position to accumulate the number of video frames, the I frame closest to and less than the V _C frame sequence number is the nearest I frame V _I before the V _C with the accumulated video frame number; The retransmission position calculation part finds the audio data block A _I corresponding to VI from the index information, reads the offset position _of A _I in the AVI file, and takes this position as the retransmission position; Skip the number calculation part, calculate the frame number difference between _VC and _VI , that is, the video skip number Skip_Frame, and the number of audio bytes or frames Audio_accum_size that should be played by the audio stream to be switched to when playing _VC The difference between the number of audio bytes or the number of frames Audio_TotalN_A _I accumulated before _AI , that is, the number of audio skips Skip_audio. 12. The AV synchronization device according to claim 11, characterized in that: When the retransmission position calculation part searches for the corresponding audio data block _{A1 of V1} , it first calculates the number of audio bytes or the number of frames Audio_TotalN_I that should be played by the audio stream that will be switched to before playing _V1 , and the formula is: To fixed stream: Audio_TotalN_I= _VI frame sequence number*(video stream time scale/video stream rate indicator)*CBR audio byte rate To variable bit stream: Audio_TotalN_I= _VI frame sequence number*(video stream time scale/video stream rate index)*VBR audio frequency sampling rate/1152 Then from the starting position of the audio and video data, add up the number of bytes or frames of the audio stream data block to be switched to one by one, compare the accumulated value with the Audio_TotalN_I obtained above, if it is less than Audio_TotalN_I, perform the next accumulation , if the accumulated value is _{equal to Audio_TotalN_I, then the next audio data block of this road audio stream is the corresponding audio data block A I of} VI, if the accumulated value is greater than Audio_TotalN_I, then this audio data block is the corresponding audio data block of _VI A _I , while recording the number of audio bytes or frames Audio_TotalN_A _I accumulated before A _I ; Finally, find the offset position dwOffset from the index information block corresponding to the audio data block _A1 , which is the retransmission position, When the said skip number calculation part calculates the number of audio bytes or the number of frames Audio_accum_size that the audio stream to be switched to when playing _VC , only the frame sequence number of _VI in the above-mentioned calculation Audio_TotalN_I formula needs to be changed into The frame number of _VC is enough. 13. The AV synchronization device according to claim 8, characterized in that: The switching control unit is also used to send a switching instruction to the data transmission unit after receiving the switching request; after the data transmission unit receives the switching instruction, immediately stop the transmission of the original audio and video stream; the switching control unit When the played AVI file contains three or more audio stream data, it is necessary to indicate which audio stream to switch to in the instructions sent to the data processing unit and the data decoding unit. 14. The AV synchronization device according to claim 8, characterized in that: The device is applied to an embedded system, the data decoding unit is realized by hardware, and the switching control unit, data transmission unit and data processing unit are realized by CPU executing corresponding software.

Images