JP2006238003A - Demultiplexer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reproduce a moving image display in a section in which data is lost or to silence sound when playing an MP4 file storing a stream in which data is lost due to a reception error such as packet loss. There was a problem.
In step 1104, when the playback time length of a sample exceeds a predetermined value, it is determined that the data is discontinuous, and skipping playback of a section in which both the moving image and the sound are discontinuous is performed. , It is possible to skip the playback of the section where the data is missing.
[Selection] Figure 10

Description

  The present invention relates to a demultiplexing apparatus that separates, decodes and reproduces multiplexed data of an encoded stream such as a moving image and audio.

  In recent years, devices and services that handle encoded multimedia data such as moving images and voices have become widespread as storage media and communication networks have increased in capacity and transmission technology has advanced. For example, in the broadcasting field, broadcasting of digitally encoded media data has started instead of conventional analog broadcasting. The current digital broadcasting is intended only for fixed reception, but in the future broadcasting for mobiles such as mobile phones is also planned. Also in the communication field, an environment for handling multimedia data on both fixed terminals and mobile terminals has been established, such as the launch of a video distribution service for third-generation mobile phones. In view of these backgrounds, content data received via broadcast or the Internet is recorded on a memory card such as an SD (Secure Digital) card or an optical disk such as a DVD-RAM (Digital Versatile Disk-Rewritable), and it is used between devices. It is expected that usage methods such as sharing content data will spread.

  When media data is broadcast, stored, or distributed via a network, header information and media data necessary for reproducing the media data are multiplexed. In multiplexing, standard multiplexing schemes are standardized for storage devices such as broadcast and DVD, and for mobile units. First, in digital broadcasting and DVD, the MPEG-2 (Moving Picture Expert Group) system standard standardized by ISO / IEC JTC1 / SC29 / WG11 (International Standardization Organization / International Engineering Consortium) is used. In mobile terminals, TS26.234 (Transparent end), which was established as a wireless video distribution standard by the 3GPP (Third Generation Partnership Project), an international standardization organization aimed at standardizing third-generation mobile communication systems. -to-end packet switched streaming service) adopts the MP4 file format standardized by ISO / IEC JTC1 / SC29 / WG 11.

  In addition, since MPEG-4 AVC (Advanced Video Coding) has been standardized as a successor to the currently popular MPEG-2 Visual and MPEG-4 Visual as a video encoding system, MPEG-4 AVC will be used in the future. It is expected that the encoded moving image data is multiplexed, broadcasted, stored, or distributed according to the MPEG-2 system standard or the MP4 file format (hereinafter referred to as MP4).

  The outline of the encoded data multiplexing method in the MPEG-2 system and MP4 will be described below. In the MPEG-2 system and MP4, since an access unit (AU) is used as a basic unit when handling encoded data, the structure of the AU will be described first. The AU is a unit including encoded data for one picture of moving picture or one frame of audio, and the AU data in MPEG-4 AVC has a structure shown in FIG. In MPEG-4 AVC, in addition to data essential for picture decoding, auxiliary information called SEI (Supplemental Enhancement Information) that is not essential for decoding, AU boundary information, and the like can be included in AU data. Are all stored in a NAL (Network Adaptation Layer) unit. As shown in FIG. 1A, the NAL unit is composed of a header and a payload, the header size is 1 byte, and a field indicating the type of data stored in the payload (hereinafter referred to as the NAL unit type). included. The NAL unit type has a value defined for each type of data such as slice and SEI, and refers to the NAL unit type when acquiring the type of data stored in the NAL unit. As shown in FIGS. 1B and 1C, in addition to slice data for one picture, the AU stores NAL units such as header information and SEI. The NAL unit stores NAL unit data. Since there is no information for identifying the boundary, the boundary information can be added to the head of each NAL unit when storing the AU. The boundary information includes a method of adding a start code prefix indicated by 3 bytes of 0x000001 as shown in FIG. 1B (hereinafter referred to as a byte stream format), and the size of the NAL unit as shown in FIG. There are two types of methods (hereinafter referred to as NAL size format). Note that it is stipulated that one or more zero_bytes (one byte whose value is 0x00) is added before the start code prefix for the head NAL unit of the AU and a NAL unit having a specific NAL unit type value. ing. The MPEG-2 system uses the byte stream format, and MP4 uses the NAL size format. Next, the slice and header information will be described in detail. The slices are divided into two types: IDR (Instantaneous Decoder Refresh) slices and other slices. The IDR slice is slice data that is intra-coded, and header information such as SPS (Sequence Parameter Set) described later can be switched only in the IDR slice. When an IDR slice is included in a picture, all other slices in the same picture are also IDR slices. Therefore, an AU including an IDR slice is hereinafter referred to as an IDR AU. A unit composed of AUs from an IDR AU to an AU immediately before the next IDR AU is called a sequence, and when decoding AU slice data, only the AU in the sequence is referred to. Can be accessed randomly. Next, there are two types of header information, SPS and PPS (Picture Parameter Set), SPS is fixed header information in sequence units, and PPS is header information that can be switched in picture units. There can be a plurality of SPS and PPS, and each SPS or PPS is distinguished by an index number. Further, one SPS or PPS is stored in one NAL unit. The index numbers of the SPS and the PPS referred to by each picture are obtained as follows. First, the PPS index number referenced by the picture is indicated in the header portion of the slice data. Next, since the PPS indicates the index number of the SPS referred to by the PPS, the index number of the SPS referenced by the picture is obtained by analyzing the PPS referenced by the picture.

  Next, a method for multiplexing AU data by the MPEG-2 system in broadcasting will be described. In the MPEG-2 system, first, the encoded data is multiplexed into a PES (Packetized Elementary Stream) packet, and the PES packet is further multiplexed into a TS (Transport Stream) packet. FIGS. 2A and 2B show the structures of the PES packet and the TS packet, respectively. Access unit (AU) data is stored in the payload of the PES packet. (1) to (3) in FIG. 2 (a) show an example of storing AU data in the payload of the PES packet. As shown in (1) and (2), one or more AUs are grouped together. The AU data may be divided and stored as shown in (3). Further, the stuffing data can be included in the payload separately from the AU data. The header of the PES packet starts with a start code prefix of 3 bytes of 0x000001 and a 4-byte start code composed of a 1-byte stream ID. The stream ID is an identification number indicating the type of encoded data included in the payload data of the PES packet. In MPEG-4AVC, the stream ID can take any value from 0xE0 to 0xEF. The header can store the decoding time and display time of the first AU starting in the payload, but these time information is not necessarily stored in all PES packets, and PES in which time information is not stored is stored. There are also packets. When the time information of the AU whose decoding time or display time is not indicated by the header of the PES packet is required, the AU data is analyzed, and the difference value between the decoding time and the display time from the previous AU is acquired. The start position of the PES packet is detected by searching for a 4-byte start code in the payload data of the TS packet. On the other hand, as shown in FIG. 2B, the data of the PES packet is divided and stored in the payload of the TS packet. The TS packet is a fixed-length packet having a size of 188 bytes, and includes a 4-byte header, an adaptation field, and payload data. Note that the adaptation field exists only when a specific flag in the header is set. The header includes an identification number called PID indicating the type of data transmitted by the TS packet and a counter called continuity_counter. The continuity_counter is a 4-bit field. In a TS packet with the same PID, the continuity_counter increases by 1 in the order of transmission and circulates when the maximum value is reached. The correspondence between the PID of the TS packet and the type of data transmitted by the TS packet is provided by program information separately transmitted by the TS packet. For this reason, when a TS packet is received, first, the PID of the TS packet is acquired, and the packet is distributed according to the value of the PID. For example, when the program information acquired at the start of reception indicates that MPEG-4 AVC data is transmitted by a TS packet with a PID of 32, by acquiring the TS packet with a PID of 32, MPEG-4 AVC AU data can be acquired. Here, when a gap occurs in the continuity_counter value of the received TS packet, it indicates that a packet loss has occurred in the transmission path. Further, when separating AU data from TS packets, PES packets are separated from payload data of TS packets, and AU data is separated from the separated PES packets.

  Finally, a method for multiplexing AU data in MP4 will be described. In MP4, header information and media data in units of samples are managed in units of objects called boxes. Here, a sample is a basic unit for handling media data in MP4, and one sample corresponds to 1 AU. Each sample is assigned a sample number so as to be in ascending order in decoding time order, and the sample number increases by one for each sample. FIG. 3A shows the box structure, which is composed of the following fields.

size: Size of the entire Box including the size field
type: A box identifier, usually represented by four alphabetic characters. The field length is 4 bytes, and when a Box is searched in the MP4 file, it is determined by determining whether or not the continuous 4 bytes of data matches the identifier of the type field.

version: Box version number
flags: Flag information set for each box Data: Header information and media data are stored.

  In addition, since version and flags are not essential, these fields do not exist depending on the Box. Hereinafter, the identifier of the type field is used for reference to Box. For example, a Box whose type is 'moov' is called moov. The Box structure in the MP4 file is shown in FIG. The MP4 file is composed of ftyp, moov, mdat, or moof, and ftyp is arranged at the head of the file. ftyp includes information for identifying the MP4 file, and media data is stored in mdat. Each media data included in mdat is called a track, and each track is identified by a track ID. Next, header information about samples included in each track of mdat is stored in moov. In moov, as shown in FIG. 4A, Boxes are arranged hierarchically, and header information is stored in separate traks for each media track such as audio and moving images. Even in trak, Boxes are arranged hierarchically, and sample size, decoding time, display start time, information on randomly accessible samples, etc. are stored in each Box in stbl (FIG. 4 (b)). . Randomly accessible samples are called sync samples, and a list of sample numbers of sync samples is indicated by stss in stbl. In the above description, the header information of all the samples in the track is stored in moov. However, the track can be divided and fragmented, and header information can be stored in fragment units. The header information for the unit into which the track is divided is indicated by moof. FIG. 5 shows an example of the structure of a fragmented MP4 file. The header information of the sample in mdat # 1 is stored in moof # 1.

  Here, consider a case in which broadcast data received by a mobile terminal is transmitted from a mobile phone as an attachment to an e-mail. 3GPP stipulates that media data should be multiplexed with MP4 when attaching moving images or voices in e-mails, and a multiplexing method when attaching received broadcast data to e-mails. Need to be converted from TS to MP4. Note that in terrestrial digital broadcasting (hereinafter referred to as 1-segment broadcasting) for mobile use standardized in ARIB (Association of Radio Industries and Businesses) and implemented in Japan, MPEG is used as a moving image encoding method. -4 MPEG-2 AAC is used as an encoding method for AVC and audio.

  FIG. 6 shows a method for storing decoding time information when receiving broadcast data including packet loss and multiplexing moving picture and audio data in the broadcast data into MP4. In this example, as shown in FIG. 6A, it is assumed that the 52nd to 100th AUs of a moving image cannot be received due to packet loss. At this time, MP4 stores AU1 to AU51 and AU101 and later as samples. Therefore, as shown in FIG. 6B, the 51st sample stores picture data of AU101. When the decoding time information for these samples is stored in the moov, a table of stts that is a box describing the difference value of the decoding time of each sample is as shown in FIG. From sample 1 to sample 50, the difference value of the decoding time is all 100 ms, so the same entry may be referred to. However, the difference value of the decoding time of sample 51 and sample 52 is 5000 ms because AU 52 to AU 100 are lost. It becomes. Therefore, a new entry is created for the sample 51 to indicate that the difference value is 5000 ms. Here, it is assumed that the received stream has a fixed frame rate of 10 Hz.

  FIG. 7 is a block diagram of a conventional demultiplexer 100 that plays back an MP4 file. The multiplexing apparatus 100 separates a moving image and audio AU from an input MP4 file, decodes and reproduces them, and includes a header separation unit 101, a header memory 102, an mdat memory 103, a sample acquisition unit 104, a time An information analysis unit 105 and a decoding display unit 106 are provided.

  The header separation means 101 separates the header data Hdat and the mdat data Dmdat from the MP4 file FileIn, and inputs the header data Hdat to the header memory 102 and the mdat data Dmdat to the mdat memory 103, respectively. Here, the header data Hdat includes moov or moof data. The time information analysis unit 105 acquires and analyzes the header time information THinf from the header memory 102, and determines a voice or moving image sample that satisfies the reproduction start time Tst input from the outside. Further, the sample number SplNum of the determined sample is input to the sample acquisition unit 104, and time information Tinf indicating the decoding time and display time of the sample having the sample number SplNum is input to the decoding display unit 106. When the decoding time and the display time are equal, only the decoding time may be included in the time information Tinf. Here, the header time information THinf refers to information on the decoding time and display time for each sample stored in stts and ctts in moov, trun in moof, and the like. The sample number is a number for identifying a sample in each track of MP4, and is incremented by 1 in the decoding order from the first sample. Therefore, after the start of reproduction, the sample number SplNum is incremented by 1 unless a new reproduction start time is given from the outside in order to perform seek reproduction or the like. Next, the sample acquisition unit acquires the sample data splData1 having the sample number SP1Num and inputs it to the decoding display unit 106. As a procedure, the access information AcsInf acquired from the header memory 102 is analyzed, the storage position of the sample having the desired sample number is specified, and the sample data splDat1 is acquired from the mdat memory 103. Finally, the decoding display means 106 decodes the sample data splDat1 based on the decoding time acquired from the time information Tinf and the display time, and outputs it as output Out.

FIG. 8 is a flowchart showing the operation of the demultiplexer 100. First, in step 1001, the header portion and the data portion of the MP4 file are separated, and then in step 1002, a sample to start decoding is determined. In step 1003, the decoding time and display time of the sample and the playback time length (Dur_in) are determined from the time information in the header part. Here, the reproduction time length is a time for which display is continued for a moving image, and is a time for continuing reproduction for a sound. Next, in step 1004, samples are decoded and displayed based on the decoding time and display time acquired in step 1003, and playback is continued during Dur_in. In the case of voice, the decoding time and the display time are the same. Steps 1003 and 1004 are repeated until the decoding of the final sample that needs to be decoded is completed. In this flowchart, after the decoding start sample is determined in step 1002, the samples are sequentially reproduced in the decoding order, and the sample number of the sample to be decoded is incremented by 1 after the decoding is started. When performing special reproduction such as seek reproduction, the operation of obtaining the sample number of the sample corresponding to the position designated by the special reproduction operation and starting decoding from the sample is repeated.
Japanese Patent Laid-Open No. 2003-114845 (Section 6-18, FIG. 6)

  The operation of reproducing the MP4 file shown in FIG. 6 in the conventional demultiplexer 100 will be described. Here, it is assumed that the decoding time and the display time are the same in the stream of FIG. The time information analysis means 105 determines the decoding time, display time, and playback time length of the sample according to the time information stored in the MP4 file header such as stts. At this time, the difference value between the decoding times of the sample 51 and the sample 52 is 5000 ms, and the playback time length of the sample 51 is determined to be 5000 ms. Therefore, the display of the decoding result of the sample 51 is continued for 5000 ms. . As a result, the sample 51 whose original reproduction time length is 100 ms is displayed 4900 ms longer than the actual one. As described above, in the conventional multiplexing apparatus 100, when the MP4 file storing the stream as shown in FIG. 6B is reproduced, the reproduction of the sample is frozen for a long time, and the reproduction quality is deteriorated. was there.

The present invention has been made to solve the above problems.
A demultiplexing apparatus according to claim 1 of the present invention separates each encoded data from data obtained by multiplexing encoded data such as moving images and audio into one or more packets, and decodes and reproduces the encoded data. A demultiplexer that analyzes a header that includes at least information indicating a decoding time or a display time of a frame in each encoded data, and a separating unit that separates a header and a payload from the multiplexed packet. Determining means for determining whether or not the attribute information for each frame in each encoded data satisfies a predetermined condition, and when the predetermined condition is satisfied, the decoding time, the display time of the frame indicated by the header, Or a correction means for correcting a reproduction time length, and the multiplexed video or audio encoded data lacks frame data. Characterized in that it comprises a band.

  A demultiplexing apparatus according to claim 2 of the present invention is the demultiplexing apparatus according to claim 1, wherein the attribute information is a reproduction time length DUR1 of a frame acquired from the header, and the determination unit If the playback time length DUR1 of the frame exceeds a predetermined value, it is determined that a discontinuous section exists between the frame and a frame whose playback time order is immediately after the frame, and the discontinuity When it is determined that there is a section, the correcting means resets the playback time length of the frame to a time length DUR2 shorter than the playback time length DUR1, and the discontinuous section is the continuous It is characterized by indicating that frame data is missing between two frames.

  A demultiplexing apparatus according to a third aspect of the present invention is the demultiplexing apparatus according to the first aspect, wherein the attribute information is not included between two consecutive frames in the decoding order in each encoded data. It indicates whether or not there is a continuous section, the determination means determines whether or not the discontinuous section exists based on the attribute information, and when it is determined that the discontinuous section exists, The correcting means resets the playback time length of the previous frame in decoding order to the time length DUR2 shorter than the playback time length DUR1 among the two consecutive frames.

  A demultiplexing apparatus according to a fourth aspect of the present invention is the demultiplexing apparatus according to the second or third aspect, wherein the multiplexed data includes both encoded video and audio encoded data. The determination unit acquires a discontinuous section for each of the moving image and the audio frame, and determines whether the discontinuous section of the moving image and the discontinuous section of the sound overlap at a reproduction time. The correction means resets the frame playback time length to a time length DUR2 shorter than the playback time length DUR1 for both the moving image and the audio when the discontinuous sections overlap. It is characterized by that.

  A demultiplexing apparatus according to claim 5 of the present invention is the demultiplexing apparatus according to claim 1, wherein when the predetermined condition is satisfied, a frame whose decoding order is immediately after the frame is: The image processing apparatus further includes a determining unit that determines a frame to be decoded in the moving image so as to be a randomly accessible frame.

  In the demultiplexing apparatus according to claim 6 of the present invention, the multiplexed data includes both moving image and audio encoded data, and the determining means performs the following in the audio when the predetermined condition is satisfied. The random accessible frame to be decoded is the frame immediately after the reproduction time of the moving image which is the same as the reproduction time of the next randomly accessible frame to be decoded.

  In the demultiplexing method according to claim 7 of the present invention, each encoded data is separated from data obtained by multiplexing encoded data such as moving images and sounds into one or more packets, and is decoded and reproduced. A demultiplexing method, comprising: a separation step of separating a header and a payload from the multiplexed packet; and analyzing the header including at least information indicating a decoding time or a display time of a frame in each encoded data A determination step for determining whether or not the attribute information for each frame in each encoded data satisfies a predetermined condition, and when the predetermined condition is satisfied, a decoding time of a frame indicated by the header, a display time, Or a correction step for correcting a reproduction time length, wherein the multiplexed moving image or audio encoded data is frame data. There characterized in that it comprises a missing region.

  The demultiplexing method according to claim 8 of the present invention is the demultiplexing method according to claim 7, wherein when the predetermined condition is satisfied, a frame whose decoding order is immediately after the frame is: The method further comprises a determining step of determining a frame to be decoded in the moving image so as to be a randomly accessible frame.

  According to a ninth aspect of the present invention, there is provided a recording medium comprising: a demultiplexer that separates and decodes and reproduces each encoded data from data obtained by multiplexing encoded data such as moving images and sounds into one or more packets. A separation step of separating a header and a payload from the multiplexed packet, and analyzing the header including at least information indicating a decoding time or a display time of a frame in each encoded data, A determination step for determining whether or not the attribute information for each frame in the encoded data satisfies a predetermined condition, and when the predetermined condition is satisfied, the decoding time, display time, or reproduction of the frame indicated by the header A correction step for correcting a time length, and the multiplexed moving image or the encoded audio data includes frame data. And characterized in that to perform the demultiplexing method characterized by comprising the dropped area.

  According to the demultiplexer of claim 1 of the present invention, when playing back an MP4 file storing an encoded stream in which a part of the data is missing, the display of the moving image freezes due to the lack of data. The effect that the degradation of reproduction quality such as can be eliminated is obtained.

  According to the demultiplexing apparatus of claim 2 of the present invention, when the MP4 file storing the encoded stream in which a part of the data is lost is reproduced without extending the MP4 standard, the data is lost. As a result, it is possible to eliminate the deterioration of the reproduction quality such as the display of the moving image being frozen.

  According to the demultiplexing apparatus of claim 3 of the present invention, since the reproduction operation can be switched after the data missing position is specified, the reproduction quality such as the display of the moving image freezes due to the data loss. It is possible to obtain an effect of more reliably preventing the lowering of.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
First, the demultiplexing apparatus 1000 according to Embodiment 1 of the present invention will be described. Here, an MP4 file in which one track each for audio and moving images is stored is input. The encoding method is MPEG-2 AAC for audio and MPEG-4 AVC for moving images. It may be an MP4 file including a plurality of audio or moving image tracks. As for the encoding method, MPEG-4 AAC and AMR (Adaptive Multi-Rate) are used for audio, and MPEG-4 Visual, H.264 are used for moving images. Other systems such as H.263, VC-1 (encoding system standardized by SMPTE) may be used. Also, the multiplexing method is not limited to MP4, and may be, for example, ASF (Advanced Systems Format, a format developed by Microsoft) or Quick Time (a format developed by Apple).

  FIG. 9 is a block diagram showing a configuration of the demultiplexer 1000. The demultiplexing apparatus 1000 includes a header separation unit 101, a header memory 102, an mdat memory 103, a sample acquisition unit 104, a time information analysis unit 1005, a decoding display unit 1002, and a correction unit 1001, and includes a correction unit 1001. Since the point and the operation of the time information analysis unit 1005 and the decoding display unit 1002 are different from those of the conventional demultiplexer 100, these differences will mainly be described.

  The time information analysis unit 1005 includes at least a decoding time and a display time of a sample having a sample number SplNum (hereinafter, a sample having a sample number SplNum is referred to as a sample [SplNum]), and a decoding time of the sample [SplNum + i]. Information Tinf1 is input to the correction means 107. Here, the sample [SplNum + i] indicates a sample immediately after the sample [SplNum] in the display order. Furthermore, the playback time length Dur_in of the sample [SplNum] calculated from the difference between the display times of the sample [SplNum + i] and the sample [SplNum] is also included in the time information Tinf1. The correction means 1001 determines the decoding time, display time, and playback time length of the sample [SplNum] based on the time information Tinf1, and inputs it to the decoding display means 1002 as the correction time ModTinf. The decoding display unit 1002 decodes and reproduces the sample [SplNum] according to the correction time ModTinf.

  FIG. 10 is a flowchart showing the operation of the demultiplexer 1000. First, in step 1001, the header portion and the data portion of the MP4 file are separated, and in step 1002, a sample to start decoding is determined. Next, in step 1103, based on time information such as stts, ctts, or trun, a sample [SplNum] that is a sample to be decoded next and a sample [SplNum + i] whose display order is immediately after the sample [SplNum] are displayed. The decoding time, the display time, and the playback time length of the sample [SplNum] are acquired.

  In step 1104, it is determined whether or not the playback time length Dur_in of the sample [SplNum] needs to be corrected. If it is determined in step 1104 that correction is necessary, the process proceeds to step 1105. If it is determined that correction is not necessary, the process proceeds to step 1107. In step 1105, the playback time length of the sample [SplNum] is corrected to Dur_out, the display time of the sample [SplNum + i] is corrected, and the process proceeds to step 1106. Here, by correcting the display time of the sample [SplNum + i], the display times of the samples after the sample [SplNum] also need to be corrected. Therefore, the decoding times of the subsequent samples are also set in time for the corrected display time. Correct it. Normally, the decoding time of the sample [SplNum] and the subsequent samples is advanced by a time equal to the difference between Dur_out and Dur_in, but the decoding time may be adjusted by an arbitrary time if it is in time for the display time. In step 1106, the sample [SplNum] is decoded and displayed, and the time reproduction of Dur_out is continued. “Continue playback” indicates that the decoding result is repeatedly displayed in the case of a moving image, and indicates that the section playback is continued in the case of a sound. However, in the case of audio, if the playback time length Dur_out is longer than the playback time length of one frame, a predetermined process such as silence or pseudo-noise is played after playing one frame. . In step 1107, the sample [SplNum] is decoded and displayed, and the time display of Dur_in is continued. In this way, the processing from step 1103 to step 1107 is repeated until the last picture that needs to be decoded. Here, when the decoding time of each sample is equal to the display time, only the decoding time needs to be acquired in the processing from step 1103 to step 1106, and the playback time length is also decoded in two consecutive samples. It can be calculated from the time difference value. Whether the decoding time and the display time are the same for each sample is, in the case of a sample in moov, whether or not ctts, which is a box for storing the difference value between the decoding time and the display time, exists in moov, or ctts It is possible to determine whether or not the difference value indicated by the entry is zero. In the case of a sample in the moof, it can be determined whether there is a field for storing a difference value between the decoding time and the display time of the sample in the trun in the moof, and whether the value of the field is 0. .

  FIGS. 11A and 11B show calculation examples of the reproduction time length Dur_in of the moving image sample. In the figure, I, P, and B indicate samples in which an intra-picture coded picture (I picture), a forward reference picture (P picture), and a bi-predictive picture (B picture) are stored, respectively. The numbers added to the types indicate the display order. Furthermore, the samples indicated by dotted lines indicate samples that could not be acquired due to reception errors such as packet loss. FIG. 11A shows an example in which the decoding time and display time of each sample are equal, and it is assumed that three pictures P5, P6, and P7 could not be acquired. In this track, since the decoding time of each sample is equal to the display time, the reproduction time length Dur_in of P4 can be calculated as 400 ms by subtracting the decoding time of P4 from the decoding time of P8 immediately after the decoding order. . On the other hand, FIG. 11B is an example in which samples having different decoding times and display times are mixed, and B5 cannot be acquired. In this track, the playback time length Dur_in matches the difference value between the display times of consecutive samples in the display order. Therefore, the playback time length Dur_in of B4 is obtained by subtracting the display time of B4 from the display time of P6. , 200 ms is calculated. Note that the playback time length Dur_in of B4 is not a difference value from the display time of P9, which is the next picture in decoding order. It is also necessary to adjust the decoding times of pictures subsequent to P9 whose decoding order is after B4 so as to be in time for the display times of these pictures. For example, if the display time of P6 is changed to 500 ms, the display time of B7 becomes 600 ms. Therefore, the decoding of B7 must be completed at 600 ms.

  Next, the determination operation in step 1104 and the method for determining the reproduction time length Dur_out in step 1105 will be described. First, in step 1104, determination is made based on whether or not the value of the reproduction time length Dur_in exceeds the threshold value THRES. The threshold value THRES is a value determined for each medium such as a moving image or sound, and may be the same value for the moving image and sound. For example, in the case of a 10 Hz moving image, the playback time length per frame is normally 100 ms, and when it can be assumed that the playback time length of one frame does not exceed 500 ms even when the frame rate varies, the threshold THRES Is set to 500 ms. That is, when the reproduction time length Dur_in exceeds 500 ms, it is considered that data is lost due to reception errors such as packet loss and is discontinuous. In this way, the threshold value THRES can be determined based on the frame rate of moving images and audio. Here, the frame rate of the audio or moving image stored in the MP4 file is obtained from the time information in the header such as stts or the parameter in the initialization information of the stream stored in the sample entry. In the case of MPEG-4 AVC, at a fixed frame rate, the frame rate can be acquired from the parameters in the SPS stored in the sample entry. Even when the frame rate is variable, since the maximum value of the frame rate can be estimated based on the parameters in the SPS, the threshold value THRES may be determined based on the result. In the case of speech, if the sampling frequency is obtained from the header information, the frame rate can be calculated by dividing the number of sample points constituting one frame by the sampling frequency. Note that a predetermined value may be set as the threshold value THRES without depending on the frame rate. Furthermore, it may be determined based on brand information of the MP4 file, or a box storing information indicating the threshold value THRES may be separately stored in the MP4 file and determined by referring to the box. Here, the threshold value THRES may be set for each track, or may be a value common to each track.

  Further, from information other than the threshold value THRES, it is determined whether or not the sample [SplNum] and the sample [SplNum + 1] are discontinuous due to a lack of data due to a reception error or the like, and the reproduction time length Dur_in is determined. The presence or absence of correction may be determined. Whether it is discontinuous may be determined from either sample data or header information. For example, in the case of MPEG-4 AVC, it can be determined from the sample data based on whether or not a NAL unit having a specific NAL unit type indicating a discontinuous point is included in the sample [SplNum]. As header information, stbl or Box in trun indicates whether or not two consecutive samples are discontinuous, and determination can be made with reference to these Boxes.

  Note that the playback time length may also be corrected when the value of the playback time length Dur_in is the same as the threshold value THRES.

FIG. 12 is a flowchart showing a method for determining the playback time length Dur_out in step 1105. At this time, even after the reproduction time length Dur_out is corrected, AV synchronization is ensured in subsequent samples. Here, it is assumed that there is always a moving image track. First, in step 1201, it is determined whether or not there is a voice. If it exists, the process proceeds to step 1202, and if not, the process proceeds to step 1205. In step 1202, it is determined whether or not there is an overlap of discontinuous sections between the moving image and the sound. If it is determined that there is an overlap, the process proceeds to step 1203, and if there is no overlap, the process proceeds to step 1204. In step 1203, the playback time length Dur_in is shortened by Dis_dur which is the time length of the portion where the discontinuous sections overlap. as a result,
Modified playback time length Dur_out
= Playback time length before correction Dur_in-Dis_dur
It becomes. Here, Dis_dur is calculated as follows.
Now, for a moving image, there is a discontinuous section between sample [V_SampleNum] and sample [V_SampleNum + i]
For speech, if there is a discontinuous section between sample [A_SampleNum] and sample [A_SampleNum + i]
Dis_dur =
MIN (display time of sample [V_SampleNum + i], display time of sample [A_SampleNum + i])
-MAX (Display time of sample [V_SampleNum] + 1 frame interval of moving image, display time of sample [A_SampleNum] + 1 frame interval of audio)
It becomes. Here, MAX (m, n) indicates the larger value of m and n, and MIN (m, n) indicates the smaller value of m and n. If both values are equal, that value is indicated. That is, playback of a section where there is no track to be played back is skipped.
In step 1204, without adjusting the playback time length,
Modified playback time length Dur_out
= Play time length before correction Dur_in
And

  In step 1205, the playback time length Dur_out = 1 frame interval. Here, when the frame rate is fixed, the one-frame interval indicates the reproduction time length per frame at the frame rate. For example, if the frame rate is 10 Hz, one frame interval corresponds to 100 ms. Even if the frame rate is variable, the playback time length of each frame is an integral multiple of the frame interval, so that the frame interval can be applied. For example, although it is usually 10 Hz, the frame may be dropped due to the bit rate at the time of encoding and the restriction of the processing time. At this time, the playback time length of one frame is an integral multiple of a frame interval of 100 ms, such as 200 ms or 300 ms. In these cases, the playback time length Dur_out may be set to one frame interval.

FIG. 13 shows a case where one track each of audio and moving images exists in the MP4 file. Note that the decoding time and the display time are the same in the moving image. In FIG. 13A, the moving image is determined to be discontinuous when the display time is between 400 ms and 600 ms, and the sound is determined to be discontinuous between 200 ms and 350 ms. At this time, the data of the other tracks are continuous even in the sections where each is discontinuous. For example, the moving image can be played back even in the section where the sound cannot be played back, so Dur_in is not corrected. The display time of the fourth moving image sample V4 is 300 ms, and the reproduction time length Dur_in is 300 ms, which is the difference between the display times of the fourth and fifth moving image samples. On the other hand, FIG. 13B shows a case in which the moving image is determined to be discontinuous when the display time is between 400 ms and 600 ms, and the sound is discontinuous between 400 ms to 500 ms. At this time, between 400 ms and 500 ms, it can be determined that both moving image and audio data are missing. Therefore, the playback time length Dur_in is corrected for the fourth moving image sample V4 and the eighth audio sample A8. Since the overlap of discontinuous sections is 100ms,
For V4,
Reproduction time length Dur_out = Reproduction time length Dur_in − 100 ms
= (600ms-300ms)-100ms
= 200 ms
It becomes.
For A8,
Reproduction time length Dur_out = Reproduction time length Dur_in − 100 ms
= (500ms-350ms)-100ms
= 50 ms
It becomes.

  Note that the playback time length Dur_in may be corrected only when a section in which both moving image and audio data are missing exceeds a predetermined time length.

  Also, not only the case where there is one voice and one moving image, but when there are two or more tracks, the playback time length is shortened only when the discontinuous sections of those tracks overlap. Also good.

  In the above description, it is assumed that an audio track always exists, and it may be determined in step 1201 whether or not a moving image track exists.

  Furthermore, the user may be shown that playback has been skipped in the discontinuous section. FIG. 14 shows an example in which a seek bar indicating the playback progress portion is used. As shown in FIG. 14 (a), when playing an MP4 file storing 60 seconds of AV content, playback is skipped because the display time is a discontinuous section from 20 seconds to 25 seconds. And FIG. 14B shows the state of the seek bar when the elapsed time from the start of reproduction is 20 seconds, and the end of the seek bar is at the position of 20 seconds. FIG. 14C shows the state of the seek bar when the elapsed time from the start of reproduction is 25 seconds. At this time, since the reproduction is skipped for 5 seconds from 20 seconds to 25 seconds in the AV data in the MP4 file, the position of the content of 25 seconds is reproduced when 20 seconds elapse, and the content is reproduced when the elapsed time is 25 seconds. Positions up to 30 seconds are reproduced. Therefore, the end of the seek bar indicates the position of 30 seconds. Although the user can know that the seek bar is a discontinuous section by moving discontinuously, the section skipped in the seek bar may be explicitly shown as shown in FIG. In addition, you may show only the position which is reproducing | regenerating, without showing the reproduction progress part by a seek bar. Alternatively, the total playback time in consideration of skipping may be indicated on the seek bar in advance, and the seek bar may be moved continuously during playback. In the example of FIG. 14, the total playback time is set to 55 seconds in advance.

  Further, whether to skip to the resynchronization point may be switched by adjusting the reproduction time length of the sample. For example, the playback terminal may operate as set in advance. Further, whether to perform the process may be selected by the user on the terminal screen prior to the start of reproduction.

  In addition, the skip processing is mainly required when incomplete data is stored, such as when there is a discontinuous section in the track of the MP4 file. Therefore, whether to perform skip processing may be determined according to identification information indicating whether or not an incomplete track is included in the file. Here, it is possible to use information in a specific box defined by an operational standard such as MP4 brand stored in ftyp or the like, 3GPP, or SDA (Secure Digital Association). For example, the determination is made based on whether or not there is a brand indicating that it is an MP4 file in which broadcast data is recorded.

  When receiving a broadcast on a portable terminal, if the reception situation of radio waves deteriorates due to the shadow of a building or the like, data cannot be received for a relatively long time such as several seconds. The portable demultiplexer of this embodiment is particularly effective for file reproduction.

(Embodiment 2)
In the demultiplexing apparatus 1000 according to Embodiment 1, when a discontinuous section is skipped, decoding is started from the next sample in decoding order. However, in particular, in the case of moving images, it is not guaranteed whether decoding can be performed from the next sample in the decoding order, and thus the skipped image may not be decoded correctly. For example, in the case where the sample to start decoding is a P picture and the P picture refers to a picture that could not be acquired due to a reception error, pictures up to the next I picture may not be decoded correctly. The demultiplexing apparatus according to the present embodiment starts decoding from randomly accessible samples after skipping, thereby ensuring that samples after decoding can be correctly decoded and improving reproduction quality.

  The demultiplexing apparatus according to the present embodiment is different from the demultiplexing apparatus according to the first embodiment in that the operations of the correction unit 1001 and the sample acquisition unit 104 are different. Hereinafter, differences from the demultiplexer according to Embodiment 1 will be described.

  The correcting unit 1001 determines the reproduction time length Dur_out of the sample [SplNum], determines the sample [SplNum + j] to be decoded next, and calculates the display time of the sample [SplNum + j]. Further, the time information analyzing unit 1005 is notified that the next sample to be decoded is the sample [SplNum + j]. The sample acquisition means 104 inputs the sample data splDat1 of the sample determined by the correction means 1001 to the decoding display means 1002, and the decoding display means 1002 uses the sample data based on the decoding time and display time input from the correction means 1001. Decode and output splDat1.

  Here, if the sample [SplNum + i] and the sample [SplNum + j] are the same, the sample decoded next to the sample [SplNum] is the same as the sample determined in the multiplexing apparatus of the first embodiment.

  FIG. 15 is a flowchart showing the operation of the demultiplexing apparatus according to the present embodiment, and the operations of Step 1301 and Step 1302 are different from those of the demultiplexing apparatus 1000. In step 1301, the playback time length of the sample [SplNum] is corrected. In step 1302, a sample [SplNum + j], which is a sample to be decoded next to the sample [SplNum], is determined, and a display time of the sample [SplNum + j] is determined. The display time of the sample [SplNum + j] is obtained by subtracting the following time length from the display time obtained from the header information of the conventional MP4 file such as stts and ctts or trun.

{Display time of sample [SplNum + j]-MAX (Display time of sample [V_SampleNum] + 1 frame interval of moving image, display time of sample [A_SampleNum] + 1 frame interval of audio)}
Here, it is assumed that there is a discontinuous section immediately after the sample [V_SampleNum] for the moving image and immediately after the sample [A_SampleNum] for the sound.

  FIG. 16 is a flowchart showing an operation when determining a sample [SplNum + j] in step 1302. In step 1401, it is determined whether or not the sample [SplNum] is a sample in the fragment. If it is a sample in the fragment, the process proceeds to step 1402. Otherwise, the process proceeds to step 1405. In step 1402, it is determined whether mfra exists in the MP4 file. mfra is a box indicating the position of a randomly accessible sample stored in the fragment part. If mfra exists, the process proceeds to step 1403; otherwise, the process proceeds to step 1404. In step 1403, it is determined that the sample whose decoding order is next to the sample [SplNum] among the randomly accessible samples indicated by mfra is the sample [SplNum + j]. In addition, in mfra, a GDR (Global Decoder Refresh) type sample can be shown in addition to the sync sample, but only the sync sample may be selected as the skip destination. Here, the GDR type sample is a sample in which a correct decoding result is obtained when a predetermined number of samples are decoded when decoding is started from the sample, and whether a correct decoding result is obtained from the decoding start sample. It is different from the sync sample. Whether or not the randomly accessible sample is a sync sample can be determined by referring to flag information in trun indicating whether or not each sample is a sync sample. In step 1404, it is determined that the first sample of the fragment immediately after in the decoding order is the sample [SplNum + j]. Here, the term “fragment immediately after” refers to a fragment in which the decoding order is immediately after the fragment in which the sample of the media is stored. If the sample [SplNum] is a sample in the moov, the process proceeds to step 1405. At this time, it is determined that the next sample in the decoding order is the sample [SplNum + j] among the sync samples. A sync sample in moov is indicated by stss. Here, in moov, when there is no sync sample after the sample [SplNum], skip to the first sample of the first fragment.

  Note that the sample [SplNum + j] may be determined so that there is always a sample whose decoding time is earlier than the moving image decoding start sample [SplNum + j] in the audio samples after the sample [A_SampleNum]. At this time, in step 1403, the sample whose display time is the same as that of the sample [A_SampleNum + 1] or that can be accessed at random later and whose display time is the earliest is set as the sample [SplNum + j]. In step 1403, among the first samples of fragments, the sample [A_SampleNum + 1] whose display time is the same as or later than that of the sample [A_SampleNum + 1] is the sample [SplNum + j]. In step 1405, the sample with the same display time as the sample [A_SampleNum + 1] or the later sync sample with the earliest display time is set as the sample [SplNum + j].

  In addition, in the audio samples after the sample [A_SampleNum], there are samples whose decoding time is later than the moving image decoding start sample [SplNum + j] and the difference value of the decoding time from the sample [SplNum + j] is equal to or smaller than a predetermined value. You may make it exist. By doing so, it is ensured that the audio can be reproduced near the reproduction start time of the moving image.

  When a moving image track exists, a moving image decoding start sample is determined by the above method. FIG. 17 is a flowchart showing an operation when determining audio decoding samples. First, in step 1501, a moving image decoding start sample (VSPL_NEXT) is determined. Next, in step 1502, it is determined whether or not an audio track exists in the MP4 file. If an audio track exists, the process proceeds to step 1503. If not, the process proceeds to step 1504. In step 1503, for audio, it is determined to start decoding from a sample having the same decoding time as the moving image decoding start sample VSPL_NEXT, or immediately after, and the process proceeds to step 1504. In step 1504, it is determined that the moving image is decoded and reproduced from VSPL_NEXT. When the video decoding start sample is a GDR type sample, the display may start from a sample from which a correct decoding result is obtained. At this time, the decoding start sample and the display start sample are different. . Furthermore, a speech decoding start sample may be determined according to a sample for starting display of a moving image.

  In addition, when there is a sample after the sample [A_SampleNum] whose decoding time is earlier than the moving image decoding start sample [SplNum + j], an audio sample having a decoding time immediately before the sample [SplNum + j] Decoding may be started from.

  Note that the audio track may start decoding from the immediately following sync sample without depending on the decoding start sample of the moving image track. At this time, in audio, since all the samples are usually sync samples, decoding starts from the immediately following sample.

  FIG. 18 shows an example of determining a sample [SplNum + j], which is the same case as FIG. 13B. However, it is assumed that only the first and seventh samples of the moving image are sync samples. In FIG. 13B, decoding of a moving image is started from V5 immediately after the discontinuous section, but in this embodiment, decoding is started from V7 that is a sync sample immediately after the discontinuous section. Also for audio, decoding starts from A15 where the decoding time is equal to V7.

  Up to this point, the case where data is lost due to packet loss or the like and a sample cannot be acquired as a result has been described. However, the same method can be applied when reproducing a sample from which a part of the data is lost. FIG. 21 shows an operation when a track including a sample in which a part of data is missing is reproduced. FIG. 21A shows an example of a stream stored in an MP4 file. A sample surrounded by a dotted line is a sample from which a part of data is missing (hereinafter referred to as an incomplete sample). In this example, the second through eleventh samples are incomplete samples. Whether or not the sample is incomplete may be indicated by including identification information in the mdat sample data, or may be indicated separately in header information such as moov or moof.

  Incomplete samples may not be decoded normally, and correct decoding results cannot be obtained even if error processing such as concealment is performed using the decoding result of the immediately preceding sample. When it is continuous, the reproduction quality is significantly reduced. On the other hand, since it is considered that a reproduction quality of a certain level or more can be obtained by error processing for a short time, it is determined whether or not to reproduce the sample in the section according to the time length of the section in which incomplete samples continue. It is effective. FIG. 21B is a flowchart showing the reproduction operation when an incomplete sample exists. First, in step 1601, it is determined whether or not an incomplete sample continues for a certain period of time. If it is determined that the incomplete sample continues for a certain period of time, the process proceeds to step 1602, and if not, the process proceeds to step 1603. In step 1602, it is determined that the reproduction of the section in which the incomplete samples are continuous is skipped, and in step 1603, it is determined that the incomplete samples in the section are also reproduced. Note that the playback operation may be determined based on whether or not the sections in which incomplete samples continue for the audio and video tracks overlap. Also, the user may decide whether to play incomplete samples. Further, the reproduction operation may be determined based on other determination conditions such as whether or not incomplete samples are continuous, and whether or not a predetermined ratio or more of samples are incomplete samples within a certain interval. .

(Embodiment 3)
Here, a system using the multiplexing method conversion apparatus and the demultiplexing apparatus shown in the first to second embodiments will be described.

  FIG. 19 is a block diagram showing an overall configuration of a system that realizes a content distribution service by broadcasting and communication. First, a case where broadcast data is received will be described. A cellular phone ex105 or a disk recorder ex104 such as a DVD recorder receives a TS packet sequence in which digitized encoded media data is multiplexed. In the mobile phone ex105, the received TS packet sequence is converted into MP4 and then recorded on the SD card ex106. The recorded MP4 file can be viewed on a mobile phone ex105, a disk recorder ex104, or a personal computer (not shown) equipped with the demultiplexer according to the present invention. Also, the MP4 file is attached to an e-mail and transmitted from the mobile phone ex105 via the radio base station ex107 to another mobile phone ex108 equipped with the demultiplexing device according to the present invention. You can also watch the file. Furthermore, instead of mail attachment, download may be performed from the mobile phone ex105 to the mobile phone ex108 or pseudo-streaming distribution using a protocol such as HTTP (Hyper Text Transport Protocol) and TCP (Transmission Control Protocol).

  Also in the disk recorder ex104, the received TS packet sequence can be converted into MP4 and recorded on an optical disk such as an SD card or a DVD, or a hard disk. The recorded MP4 file may be downloaded to a mobile phone or a personal computer (not shown), or pseudo-streamed.

  When receiving the TS packet sequence distributed from the content server ex102 via the Internet at the mobile phone ex105 or the disc recorder ex104, the MP4 file can be used in the same manner as when the broadcast data is received.

  In addition, the demultiplexer according to the present invention is also used when MP4 is used to record data transmitted by a protocol such as RTP (Real-time Transport Protocol) used for streaming delivery on the Internet regardless of TS. Is applicable.

(Embodiment 4)
By recording a program for realizing the demultiplexing method in the demultiplexing apparatus shown in each of the above embodiments on a storage medium such as a flexible disk, the processing shown in each of the above embodiments is performed. It can be easily implemented in an independent computer system.

  FIG. 20 is an explanatory diagram when the demultiplexing method in the demultiplexing apparatus of each of the above embodiments is implemented by a computer system using a program recorded on a recording medium such as a flexible disk.

  FIG. 20B shows the appearance, cross-sectional structure, and flexible disk as seen from the front of the flexible disk, and FIG. 20A shows an example of the physical format of the flexible disk that is the recording medium body. The flexible disk FD is built in the case F, and a plurality of tracks Tr are formed concentrically on the surface of the disk from the outer periphery toward the inner periphery, and each track is divided into 16 sectors Se in the angular direction. ing. Therefore, in the flexible disk storing the program, the program is recorded in an area allocated on the flexible disk FD.

  FIG. 20C shows a configuration for recording and reproducing the program on the flexible disk FD. When the above-described program for realizing the multiplexing method conversion method in the multiplexing method conversion device and the demultiplexing method in the demultiplexing device is recorded on the flexible disk FD, the program is transferred from the computer system Cs via the flexible disk drive. Write. In the case where a demultiplexing method in the demultiplexing device in each of the above embodiments for realizing the demultiplexing method in the demultiplexing device in each of the above embodiments by a program in a flexible disk is constructed in a computer system. The program is read from the flexible disk by the flexible disk drive and transferred to the computer system.

  In the above description, a flexible disk is used as the recording medium, but the same can be done using an optical disk. Further, the recording medium is not limited to this, and any recording medium such as an IC card or a ROM cassette capable of recording a program can be similarly implemented.

  The demultiplexing apparatus according to the present invention can be applied to all devices that play MP4 files that record received streams in an environment in which data loss due to packet loss occurs, such as broadcasting or communication, and in particular, SD (Secure Digital). It is effective in a portable terminal that can reproduce content stored in a recording medium such as a card.

The figure which shows the data structure of AU in MPEG-4 AVC The figure which shows the data structure of a PES packet and TS packet The figure which shows the Box structure of MP4 The figure which shows the hierarchical structure of moov in MP4 Example of fragmented MP4 structure The figure explaining the MP4 file which stores the stream from which some data are missing The block diagram which shows the structure of the conventional demultiplexing apparatus 100 Flowchart showing the operation of the conventional demultiplexer 100 1 is a block diagram showing the overall configuration of a demultiplexer 1000 according to Embodiment 1 of the present invention. The flowchart which shows the operation | movement outline | summary of the demultiplexing apparatus 1000 which concerns on Embodiment 1 of this invention. The figure which shows the operation example of the demultiplexing apparatus 1000 which concerns on Embodiment 1 of this invention. The figure which shows the operation example at the time of determining whether the demultiplexing apparatus 1000 which concerns on Embodiment 1 of this invention corrects the reproduction time length of a sample. The figure which shows the operation example at the time of determining the reproduction time length of a sample in the demultiplexing apparatus 1000 which concerns on Embodiment 1 of this invention. The figure which shows an example of the notification method to a user that reproduction | regeneration of the discontinuous area was skipped The flowchart which shows the operation | movement outline | summary of the demultiplexing apparatus which concerns on Embodiment 2 of this invention. The flowchart which shows the operation | movement which determines the decoding start sample of the moving image after a discontinuous area in the demultiplexing apparatus which concerns on Embodiment 2 of this invention. The flowchart which shows the operation | movement which determines the decoding start sample of the audio | voice after a discontinuous area in the demultiplexing apparatus which concerns on Embodiment 2 of this invention. The flowchart which shows the operation example of the demultiplexing apparatus which concerns on Embodiment 2 of this invention. The figure which shows the practical example of the demultiplexing apparatus which concerns on Embodiment 1 and Embodiment 2 of this invention Explanatory drawing about the storage medium for storing the program for implement | achieving the demultiplexing method in the demultiplexing apparatus of each said embodiment with a computer system Flow chart showing the operation when playing an MP4 file containing incomplete samples

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Header separation means 102 Header memory 103 mdat memory 104 Sample acquisition means 105 Time information analysis means 106 Decoding display means

Claims (9)

A demultiplexer that separates and decodes and reproduces each encoded data from data obtained by multiplexing encoded data such as moving images and voices into one or more packets;
Separating means for separating a header and a payload from the multiplexed packet;
Determination means for analyzing whether or not the attribute information for each frame in each encoded data satisfies a predetermined condition by analyzing the header including at least information indicating a decoding time or a display time of the frame in each encoded data When,
If the predetermined condition is satisfied, correction means for correcting the decoding time, display time, or playback time length of the frame indicated by the header;
With
The demultiplexer according to claim 1, wherein the multiplexed moving image or audio encoded data includes a region where frame data is missing. The attribute information is a playback time length DUR1 of the frame acquired from the header,
The determination means determines that there is a discontinuous section between the frame and a frame whose playback time order is immediately after the frame when the playback time length DUR1 of the frame exceeds a predetermined value;
When it is determined that the discontinuous section exists, the correcting unit resets the playback time length of the frame to a time length DUR2 shorter than the playback time length DUR1,
The demultiplexing apparatus according to claim 1, wherein the discontinuous section indicates that frame data is missing between the two consecutive frames. The attribute information indicates whether or not the discontinuous section exists between two consecutive frames in the decoding order in each encoded data,
The determination means determines whether the discontinuous section exists based on the attribute information,
When it is determined that the discontinuous section exists, the correcting means sets the playback time length of the previous frame in decoding order to the time length DUR2 shorter than the playback time length DUR1 among the two consecutive frames. The demultiplexing apparatus according to claim 1, wherein the demultiplexing apparatus is reset to The multiplexed data includes both moving image and audio encoded data,
The determination means acquires a discontinuous section for each of the moving image and the audio frame, determines whether the discontinuous section of the moving image and the discontinuous section of the audio overlap at a reproduction time,
The correction means resets the frame playback time length to a time length DUR2 shorter than the playback time length DUR1 for both the moving image and the audio when the discontinuous sections overlap. The demultiplexing apparatus according to claim 2 or 3, wherein The demultiplexer according to claim 1, wherein
If the predetermined condition is satisfied, a determination unit that determines a frame to be decoded in the moving image so that a frame whose decoding order is immediately after the frame is a randomly accessible frame;
The demultiplexer further comprising: The demultiplexer according to claim 1, wherein
The multiplexed data includes both moving image and audio encoded data,
When the predetermined means satisfies the predetermined condition, the random accessible frame to be decoded next in the audio has the same reproduction time as the reproduction time of the next randomly accessible frame to be decoded in the moving image. A demultiplexer characterized by being a frame immediately after. A demultiplexing method for separating, decoding, and reproducing each encoded data from data obtained by multiplexing encoded data such as moving images and audio into one or more packets,
A separation step of separating a header and a payload from the multiplexed packet;
A determination step of analyzing whether or not the attribute information for each frame in each encoded data satisfies a predetermined condition by analyzing the header including information indicating at least a decoding time or a display time of a frame in each encoded data When,
If the predetermined condition is satisfied, a correction step of correcting the decoding time, display time, or playback time length of the frame indicated by the header;
With
2. The demultiplexing method according to claim 1, wherein the multiplexed moving image or audio encoded data includes an area where frame data is missing. The demultiplexing method according to claim 7, comprising:
A determination step of determining a frame to be decoded in the moving image so that a frame whose decoding order is immediately after the frame is a randomly accessible frame when the predetermined condition is satisfied;
The demultiplexing method further comprising: A demultiplexing method for separating, decoding, and reproducing each encoded data from data obtained by multiplexing encoded data such as moving images and audio into one or more packets,
A separation step of separating a header and a payload from the multiplexed packet;
A determination step of analyzing whether or not the attribute information for each frame in each encoded data satisfies a predetermined condition by analyzing the header including information indicating at least a decoding time or a display time of a frame in each encoded data When,
If the predetermined condition is satisfied, a correction step of correcting the decoding time, display time, or playback time length of the frame indicated by the header;
With
The multiplexed moving image or audio encoded data includes a demultiplexing method characterized by including a region where frame data is missing.

Images