CN1650626A - Method and apparatus for supporting AVC in MP4 - Google Patents

Abstract

Paramater set metadata identifying parameter sets for multiple portions of multimedia data is created. Further, a file associated with the multimedia data is formed. This file includes the parameter set metadata, as well as other information pertaining to the multimedia data.

Description

Method and apparatus for supporting AVC in MP4

related application

This application is related to and claims the benefit of the following U.S. provisional patent applications: 60/359,606, filed February 25, 2002; 60/361,773, filed March 5, 2002; Patent Application No. 60/363,643, filed May 8, which is hereby incorporated by reference into these Provisional Patent Applications.

field of invention

The present invention relates generally to storing and retrieving audiovisual content in multimedia file formats, and in particular to file formats compatible with the ISO media file format.

Copyright Notation/Permission

Portions of the disclosure of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it has been published in the patent document or records in the Patent and Trademark Office, but otherwise reserves all copyright rights whatsoever. The following marks apply to the software and data described below and are included in the drawings: Copyright2001, Sony Electronics, Inc. All rights reserved.

Background of the invention

With the rapid growth of network, multimedia, database and other digital capacity needs, many multimedia encoding and storage schemes have evolved. One of the well-known file formats for encoding and storing audiovisual data is the QuickTime ^(R) file format developed by Apple Computer Corporation. Use of the QuickTime file format as a starting point for the creation of the International Organization for Standardization (ISO) multimedia file format, ISO/IEC 14496-12, Information technology — Coding of audiovisual objects — Part 12: ISO media file format (aka ISO file format), The QuickTime file format is used in turn as a template for the following two standard file formats: (1) for the MPEG-4 file format developed by the Moving Picture Experts Group, commonly known as MP4 (ISO/IEC 14496-14, Information technology—Coding of audiovisual objects—Part 14: MP4 file format); and (2) the file format of JPEG 2000 (ISO/IEC 15444-1), developed by the Joint Photographic Experts Group (JPEG).

The ISO media file format consists of object-oriented structures called boxes (also known as atoms or objects). Two important top-level boxes contain media data or metadata. Most boxes describe a hierarchy of metadata that provides descriptive, structural and temporal information about the actual media data. This collection of boxes is contained in a box known as a movie box. The media data itself can be inside or outside the media data frame. Each stream of media data is called a track (aka elementary stream or simply stream).

The original metadata is the movie object. The movie box includes a track box that describes temporarily displayed media data. The media data corresponding to the track may be of various types (eg, video data, audio data, binary format screen representation (BIFS), etc.). Each track is further divided into samples (aka access units or frames). A sample represents a unit of media data at a particular point in time. Sampling metadata is contained in a set of sampling boxes. Each track box contains a sample table box metadata box, which contains boxes giving its media data, etc. the time of each sample, its size in bytes, and its location (outside or inside the file), etc. A sample is the smallest data entity that can represent time, location, and other metadata information.

Recently, the MPEG Video Group started work together with the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU) as the Joint Video Team (JVT) to develop a new video coding called ITU Recommendation H.264 or MPEG-4-Part 10 /decode(codec) standard, Advanced Video Codec (AVC) or JVT codec. Herein, these terms and their abbreviations are used interchangeably, such as H.264, JVT and AVC.

The JVT codec design distinguishes two different conceptual layers: Video Coding Layer (VCL) and Network Abstraction Layer (NAL). VCL contains codec related coding parts, such as motion compensation, coefficient transform coding and entropy coding. The output of VCL is time slice (slice), each time slice contains a series of macroblocks and associated header information. NAL abstracts VCL from the details of the transport layer used for VCL data. It defines a general and transport-independent representation for information on time slices. NAL defines the interface between the video codec itself and the outside world. Internally, NAL uses NAL grouping. A NAL packet includes a type field indicating the payload type plus a set of bits in the payload. Data within a single time slice can be further divided into different data partitions.

In many existing video encoding formats, the encoded data stream includes a header that contains parameters that control the decoding process. For example, the MPEG-2 video standard includes a sequence header, an enhanced group of pictures (GOP), and a picture header in front of video data corresponding to those items. In JVT, the information needed to decode VCL data is assembled into a parameter set. Each parameter set is given an identifier, which is then used as a reference from the timeslice. The parameter set may be sent outside the stream (out-of-band) instead of inside the stream (in-band).

Existing file formats do not provide facilities for storing parameter sets associated with encoded media data; nor do they provide facilities for efficiently linking media data (i.e., samples or subsamples) to parameter sets in order to be able to efficiently A means of retrieving and sending parameter sets efficiently.

In the ISO media file format, the smallest unit that can be accessed without parsing the media data is the sample, that is, the entire picture in AVC. In many encoding formats, samples can be further divided into smaller units called subsamples (also known as sample slices or access unit slices). As far as AVC is concerned, subsampling is equivalent to time slices. However, existing file formats do not support access to sampled subsections. This lack of access to subsampling prevents flexible packetization of JVT media data for streaming for systems that require flexible shaping of data stored in files for streaming.

Another limitation of existing storage formats relates to switching between stored streams and different bandwidths in response to changing network conditions while streaming media data. In a typical streaming scenario, one of the key requirements is to scale the bitrate of the compressed data in response to changing network conditions. Typically, this is accomplished by encoding multiple streams with different bandwidths and qualities set for typical network conditions and storing them in one or more files. The server can then switch among these pre-encoded streams in response to network conditions. In existing file formats, switching between streams is only possible on those samples that do not depend on previous samples used for reconstruction. Such samples are called I-frames. Currently, no support is provided for switching between streams on those samples that depend on previous samples for reconstruction (ie, P-frames or B-frames that depend on multiple samples for reference).

The AVC standard provides facilities known as switching pictures (referred to as SI pictures and SP pictures) to enable efficient switching between streams, random access and error resilience, among other features. A cutscreen is a special type of screen whose reconstruction value is exactly equal to the value of the screen it should switch to. Switching pictures can use reference pictures different from those used to predict their matching pictures, thus providing more efficient coding than using I-frames. In order to effectively use cutaway pictures stored in a file, it must be known which group pictures are equivalent and which pictures are used for prediction. Existing file formats do not provide this information, so it must be extracted by parsing the encoded stream, which would be inefficient and slow.

Accordingly, there is a need to enhance storage methods in order to address new capabilities provided by emerging video coding standards, and to address existing limitations of those storage methods.

Summary of the invention

Parameter set metadata identifying parameter sets of portions of the multimedia data is created. In addition, a file associated with the multimedia data is formed. This file includes parameter set metadata and other information related to the multimedia data.

Brief description of the drawings

The present invention is illustrated in the accompanying drawings by way of example and not limitation, and like reference numerals refer to similar elements in the drawings, in which:

Figure 1 is a block diagram of one embodiment of an encoding system;

Figure 2 is a block diagram of one embodiment of a decoding system;

Figure 3 is a block diagram of a computer environment suitable for practicing the invention;

4 is a flowchart of a method for storing subsampled metadata on an encoding system;

5 is a flowchart of a method for using subsampled metadata on a decoding system;

Figure 6 illustrates an extended MP4 media stream model with subsampling;

7A-7K illustrate exemplary data structures for storing subsampling metadata;

8 is a flowchart of a method for storing parameter set metadata on an encoding system;

9 is a flowchart of a method for using parameter set metadata on a decoding system;

10A-10E illustrate exemplary data structures for storing parameter set metadata;

Figure 11 illustrates an exemplary enhanced group of pictures (GOP);

Figure 12 is a flow diagram for storing sequence metadata on an encoding system;

13 is a flowchart of a method for using sequence metadata on a decoding system;

14A-14E illustrate exemplary data structures for storing sequence metadata;

Figures 15A and 15B illustrate the use of switching sample sets for bitstream switching;

Figure 15C is a flowchart of one embodiment of a method for determining a point at which to perform a switch between two bitstreams;

Figure 16 is a flowchart of a method for storing switched sample metadata on an encoding system;

17 is a flowchart of a method for using switched sample metadata on a decoding system;

Figure 18 illustrates an exemplary data structure for storing switching sample metadata;

Figures 19A and 19B illustrate the use of switched sample sets to simplify random access entry points into the bitstream;

Figure 19C is a flowchart of one embodiment of a method for determining random access points for samples;

Figures 20A and 20B illustrate the use of switched sample sets to simplify error recovery; and

Figure 20C is a flowchart of one embodiment of a method for simplifying error recovery when sending samples.

Detailed description of the invention

In the following detailed description of embodiments of the invention, reference is made to the accompanying drawings, in which like reference numerals indicate like elements, and in which are shown by way of illustration specific embodiments in which The present invention can be implemented in the embodiments. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the invention, and it will be understood that other embodiments may be utilized and that logical, mechanical, and other embodiments may be made without departing from the scope of the invention. physical, electrical, functional and other changes. Accordingly, the following detailed description should not be taken in a limiting sense, but the scope of the invention should be defined only by the appended claims.

Overview

From an overview of the operation of the present invention, Figure 1 illustrates one embodiment of an encoding system 100 . The encoding system 100 includes: a media encoder 104 , a metadata generator 106 and a file creator 108 . The media encoder 104 receives data that may contain video data (e.g., video objects created from a natural source video scene and other external video objects), audio data (e.g., audio objects created from a natural source audio scene and other external audio objects) , composite objects, or any combination of the above. Media encoder 104 may consist of many individual encoders or include sub-encoders to process various types of media data. The media encoder 104 encodes the media data and passes it to the metadata generator 106 . The metadata generator 106 generates metadata providing information about media data according to a media file format. The media file format may be derived from ISO media file format (or any of its variants, such as MPEG-4, JPEG 2000, etc.), QuickTime or any other media file format, and also includes some additional data structures. In one embodiment, additional data structures are defined to store metadata related to sub-samples within the media data. In another embodiment, an additional data structure is defined to store metadata linking parts of the media data (e.g., samples or subsamples) to corresponding parameter sets containing parameters that have traditionally been stored in the media data Decode the message. In yet another embodiment, additional data structures are defined to store metadata related to various groups of samples within metadata created from interdependencies of samples in the media data. In yet another embodiment, an additional data structure is defined to store metadata related to switching sample sets associated with the media data. A switched sample set refers to a set of samples that have the same decoded value but can depend on different samples. In other embodiments, various combinations of additional data structures are defined in the file format being used. These additional data structures and their functions are described in more detail below.

The file creator 108 stores metadata in files whose structure is defined by the media file format. In one embodiment, the file contains both encoded media data and metadata related to that media data. Alternatively, the encoded media data is partially or fully contained in a separate file and linked to the metadata by reference (eg, via a URL) contained in the metadata file. Files created by file creator 108 are available on channel 110 for storage or transmission.

FIG. 2 illustrates one embodiment of a decoding system 200 . The decoding system 200 includes: a metadata extractor 204, a media data stream processor 206, a media decoder 210, a synthesizer 212 and a renderer. Decoding system 200 may reside on a client device and be used for local playback. Alternatively, the decoding system 200 may be used to stream data and have a server portion and a client portion that communicate with each other over a network (eg, the Internet) 208 . The server portion may include a metadata extractor 204 and a media data stream processor 206 . The client portion may include a media decoder 210 , a compositor 212 and a renderer 214 .

Metadata extractor 204 is responsible for extracting metadata from files stored in database 216 or receiving metadata over the network (from encoding system 100). The file may or may not include media data associated with the metadata being extracted. Metadata extracted from a file includes one or more of the additional data structures described above.

The extracted metadata is passed to the media data stream processor 206, which also receives the associated encoded media data. Media data stream processor 206 utilizes the metadata to form a media data stream to be sent to media decoder 210 . In one embodiment, the media data stream processor 206 utilizes metadata related to the subsamples to locate the subsamples in the media data (eg, for packetization). In another embodiment, the media data stream processor 206 utilizes metadata associated with parameter sets to link portions of media data to their corresponding parameter sets. In yet another embodiment, the media data stream processor 206 utilizes metadata defining various sample groups within the metadata to access samples in a certain group (e.g., for scaling by discarding groups containing samples in response to Depending on transmission conditions, no other samples depend on the sample to reduce the transmitted bit rate). In yet another embodiment, the media data stream processor 206 utilizes the metadata defining the set of switched samples to locate switched samples that have the same decoding value as the sample that should be switched to, but independent of those samples that this resulting sample will depend on ( For example, to allow switching to streams with different bitrates on P-frames or B-frames).

Once the media data stream is formed, it is sent to the media decoder 210 for decoding, either directly (eg, for local playback) or over the network 208 (eg, for streaming data). A compositor 212 receives the output of the media decoder 210, and a renderer 214 composes the scene which will then be rendered on the user display device.

The following description of FIG. 3 is intended to provide an overview of computer hardware and other operating components suitable for implementing the invention, and is not intended to limit the applicable environments. FIG. 3 illustrates one embodiment of a computer system suitable for use as metadata generator 106 and/or file creator 108 of FIG. 1 or metadata extractor 204 and/or media data stream processor 206 of FIG. 2 .

Computer system 340 includes processor 350 coupled to system bus 365 , memory 355 and input/output capabilities 360 . Memory 355 is configured to store instructions that, when executed by processor 350, perform the methods described herein. Input/output 360 also encompasses various types of computer-readable media, including any type of storage accessible by processor 350 . Those skilled in the art will immediately recognize that the term "computer-readable medium/media" also encompasses a carrier wave encoding a data signal. It should also be appreciated that system 340 is under the control of operating system software executing in memory 355 . Input/output and related media 360 store computer-executable instructions for the operating system and methods of the present invention. Each of the metadata generator 106, file creator 108, metadata extractor 204, and media stream processor 206 shown in FIGS. 1 and 2 may be separate components coupled to processor 350, or may It is embodied by computer-executable instructions executed by processor 350 . In one embodiment, computer system 340 may be part of or coupled to an ISP (Internet Service Provider) via input/output 360 for sending or receiving media data over the Internet. It will be apparent that the present invention is not limited to Internet access and Web-based Internet sites; it is also intended to include directly coupled networks and private networks.

It will be appreciated that computer system 340 is but one example of many possible computer systems having different configurations. A typical computer system will usually include at least a processor, memory, and a bus coupling the memory to the processor. Those skilled in the art will immediately recognize that the invention can be practiced using other computer system configurations, including multiprocessor systems, microcomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

Subsampling for accessibility

4 and 5 illustrate the processes performed by the encoding system 100 and decoding system 200, respectively, for storing and retrieving sub-sampled metadata. The process may be performed by processing logic that may include: hardware (eg, circuitry, dedicated logic, etc.), software (eg, running on a general-purpose computer system or a dedicated machine), or a combination of both. For a software-implemented process, the illustration of the flowchart enables one skilled in the art to develop such a program containing instructions for processing on a suitably configured computer (a computer's processor that executes instructions from a computer-readable medium, including a memory ) to execute the process. Computer-executable instructions may be written in a computer programming language, or may be embodied in firmware logic. If written in a programming language conforming to recognized standards, such instructions can be executed on a variety of hardware platforms and interfaces to a variety of operating systems. In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be appreciated that various programming languages can be used to implement the teachings described herein. Furthermore, it is common in the art to speak of software in one form or another (eg, program, method, process, application, module, logic, etc.) when taking an action or producing a result. Such expressions are simply a shorthand way of saying that execution of the software by a computer causes the processor of the computer to perform an action or produce a result. It will be appreciated that more or fewer operations may be incorporated into the processes illustrated in FIGS. 4 and 5 without departing from the scope of the invention, and that the blocks described and illustrated herein The layout scheme does not imply a particular order.

FIG. 4 is a flowchart of one embodiment of a method 400 for creating subsampled metadata on encoding system 100 . Initially, method 400 begins with processing logic that receives a file with encoded media data (processing block 402). Next, processing logic extracts information identifying boundaries of subsamples in the media data (processing block 404). Depending on the file format being used, the smallest unit of a data stream to which time attributes can be attached is called: a sample (as defined by the ISO media file format or QuickTime), an access unit (as defined by MPEG-4), or a picture (as defined by MPEG-4) as defined by the JVT), etc. Subsamples represent successive portions of the data stream below the sampling layer. The definition of a subsample depends on the encoding format, but in general, a subsample is a meaningful subunit of sampling that can be encoded as a single entity or as a combination of subunits in order to obtain a partial reconstruction of the sample. Subsampling may also be referred to as an access unit fragment. Often, a subsample represents a division of the sampled data stream such that each subsample has little or no dependence on other subsamples within the same sample. For example, in JVT, subsampling is a NAL packet. Likewise, for MPEG-4 video, the subsamples would be video packets.

In one embodiment, encoding system 100 operates on the network abstraction layer defined by the JVT described above. A JVT media data stream consists of a series of NAL packets, where each NAL packet (also called a NAL unit) includes a header part and a payload part. One type of NAL packet is used to contain encoded VCL data per time slice, or to contain a single partition of data for a time slice. In addition, the NAL packet may be an information packet containing a Supplemental Enhancement Information (SEI) message. The SEI message represents optional data to be used when decoding the corresponding slot. In JVT, a subsample may be a complete NAL packet with both header and payload.

In processing block 406, processing logic creates subsampling metadata that defines subsampling in the media data. In one embodiment, subsampling metadata is grouped into a set of predetermined data structures (eg, a set of boxes). The predetermined set of data structures may include: a data structure containing information about the size of each subsample, a data structure containing information about the total number of subsamples in each sample, a data structure containing information describing each subsample (e.g., what defines is a subsampling), or any other data structure containing subsampling-related data.

Next, in one embodiment, processing logic determines whether any data structure contains a repeating sequence of data (decision block 408). If so, processing logic converts each repetition of the data into a reference to the occurrence of the sequence and the number of occurrences of the repetition (processing block 410).

Then, in processing block 412, processing logic incorporates the subsampled metadata into files associated with the media data using a particular media file format (eg, JVT file format). Depending on the media file format, the subsampling metadata can be stored together with the sampling metadata (for example, the subsampling data structure can be grouped into a sample table box containing the sampling data structure), or it can be stored separately from the sampling metadata stand up.

FIG. 5 is a flowchart of one embodiment of a method 500 for using subsampled metadata on the decoding system 200 . Initially, method 500 begins with processing logic that receives a file associated with encoded media data (processing block 502). The file may be received from a database (local or external), from the encoding system 100, or from any other device on the network. The file includes subsampling metadata defining subsamples in the media data.

Next, processing logic extracts subsampling metadata from the file (processing block 504). As discussed above, subsampling metadata can be stored in a set of data structures (eg, a set of boxes).

Additionally, in processing block 506, processing logic utilizes the extracted metadata to identify sub-samples in the encoded media data (stored in the same file or in different files) and combine the various sub-samples into Packets sent to a media decoder, thereby enabling flexible packetization of media data for streaming (eg, to support error resilience, scalability, etc.).

An exemplary subsampling metadata structure will now be described with reference to an extended ISO media file format, referred to as extended MP4. It will be apparent to those skilled in the art that other media file formats are readily extended to incorporate similar data structures for storing sub-sampled metadata.

Figure 6 illustrates an extended MP4 media stream model with subsampling. Display data (eg, a display including simultaneous audio and video) is represented by movie 602 . The movie 602 includes a set of tracks 604 . Each track 604 represents a stream of media data. Each track 604 is divided into samples 606 . Each sample 606 represents a unit of media data at a particular point in time. Sample 606 is also divided into sub-samples 608 . In the JVT standard, subsamples 608 may represent NAL packets or units, such as a single time slice of a picture, one data partition of a time slice with multiple data partitions, an in-band parameter set, or a packet of SEI information. Alternatively, subsamples 606 may represent any other structural elements of sampling, such as encoded data representing spatial or temporal regions in the media. In one embodiment, any partition of the encoded media data according to some structural or semantic criteria can be treated as a sub-sample.

7A-7L illustrate exemplary data structures for storing subsampling metadata.

Referring to FIG. 7A, the sample table box 700 containing the sample metadata box defined by the ISO media file format is extended to include items such as a subsample size box 702, a subsample description association box 704, a subsample-sample box 706, and a subsample description box 708. Subsampling access boxes like . In one embodiment, the use of subsampling access boxes is optional.

Referring to FIG. 7B , for example, samples 710 may be divided into time slices such as time slice 712 , data partitions such as partition 714 , and regions of interest (ROIs) such as ROI 716 . Each of these examples represents a different kind of sampling-to-subsampling division. Subsamples within a single sample can have different sizes.

The subsample size box 718 contains: a version field specifying the version of the subsample size box 718, a subsample size field specifying the default subsample size, a subsample count field providing the number of subsamples in a track, and a subsample count field specifying the The size of the entry size field. If the subsample size field is set to 0, then the subsample has a different size stored in the subsample size table 720 . If the subsample size field is not set to 0, it specifies a constant subsample size indicating that the subsample size table 720 is empty. Table 720 may have a fixed size of 32 bits or a variable length field for representing the subsample size. If the field is variable length, the subsample table contains a field indicating the byte length of the subsample size field.

Referring to FIG. 7C , the subsampling-sampling box 722 includes a version field specifying the version of the subsampling-sampling box 722 and an entry count field providing the number of entries in the table 723 . Each entry in the subsample-sample table contains: a first-sample field that provides the index of the first sample in the streaming process for those samples that share the same number of subsample-per-samples, and the streaming process that provides the sample The number of subsamples in each sample within the subsample-per-sample field.

Table 723 can be used to find the total number of subsamples in a track by counting how many samples are being streamed, multiplying this value by the appropriate subsamples-per-sample, and adding up the results for all streams.

Referring to FIG. 7D, the subsample description association box 724 includes a version field specifying the version of the subsample description association box 724, a description type identifier indicating the type of subsample (e.g., NAL packet, region of interest, etc.) being described, and a provision An entry count field for the number of entries in table 726. Each entry in table 726 includes: a subsample description type identifier field indicating the subsample description ID, and an index giving the first subsample in the streaming of those subsamples sharing the same subsample description ID The first subsample field of .

The subsample description type identifier controls the use of the subsample description ID field. That is, depending on the type specified in the description type identifier, the subsample description ID field itself may specify a description ID directly encoding the subsample description inside the ID itself, or the subsample description ID field may serve as a link to a different table ( i.e., the index of the subsampling description table) as described below? For example, if the description type identifier represents a JVT description, the subsample description ID identifier field may include a code specifying the nature of the JVT subsample. In this case, the Subsample Description ID Identifier field may be a 32-bit field with the least significant 8 bits used as a bitmask to indicate the presence of a predetermined data partition within the subsample, and a field to indicate the NAL packet type Or higher order 24 bits for future extensions.

Referring to FIG. 7E, the subsampling description box 728 includes: a version field specifying the version of the subsampling description box 728; an entry count field providing the number of entries in the table 730; a description type identifier field providing the description type of the subsampling description field, The subsample description field provides information about the characteristics of the subsample; and a table containing one or more subsample description entries 730 . The subsample description type identifies the type to which the descriptive information relates, and corresponds to the same field in the subsample description association table 724 . Each entry in table 730 contains a subsample description entry with information about the characteristics of the subsample that is associated with this description entry. The information and format of the description entry depends on the description type field. For example, when the description type is a parameter set, then each description entry will contain the value of that parameter set.

The descriptive information may relate to parameter set information, ROI related information or any other information needed to characterize the subsampling. For parameter sets, the subsample description association table 724 indicates the parameter set associated with each subsample. In this case the subsample description ID corresponds to the parameter set identifier. Also, subsampling can represent different regions of interest as described below. The sub-sampling is defined as one or more coded macroblocks, and then the sub-sampling description association table is used to represent the division of the coded macroblocks into image frames or images in different regions. For example, coded macroblocks in a frame can be divided into foreground and background macroblocks with two subsample description IDs (e.g., subsample description IDs 1 and 2) to indicate assignments to foreground and background regions, respectively.

Figure 7F illustrates different types of subsampling. Subsampling can represent: time slice without partition 732, time slice with multiple data partitions 734, header within a time slice 736, data partition in the middle of a time slice 738, data partition at the end of a time slice 740, SEI information packet 742 etc. Each of these subsampling types may be associated with a particular value of the 8-bit mask 744 shown in FIG. 7G. As discussed above, an 8-bit mask can form the 8 least significant bits of the 32-bit subsample description ID field. Figure 7H illustrates a subsample description association box 724 with a description type identifier equal to "jvtd. Table 726 includes a 32-bit subsample description ID identifier field that stores the value illustrated in Fig. 7G.

7H-7K illustrate data compression in subsampling description association tables.

Referring to FIG. 71 , the uncompressed table 726 includes a sequence 750 of subsample description IDs that repeat the sequence 748 . In compressed table 746, repeated sequence 750 has been compressed into a reference to sequence 748 and the number of occurrences of this sequence.

In one embodiment illustrated in FIG. 7J , it is possible to use the most significant bit of a sequence occurrence as the run of the sequence flag 754, use its next 23 bits as the occurrence index 756, and use its least significant bit as As occurrence length 758, the sequence occurrence is encoded in the subsample description ID identifier field. If flag 754 is set to 1, it means that this entry is a repeated sequence. Otherwise, this entry is the subsampling description ID. Occurrence index 756 is the subsample description index in association box 724 for the first occurrence of the sequence, and length 758 indicates the length of the repeated sequence occurrence.

In another embodiment, illustrated in FIG. 7K , repeated sequence occurrences are represented using a repeat sequence occurrence table 760 . The most significant bit of the subsample description ID field is used as the run of the sequence flag 762, indicating whether the entry is a subsample description ID, or as the sequence index of an entry in a repeating sequence occurrence table 760, which is part of the subsampling description association box 724. The repeat sequence occurrence table 760 includes an occurrence index field specifying the index in the subsample description association box 724 of the first item in the repeat sequence, and a length field specifying the length of the repeat sequence.

parameter set

In some media formats, such as JVT, the "header" information containing the critical control values needed for proper decoding of the media data is separated/decoupled from the remainder of the encoded data and stored in the parameter set among. The encoded data is then able to use mechanisms such as unique identifiers to refer to the necessary set of parameters, rather than mixing these control values with the encoded data in the stream. This approach decouples the sending of high-level encoding parameters from the encoded data. At the same time, redundancy is also reduced by sharing a common set of control values as a parameter set.

To support efficient transmission of stored media streams using parameter sets, a sender or player must be able to quickly link the encoded data to the corresponding parameters in order to know when and where the parameter sets must be sent or accessed. One embodiment of the present invention provides this capability by storing data specifying the association between parameter sets and corresponding portions of media data as parameter set metadata in a media file format.

8 and 9 illustrate the processes performed by encoding system 100 and decoding system 200 for storing and retrieving parameter set metadata, respectively. The processes may be performed by processing logic that may comprise hardware (eg, circuitry, dedicated logic, etc.), software (eg, run on a general purpose computer system or a dedicated machine), or a combination of both.

FIG. 8 is a flowchart of one embodiment of a method 800 for creating parameter set metadata at encoding system 100 . Initially, method 800 begins with processing logic that receives a file with encoded media data (processing block 802). The file includes a set of encoding parameters specifying how portions of the media data are to be decoded. Next, processing logic examines the relationship between a set of encoding parameters, called a parameter set, and the corresponding portion of the media data (processing block 804), and utilizes the media data portion to create the defined parameter set and its associated parameter set metadata ( Processing block 806). The media data portions may be represented in samples or subsamples.

In one embodiment, parameter set metadata is organized into a set of predetermined data structures (eg, a set of boxes). The predetermined set of data structures may include data structures containing descriptive information about the parameter sets, and data structures containing information defining associations between samples and corresponding parameter sets. In one embodiment, the predetermined set of data structures further comprises: data structures containing information defining associations between subsamples and corresponding parameter sets. The data structure containing information on associations between subsamples and parameter sets may or may not override the data structure containing information on associations between samples and parameter sets.

Next, in one embodiment, processing logic determines whether any parameter set data structures contain repeating sequences of data (decision block 808). If this determination is positive, processing logic converts each repeated sequence of data into a reference to the sequence occurrence and the number of occurrences of the sequence (processing block 810).

Then, in processing block 812, processing logic incorporates the parameter set metadata into a file associated with the media data using a particular media file format (eg, the JVT file format). Depending on the media file format, parameter set metadata may be stored together with track metadata and/or sample metadata (e.g., data structures containing descriptive information about parameter sets may be grouped into track boxes, and A data structure containing associated information is subsumed into a sample table box), or the parameter set metadata is stored independently of the track metadata and/or sample metadata.

FIG. 9 is a flowchart of one embodiment of a method 900 for using parameter set metadata on the decoding system 200 . Initially, method 900 begins with processing logic that receives a file associated with encoded media data (processing block 902). The file may be received from a database (local or external), from the encoding system 100, or from any other device on the network. The file includes parameter set metadata defining parameter sets of media data and associations between parameter sets and corresponding portions of media data (eg, corresponding samples or subsamples).

Next, processing logic extracts parameter set metadata from the file (processing block 904). As discussed above, parameter set metadata can be stored in a set of data structures (eg, a set of boxes).

Additionally, in processing block 906, processing logic utilizes the extracted metadata to determine which parameter set is associated with a particular media data portion (eg, sample or sub-sample). This information can then be used to control the transmission time of media data parts and corresponding parameter sets. That is, the set of parameters to be used to decode a particular sample or subsample must be sent before or with the packet containing the sample or subsample.

Thus, the use of parameter set metadata enables independent transmission of parameter sets on a more reliable channel, reducing the chance of errors or data loss where part of the media stream is lost.

An exemplary parameter set metadata structure will now be described with reference to the extended ISO media file format (referred to as extended ISO). It should be noted, however, that other media file formats may also be extended to incorporate various data structures for storing parameter set metadata.

10A-10E illustrate exemplary data structures for storing parameter set metadata.

Referring to FIG. 10A , a track box 1002 containing a track metadata box defined in the ISO file format is extended to include a parameter set description box 1004 . Additionally, the Sample Table box 1006 containing the Sample Metadata box defined in the ISO file format is extended to include a Sample to Parameter Set box 1008 . In one embodiment, the sample list box 1006 includes a subsample to parameter set box, which may override the sample to parameter set box 1008 as will be discussed in more detail below.

In one embodiment, parameter set metadata boxes 1004 and 1008 are mandatory. In another embodiment, only the parameter set description box 1004 is mandatory. In yet another embodiment, all parameter set metadata boxes are arbitrary.

Referring to Figure 10B, the parameter set description box 1010 contains: a version field specifying the version of the parameter set description box 1010, a parameter set description count field to provide the number of entries in the table 1012, and a parameter set entry containing an entry for the corresponding parameter set itself field.

Parameter sets can be referenced from a sampling layer or a subsampling layer. Referring to FIG. 10C , a sample to parameter set box 1014 provides a reference to a parameter set from the sampling layer. Sampled to parameter set box 1014 includes a version field specifying the version of sampled to parameter set box 1014 , a default parameter set ID field specifying a default parameter set ID, an entry count field providing the number of entries in table 1016 . Each entry in table 1016 contains a first sample field that provides the index of the first sample in the run of those samples that share the same parameter set, and a parameter set index that specifies the index to parameter set description box 1010 . If the default parameter set ID is equal to 0, then the sample has a different parameter set stored in table 1016. Otherwise, constant parameter settings are used and no array follows.

In one embodiment, the data in table 1016 is compressed by converting each repeating sequence into a reference to the original sequence and the number of occurrences of this sequence, as discussed in more detail above in connection with subsampling description association tables.

A parameter set can be referenced from a subsampling layer by defining an association between the parameter set and the subsampling. In one embodiment, the association between the parameter set and the sub-sampling is defined using the above-mentioned sub-sampling description association box. FIG. 10D illustrates a subsampling description association box 1018 with a description type identifier referencing a parameter set (eg, description type identifier equal to "parse"). The subsample description ID in table 1020 indicates the index in parameter set description box 1010 according to this description type identifier.

In one embodiment, the subsample description association box 1018 reloads the sample to parameter set box 1014 when present with a description type identifier referencing the parameter set.

A parameter set may vary between when the parameter set is created and when the parameter set is used to decode the corresponding portion of media data. If such changes occur, decoding system 200 receives parameter update packets specifying changes to the parameter set. The parameter set metadata includes data identifying the state of the parameter set before and after the update.

Referring to FIG. 10E , parameter set description box 1010 includes an entry for initial parameter set 1022 created at time t0 and an entry for updated parameter set 1024 created in response to parameter update packet 1026 received at time t1 . A subsample description association block 1018 associates two parameter sets with corresponding subsamples.

sampling group

While samples within a track may have various logical groupings (partitions) of samples grouped into sequences that represent high-level structures in the media data, existing file formats provide no means for representing and storing such groupings convenience mechanism. For example, advanced encoding formats such as JVT group samples within a single track into groups according to their interdependence. These groups (referred to herein as sequences or sample groups) can be used to identify chains of samples that can be processed arbitrarily when network conditions require it, thereby supporting temporal scalability. Storing metadata defining groups of samples in a file format enables senders of media to implement the above features easily and efficiently.

An example of a sample group is a set of samples whose inter-frame dependencies allow them to be decoded independently of other samples. In JVT, such groups of samples are called Enhanced Group of Pictures (Enhanced GOP). In an enhanced GOP, samples may be divided into subsequences. Each subsequence consists of a set of samples that are interdependent and can be treated as a unit. In addition, the samples of an enhanced GOP can be structured hierarchically into layers such that samples in higher layers are only predicted from samples in lower layers, thereby allowing processing of the highest layer's sampling. The lowest layer that includes those samples that do not depend on samples in any other layer is called the base layer. Any other layer than the base layer is called an enhancement layer.

FIG. 11 illustrates an exemplary enhanced GOP in which samples are divided into two layers—a base layer 1102 and an enhancement layer 1104 , and two subsequences 1106 and 1108 . Each of the two subsequences 1106 and 1108 can be discarded independently of each other.

12 and 13 illustrate the processes performed by encoding system 100 and decoding system 200, respectively, for storing and retrieving sample group metadata. The process may be comprised of processing logic that may include hardware (eg, circuitry, dedicated logic, etc.), software (eg, run on a general-purpose computer system or a dedicated machine), or a combination of both.

FIG. 12 is a flowchart of one embodiment of a method 1200 for creating sample group metadata on encoding system 100 . Initially, method 1200 begins with processing logic that receives a file having encoded media data (processing block 1202). The samples within a track of media data have certain interdependencies. For example, the track may include: I-frames that do not depend on any other samples, P-frames that depend on a single previous sample, and B-frames that depend on two previous samples, including I-frames, P-frames, and B-frames any combination of . Samples in a track can be logically grouped into sample groups (eg, enhanced GOPs, layers, subsequences, etc.) according to their interdependencies.

Next, processing logic examines the media data to identify sample groups in each track (processing block 1204), and creates sample group metadata describing the sample groups, and defines which samples will be included in each sample group (processing block 1204). block 1206). In one embodiment, sample group metadata is grouped into a set of predetermined data structures (eg, a set of boxes). The predetermined set of data structures may include: a data structure containing descriptive information about each sample group and a data structure containing information identifying what is contained in each sample group.

Next, in one embodiment, processing logic determines whether any of the sample packet data structures contain repeating sequences of data (decision block 1208). If this determination is positive, then processing logic converts each repeated sequence of data into a reference to the sequence occurrence and the number of sequence occurrences (processing block 1210).

Then, at processing block 1212, processing logic incorporates the sample group metadata into a file associated with the media data using a particular media file format (eg, the JVT file format). Depending on the media file format, the sample group metadata can be stored together with the sample metadata (for example, a sample group data structure can be included into the sample table box), or it can be stored independently of the sample metadata stand up.

FIG. 13 is a flowchart of one embodiment of a method 1300 for using sample group metadata on the decoding system 200 . Initially, method 1300 begins with processing logic that receives a file associated with encoded media data (processing block 1302). The file may be received from a database (local or external), from the encoding system 100, or from any other device on the network. The file includes sample group metadata defining sample groups in the media data.

Next, processing logic extracts sample group metadata from the file (processing block 1304). As discussed above, sampling group metadata can be stored in a data structure group (eg, a group of boxes).

Furthermore, at processing block 1306, processing logic utilizes the extracted sample group metadata to identify chains of samples that can be processed without affecting the ability to decode other samples. In one embodiment, this information can be used to access samples in a particular sample group and to determine which samples can be discarded in response to changes in network capabilities. In other embodiments, the samples are filtered using the sample group metadata so that only a portion of the samples in the track are processed or rendered.

Thus, sample group metadata facilitates selective access and scalability to samples.

An exemplary sample group metadata structure will now be described with reference to an extended ISO media file format, referred to as extended MP4. It should be noted, however, that other media file formats may also be extended to incorporate various data structures for storing sample group metadata.

14A-14E illustrate exemplary data structures for storing sample group metadata.

Referring to FIG. 14A , the sample table box 1400 containing the sample metadata box defined by MP4 is expanded to include a sample group box 1402 and a sample group description box 1404 . In one embodiment, sample group metadata boxes 1402 and 1404 are arbitrary.

Referring to Figure 14B, the sample group box 1406 is used to find the set of samples contained in a particular sample group. Multiple instances of the sample group box 1406 are allowed to correspond to different types of sample groups (eg, enhanced GOP, subsequence, layer, parameter set, etc.). The sample group box 1406 contains: a version field specifying the version of the sample group box 1406, an entry count field to provide the number of entries in the table 1408, a sample group identifier field to identify the sample group type, The first sample field of the index of the first sample in the streaming process of those samples, and the sample group description index specifying the index to the sample group description box.

Referring to Figure 14C, the sample group description box 1410 provides information about the characteristics of the sample group. The sample group description box 1410 contains: a version field specifying the version of the sample group description box 1410, an entry count field to provide the number of entries in the table 1412, a sample group identifier field to identify the sample group type, and a sample group identifier field to provide The sample group description field of the sample group descriptor.

Referring to Figure 14D, the use of the sample group box 1416 of the layer ("layr") sample group type is illustrated. Samples 1 to 11 are divided into three layers according to their interdependence. In layer 0 (the base layer), samples (samples 1, 6, and 11) all only depend on each other, not on samples in any other layer. In layer 1, samples (samples 2, 5, 7, 10) depend on samples in lower layers (ie, layer 0) and samples within this layer 1. In layer 2, samples (samples 3, 4, 8, 9) depend on samples in lower layers (layers 0 and 1) and samples within this layer 2. Thus, layer 2 samples can be arranged without affecting the ability to decode samples from lower layers 0 and 1 .

The data in the Samples group box 1416 illustrates the aforementioned association between samples and the layer. As shown, this data includes repeated layer patterns 1414 that can be compressed by converting each repeated layer pattern into a reference to the initial layer pattern and the number of occurrences of the pattern, as detailed above as discussed.

Referring to Figure 14E, the use of the sample group box 1418 of the subsequence ("sseq") sample group type is illustrated. Samples 1 to 11 are divided into four subsequences according to their interdependencies. With the exception of subsequence 0 at level 0, each subsequence includes samples on which no other subsequence depends. Therefore, samples in subsequences can be arranged as units if necessary.

The data in Samples group box 1418 describes the association between samples and subsequences. This data allows random access to samples at the beginning of the corresponding subsequence.

stream switching

In a typical streaming situation, one of the key requirements is to scale the bitrate of the compressed data in response to changing network conditions. A simple way to achieve this is to encode multiple streams with different bitrate and quality settings for typical network conditions. Servers can then be switched among these pre-encoded streams in response to network conditions.

The JVT standard provides a new type of picture called a cutout picture that allows one picture to reconstruct the other equally without requiring both pictures to use the same frame for prediction. Specifically, JVT provides two types of switching pictures: SI pictures, which are like I-frames, are coded independently of any other pictures; and SP pictures, which are coded with reference to other pictures. Switching between streams with different bitrate and quality settings in response to changing delivery conditions can be achieved using a toggle screen to provide error resilience and enable trick mode like fast forward and rewind. ).

However, in order to effectively switch scenes using the JVT when implementing stream switching, error resilience, stick mode, and other features, the player must know which samples in the stored media data have alternative representations and their dependencies are What. Existing file formats do not provide this capability.

One embodiment of the present invention addresses the above limitations by defining a switching sample set. A switched sample set represents a set of samples whose decoded values are equal, but which may use different reference samples. A reference sample is a sample used to predict the value of another sample. Each member of a switched sample set is called a switched sample. Figure 15A illustrates the use of switched sample sets for bitstream switching.

Referring to FIG. 15A, stream 1 and stream 2 are two encodings of the same content with different quality and bit rate parameters. Sample S12 is an SP picture that does not appear in every stream, and it is used to realize switching from stream 1 to stream 2 (switching is a directional property). Samples S12 and S2 are included in the switching sample set. Both S1 and S12 are predicted from sample P12 in track 1, while S2 is predicted from sample P22 in track 2. Although samples S12 and S2 use different reference samples, their decoded values are equal. Thus, switching from stream 1 to stream 2 can be achieved by switching sample S12 (at sample 1 in stream 1 and at S2 in stream 2).

16 and 17 illustrate the processes performed by the encoding system 100 and the decoding system 200, respectively, for storing and retrieving switching sample metadata. The process may be performed by processing logic that may include hardware (eg, circuitry, dedicated logic, etc.), software (eg, executed on a general-purpose computer system or a dedicated machine), or a combination of both.

FIG. 16 is a flowchart of one embodiment of a method 1600 for creating switching sample metadata on encoding system 100 . Initially, method 1600 begins with processing logic that receives a file with encoded media data (processing block 1602). The file includes one or more optional encodings for the media data (eg, different bandwidth and quality settings for typical network conditions). The optional encoding includes one or more switching pictures. Such scenes may be included within optional media data streams, or as separate entities implementing specific features such as error resilience or gearshift modes. The method for creating these tracks and cutscenes is not specified by the present invention, but various possibilities will be apparent to those skilled in the art. For example, a periodic (eg, every second) setting for switching samples between each pair of tracks containing optional encoding.

Next, processing logic examines the file to create a switched sample set (processing block 1604) that includes those samples that have the same decoded value when a different reference sample is used, and creates the Switch sample metadata and describe the samples within the switch sample set (processing block 1606). In one embodiment, the switching sample metadata is organized into a predetermined data structure, such as a table box containing a set of nested tables.

Next, in one embodiment, processing logic determines whether the toggle sample metadata structure contains a repeating sequence of data (decision block 1608). If this determination is positive, then processing logic converts each repeated sequence of data into a reference to the sequence occurrence and the number of sequence occurrences (processing block 1610).

Then, in processing block 1612, processing logic includes switched sample metadata into the file associated with the media data using a particular media file format (eg, the JVT file format). In one embodiment, switch sample metadata may be stored in a separate track marked for stream switching. In another embodiment, switching sample metadata is stored with the sample metadata (eg, a sequence data structure may be included in a sample table box).

FIG. 17 is a flowchart of one embodiment of a method 1700 for using switching sample metadata on the decoding system 200 . Initially, method 1700 begins with processing logic that receives a file associated with encoded media data (processing block 1702). The file may be received from a database (local or external), from the encoding system 100, or from any other device on the network. The file includes switch sample metadata defining a set of switch samples associated with the media data.

Next, processing logic extracts switched sample metadata from the file (processing block 1704). As discussed above, toggle sampling metadata may be stored in a data structure such as a table box containing a set of nested tables.

Additionally, in processing block 1706, processing logic utilizes the extracted metadata to find a switch sample set that contains a particular sample and select an alternative sample from the switch sample set. In response to changing network conditions, an optional sample that is the same decoded value as the original sample can be used to switch between two differently encoded bitstreams, in order to provide a random access entry point into the bitstream to facilitate error recovery etc.

An exemplary switched sample metadata structure will now be described with reference to an extended ISO media file format, referred to as extended MP4. It should be noted, however, that other media file formats can also be extended to incorporate various data structures for storing switched sample metadata.

Figure 18 illustrates an exemplary data structure for storing switching sample metadata. The exemplary data structure is in the form of a toggle sample table box comprising a set of nested tables. Each entry in table 1802 identifies a switch sample set. Each switched sample set consists of a set of switched samples whose reconstructions are objectively equivalent (or perceptually equivalent), but capable of being on the same orbit as may or may not necessarily be switched samples ( stream) to predict the switch sample set. Each entry in table 1802 is linked to a corresponding table 1804 . Table 1804 identifies each toggle sample contained in the toggle sample set. Each entry in table 1804 is also linked to a corresponding table 1806, which defines the position of the switch sample (i.e., its track number and sample number), said track contains: the reference sample used by the switch sample, The total number of reference samples used to switch samples and each reference sample used to switch samples.

As illustrated in Figure 15A, in one embodiment, switching sample metadata may be used to switch between different encoded versions of the same content. In MP4, each optional encoding is stored as an independent MP4 track, and the "optional group" in the track header indicates that it is an optional encoding for a specific content.

FIG. 15B illustrates a table according to FIG. 15A containing metadata defining a switched sample set 1502 consisting of samples S2 and S12.

Figure 15C is a flowchart of one embodiment of a method 1510 for determining the point at which switching between two bitstreams is to be performed. Assuming a switch is to be performed from stream 1 to stream 2, method 1510 begins by searching the switch sample metadata to find all switch sample sets that contain switch samples of the reference track with stream 1 and switch samples of the switch sample track with stream 2 (processing block 1512). Next, the resulting switched sample set is evaluated to select a switched sample set in which all reference samples with switched samples of the reference track of stream 1 are available (processing block 1514). For example, if the switch sample of the reference track with stream 1 is a P frame, then one sample is required to be available before switching. Additionally, a switch point is determined using the samples in the selected switch sample set (processing block 1516). That is, the switch point is considered to be the highest reference sample that will immediately follow the switch samples of the reference track with stream 1 via the switch sample of the reference track with stream 1, and until immediately following the switch sample with stream 2 The track's toggle sample-to-sample there.

In another embodiment, switching sample metadata may be used to facilitate random access to entry points in the bitstream, as illustrated in Figures 19A-19C.

Referring to Figures 19A and 19B, switch sample 1902 consists of samples S2 and S12. S2 is a P frame predicted from P22 and is used during normal stream playback. S12 is used as a random access point (for splicing). Once S12 is decoded, stream playback proceeds with decoding of P24 as if P24 was decoded after S2.

Figure 19C is a flowchart of one embodiment of a method 1910 for determining a random access point for a sample (eg, sample S on track T). Method 1910 begins by searching the switch sample metadata to find all switch sample sets that contain switch samples with switch sample track T (processing block 1912). Next, the resulting switched sample set is evaluated in order to select the switched sample set in which the switched sample with switched sample track T is the nearest sample preceding sample S in decoding order (processing block 1914). Additionally, a switched sample (sample SS) other than the switched sample with switched sample track T is selected from the selected set of switched samples as a random access point to sample S (processing block 1916). During stream playback, a sample SS (followed by any reference samples specified in the entry for the corresponding sample SS) is decoded instead of a sample S.

In yet another embodiment, switching sample metadata may be used to facilitate error recovery, as exemplified in Figures 20A-20C.

Referring to FIGS. 20A and 20B , the switching sample 2002 is composed of samples S2 , S12 and S22 . Sample S2 is predicted from sample P4. Sample S12 is predicted from sample S1. If an error occurs between samples P2 and P4, then switching sample S12 may be decoded instead of sample S2. Streaming then continues with sampling P6 as usual. If the error also affected sample S1, then switching sample S22 could be decoded instead of sample S2, and streaming would then continue with sample P6 as usual.

Figure 20c is a flowchart of one embodiment of a method 2010 for facilitating error recovery when sending samples (eg, sample S). Method 2010 begins by searching the switched sample metadata to find all switched sample sets that contain switched samples equal to or immediately following sample S in decoding order (processing block 2012). Next, the resulting switched sample set is evaluated to select the switched sample set with switched sample SS that is closest to sample S and for which it is known (via feedback or other information source) that its reference sample will be correct ( processing block 2014). Also, instead of sending samples S, switch samples SS are sent (processing block 2016).

Storage and retrieval of audiovisual metadata has been described. Although specific embodiments have been illustrated and described herein, those of ordinary skill in the art will recognize that any layout scheme suitable for accomplishing the same purpose may be substituted for the specific embodiments shown. This application is intended to cover any adaptations or variations of the present invention.

Claims (85)

1. A method comprising: parameter set metadata for creating one or more parameter sets identifying portions of the multimedia data; and A file is formed associated with the multimedia data, the file including parameter set metadata. 2. The method of claim 1, wherein each of the plurality of portions of the multimedia data is a sample within the multimedia data. 3. The method of claim 1, wherein each of the plurality of portions of the multimedia data is a sub-sample within a portion of the multimedia data. 4. The method of claim 1, wherein creating parameter set metadata comprises: receiving a file with encoded multimedia data; Examining relationships between one or more parameter sets and portions of multimedia data; and Define parameter set metadata based on checked out relationships. 5. The method of claim 1, wherein creating parameter set metadata includes organizing parameter set metadata into a set of predetermined data structures. 6. The method of claim 5, wherein creating parameter set metadata further comprises: Each repeating sequence of data within a predetermined set of data structures is converted into a reference to the sequence occurrence and the number of occurrences. 7. The method of claim 5, wherein the predetermined set of data structures comprises: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing Information about the association between a parameter set and multiple parts of multimedia data. 8. The method of claim 1, further comprising: sending the files associated with the multimedia data to the decoding system; receiving a file associated with the multimedia data on the decoding system; and Parameter set metadata is extracted from a file associated with the multimedia data at the decoding system, the extracted parameter set metadata is then used to identify any of one or more parameter sets required to decode at least a portion of the multimedia data . 9. A method comprising: receiving a file associated with the multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data; and Parameter set metadata is extracted from the file, and the extracted parameter set metadata is then used to determine a relationship between one or more parameter sets and portions of the multimedia data. 10. The method of claim 9, wherein each of the plurality of portions of multimedia data is a sample within the multimedia data. 11. The method of claim 9, wherein each of the plurality of portions of multimedia data is a sub-sample within a portion of the multimedia data. 12. The method of claim 9, further comprising: The determined relationship is utilized to control the timing of transmission of portions of multimedia data and one or more sets of parameters. 13. The method of claim 9, wherein the extracted parameter set metadata is organized into a set of predetermined data structures. 14. The method of claim 13, wherein the predetermined set of data structures comprises: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing Information about the association between a parameter set and multiple parts of multimedia data. 15. A method comprising: creating parameter set metadata identifying one or more parameter sets of portions of the multimedia data; creating sample group metadata defining a grouping of a plurality of samples within the multimedia data; and A file is formed associated with the multimedia data, the file including parameter set metadata and sample group metadata. 16. The method of claim 15, wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data. 17. The method of claim 15, wherein creating parameter set metadata comprises: organizing parameter set metadata into a predetermined set of data structures comprising: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing Information about associations between parts of multimedia data. 18. The method of claim 15, wherein the grouping is based on interdependencies of a plurality of samples. 19. The method of claim 15, wherein creating sample group metadata comprises: grouping sample group metadata into a predetermined set of data structures comprising: a first data structure containing descriptive information about a plurality of sample groups within the multimedia data; and a second data structure containing The sampled information in each of the . 20. A method comprising: receiving a file associated with the multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data and sample group metadata defining groupings of a plurality of samples within the multimedia data; and Extract parameter set metadata and sample group metadata from the file, the extracted parameter set metadata is then used to determine the relationship between one or more parameter sets and multiple parts of the multimedia data, and the extracted The sample group metadata is then used to identify samples that can be processed in future processing. 21. The method of claim 20, wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data. 22. The method of claim 20, further comprising: The determined relationship is utilized to control the timing of transmission of portions of multimedia data and one or more sets of parameters. 23. The method of claim 20, wherein grouping the extracted parameter set metadata into a set of predetermined data structures comprises: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing information defining associations between one or more parameter sets and portions of the multimedia data. 24. The method of claim 20, wherein the grouping is based on interdependencies of a plurality of samples. 25. The method of claim 20, further comprising: In response to a change in network capabilities, one or more samples are found that can be processed without affecting decoding of remaining samples of the multimedia data. 26. The method of claim 20, further comprising: Multiple samples are filtered according to the extracted sample group metadata to reduce the number of samples to be rendered. 27. The method of claim 20, wherein grouping the extracted sample group metadata into a predetermined set of data structures comprises: a first data structure containing descriptive information about a plurality of sample groups within the multimedia data information; and a second data structure containing information identifying samples in each of the plurality of sample groups. 28. A method comprising: creating parameter set metadata identifying one or more parameter sets of portions of the multimedia data; creating sample group metadata defining a grouping of a plurality of samples within the multimedia data; creating switched sample metadata defining a plurality of switched sample sets associated with the multimedia data, each of the multiple switched sample sets comprising samples having the same decoded value; and A file is formed associated with the multimedia data, the file including parameter set metadata, sample group metadata, and switch sample metadata. 29. The method of claim 28, wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data. 30. The method of claim 28, wherein creating parameter set metadata comprises: organizing parameter set metadata into a predetermined set of data structures comprising: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing Information about associations between parts of multimedia data. 31. The method of claim 28, wherein the grouping is based on interdependencies of a plurality of samples. 32. The method of claim 28, wherein creating sample group metadata comprises: grouping sample group metadata into a predetermined set of data structures comprising: a first data structure containing descriptive information about a plurality of sample groups within the multimedia data; and a second data structure containing The sampled information in each of the . 33. The method of claim 28, wherein samples in each of the plurality of switched sample sets use different reference samples. 34. The method of claim 28, wherein creating switching sample metadata comprises: The switch sampling metadata is organized into a predetermined data structure represented as a table box containing a set of nested tables. 35. A method comprising: Receiving a file associated with the multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data, sample group metadata defining groupings of a plurality of samples within the multimedia data, and defining parameters associated with the multimedia data switch sample metadata for multiple switch sample sets linked together; and extracting parameter set metadata, sample group metadata, and switching sample metadata from the file, the extracted parameter set metadata is then used to determine the relationship between one or more parameter sets and multiple parts of the multimedia data, The extracted sample group metadata is then used to identify samples that can be processed in future processing, while the extracted switch sample metadata is then used to find a surrogate for a particular sample. 36. The method of claim 35, wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data. 37. The method of claim 35, further comprising: The determined relationship is utilized to control the timing of transmission of portions of multimedia data and one or more sets of parameters. 38. The method of claim 35, wherein grouping the extracted parameter set metadata into a predetermined set of data structures comprises: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing information defining associations between one or more parameter sets and portions of the multimedia data. 39. The method of claim 35, wherein the grouping is based on interdependencies of multiple samples. 40. The method of claim 35, further comprising: In response to a change in network capabilities, one or more samples are found that can be processed without affecting decoding of remaining samples of the multimedia data. 41. The method of claim 35, further comprising: The plurality of samples are filtered according to the extracted sample group metadata to reduce the number of samples to be rendered. 42. The method of claim 35, wherein grouping the extracted sample group metadata into a predetermined set of data structures comprises: a first data structure containing descriptive information about a plurality of sample groups within the multimedia data information; and a second data structure containing information identifying samples in each of the plurality of sample groups. 43. The method of claim 35, wherein each of the plurality of switched sample sets contains samples having the same decoded value when different reference samples are used. 44. The method of claim 35, further comprising: finding a switched sample set containing a particular sample among multiple switched sample sets; and Select an optional sample from the found switch sample set. 45. The method of claim 35, wherein the extracted switched sample metadata is organized into a predetermined data structure represented as a table box containing a set of nested tables. 46. A method comprising: creating parameter set metadata identifying one or more parameter sets of portions of the multimedia data; creating switched sample metadata defining a plurality of switched sample sets associated with the multimedia data, each of the multiple switched sample sets comprising samples having the same decoded value; and A file is formed associated with the multimedia data, the file including parameter set metadata and switching sample metadata. 47. The method of claim 46, wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data. 48. The method of claim 46, wherein creating parameter set metadata comprises: organizing parameter set metadata into a predetermined set of data structures comprising: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing Information about associations between parts of multimedia data. 49. The method of claim 46, wherein samples in each of the plurality of switched sample sets use different reference samples. 50. The method of claim 46, wherein creating switching sample metadata comprises: The switch sampling metadata is organized into a predetermined data structure represented as a table box containing a set of nested tables. 51. A method comprising: receiving a file associated with the multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data and switching sample metadata defining a plurality of switching sample sets associated with the multimedia data; and Extract parameter set metadata and switching sampling metadata from the file, the extracted parameter set metadata is then used to determine the relationship between one or more parameter sets and multiple parts of the multimedia data, and the extracted Switching sample metadata is then used to find substitutes for a particular sample. 52. The method of claim 51, wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data. 53. The method of claim 51 , further comprising: The determined relationship is utilized to control the timing of transmission of portions of multimedia data and one or more sets of parameters. 54. The method of claim 51 , wherein grouping the extracted parameter set metadata into a set of predetermined data structures comprises: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing information defining associations between one or more parameter sets and portions of the multimedia data. 55. The method of claim 51, wherein each of the plurality of switched sample sets contains samples having the same decoded value when different reference samples are used. 56. The method of claim 51 , further comprising: finding a switched sample set containing a particular sample among multiple switched sample sets; and Select an optional sample from the found switch sample set. 57. The method of claim 51, wherein the extracted switched sample metadata is organized into a predetermined data structure represented as a table box containing a set of nested tables. 58. A memory for storing data accessed by an application program executing on a data processing system, comprising: a plurality of data structures stored in the memory, the plurality of data structures residing in files used by the application, the files being associated with the multimedia data and including a or parameter set metadata for multiple parameter sets. 59. The memory of claim 58, wherein the file including parameter set metadata further includes associated multimedia data. 60. The memory of claim 58, wherein the file including parameter set metadata includes a reference to a file containing associated multimedia data. 61. The memory of claim 58, wherein the plurality of data structures includes: a first data structure containing descriptive information about one or more parameter sets; and a second data structure containing Information on the association between the plurality of parameter sets and the plurality of parts of the multimedia data. 62. A memory for storing data accessed by an application program executing on a data processing system, the memory comprising: A plurality of data structures stored in the memory, the plurality of data structures residing in files used by the application, the files being associated with multimedia data and comprising: parameter set metadata defining one or more parameter sets of parts of the multimedia data, and Sample group metadata defining a grouping of a plurality of samples within the multimedia data. 63. A memory for storing data accessed by an application program executing on a data processing system, the memory comprising: A plurality of data structures stored in the memory, the plurality of data structures residing in files used by the application, the files being associated with multimedia data and comprising: parameter set metadata defining one or more parameter sets of parts of the multimedia data, sample group metadata defining a grouping of multiple samples within the multimedia data, and Switching sample metadata defining a plurality of switching sample sets associated with the multimedia data. 64. A memory for storing data accessed by an application program executing on a data processing system, the memory comprising: A plurality of data structures stored in the memory, the plurality of data structures residing in files used by the application, the files being associated with multimedia data and comprising: parameter set metadata defining one or more parameter sets of parts of the multimedia data, and Switching sample metadata defining a plurality of switching sample sets associated with the multimedia data. 65. A device comprising: a metadata generator for creating parameter set metadata identifying one or more parameter sets of portions of the multimedia data; and A file creator for forming a file associated with the multimedia data, the file including parameter set metadata. 66. The apparatus of claim 65, wherein each of the plurality of portions of the multimedia data is any one of a sample and a sub-sample within the multimedia data. 67. The apparatus as claimed in claim 65, wherein the metadata generator is configured to receive a file with encoded multimedia data, check the relationship between one or more parameter sets and a plurality of parts of the multimedia data and based on Check the relationship definition parameter set metadata to create parameter set metadata. 68. The device of claim 65, further comprising: a metadata extractor for receiving a file associated with the multimedia data at the decoding system and for extracting parameter set metadata from the file associated with the multimedia data; and A media data stream processor configured to use the extracted parameter set metadata to identify any one of the one or more parameter sets required to decode at least a portion of the multimedia data. 69. A device comprising: a metadata extractor for receiving a file associated with the multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data, and for extracting the parameter set metadata from the file; and The media data stream processor is configured to use the extracted parameter set metadata to determine the relationship between one or more parameter sets and multiple parts of the multimedia data. 70. The apparatus of claim 69, wherein each of the plurality of portions of the multimedia data is a sample or sub-sample within the multimedia data. 71. The apparatus of claim 69, wherein the media data stream processor is further configured to utilize the determined relationship to control transmission times of portions of multimedia data and one or more parameter sets. 72. A device comprising: a metadata generator for creating parameter set metadata identifying one or more parameter sets of portions of the multimedia data, and for creating sample group metadata defining groupings of samples within the multimedia data; and A file creator for forming a file associated with the multimedia data, the file including parameter set metadata and sample group metadata. 73. A device comprising: a metadata extractor for receiving a file associated with the multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data and a sample component defining a grouping of a plurality of samples within the multimedia data data, and used to extract parameter set metadata and sample group metadata from the file; and a media data stream processor, configured to use the extracted parameter set metadata to determine the relationship between one or more parameter sets and portions of the multimedia data, and to use the extracted sample group metadata to identify Samples that can be processed in future processing. 74. A device comprising: a metadata generator for creating parameter set metadata identifying one or more parameter sets of portions of the multimedia data, for creating sample group metadata defining groupings of a plurality of samples within the multimedia data, and for creating switch sample metadata defining a plurality of switch sample sets associated with the multimedia data; and A file creator for forming a file associated with the multimedia data, the file including parameter set metadata, sample group metadata, and switch sample metadata. 75. A device comprising: a metadata extractor for receiving a file associated with the multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data, a sample component defining a grouping of a plurality of samples within the multimedia data data and toggle sample metadata defining a plurality of toggle sample sets associated with the multimedia data and for extracting parameter set metadata, sample group metadata, and toggle sample metadata from the file; and A media data stream processor, configured to use the extracted parameter set metadata to determine the relationship between one or more parameter sets and multiple parts of the multimedia data, and to use the extracted sample group metadata to identify the relationship between Samples processed during future processing and used to find substitutes for specific samples using the extracted toggle sample metadata. 76. A device comprising: a metadata generator for creating parameter set metadata identifying one or more parameter sets of portions of the multimedia data, and for creating switch sample metadata defining a plurality of switch sample sets associated with the multimedia data; and A file creator for forming a file associated with the multimedia data, the file including parameter set metadata and switching sample metadata. 77. A device comprising: a metadata extractor for receiving a file associated with the multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data and switches defining a plurality of switch sample sets associated with the multimedia data sample metadata, and is used to extract parameter set metadata and switch sample metadata from the file; and a media data stream processor configured to use the extracted parameter set metadata to determine a relationship between one or more parameter sets and portions of the multimedia data, and to use the extracted switch sample metadata to find Surrogates for Specific Sampling. 78. A device comprising: means for creating parameter set metadata identifying one or more parameter sets of portions of multimedia data; and Means for forming a file associated with multimedia data, the file including parameter set metadata. 79. A device comprising: means for receiving a file associated with multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data; and Means for extracting parameter set metadata from a file, the extracted parameter set metadata being subsequently used to determine a relationship between one or more parameter sets and portions of multimedia data. 80. A device comprising: means for creating parameter set metadata identifying one or more parameter sets of portions of multimedia data; means for creating sample group metadata defining a grouping of a plurality of samples within the multimedia data; and Means for forming a file associated with multimedia data, the file including parameter set metadata and sample group metadata. 81. A device comprising: means for receiving a file associated with multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data and sample group metadata defining groupings of a plurality of samples within the multimedia data; and means for extracting parameter set metadata and sample group metadata from a file, the extracted parameter set metadata is then used to determine the relationship between one or more parameter sets and a plurality of parts of the multimedia data, and The extracted sample group metadata is then used to identify samples that can be processed in future processing. 82. A device comprising: means for creating parameter set metadata identifying one or more parameter sets of portions of multimedia data; means for creating sample group metadata defining a grouping of a plurality of samples within multimedia data; means for creating switching sample metadata defining a plurality of switching sample sets associated with the multimedia data, each of the plurality of switching sample sets comprising samples having the same decoded value; and Means for forming a file associated with multimedia data, the file including parameter set metadata, sample group metadata, and switch sample metadata. 83. A device comprising: means for receiving a file associated with multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data, sample group metadata defining groupings of a plurality of samples within the multimedia data, and definitions switching sample metadata for a plurality of switching sample sets associated with the multimedia data; and Means for extracting parameter set metadata, sample group metadata and switching sample metadata from a file, the extracted parameter set metadata being used to determine the relationship between one or more parameters and portions of multimedia data , the extracted sample group metadata is then used to identify samples that can be processed in future processing, while the extracted switch sample metadata is then used to find a surrogate for a particular sample. 84. A device comprising: means for creating parameter set metadata identifying one or more parameter sets of portions of multimedia data; means for creating switching sample metadata defining a plurality of switching sample sets associated with the multimedia data, each of the plurality of switching sample sets comprising samples having the same decoded value; and Means for forming a file associated with multimedia data, the file including parameter set metadata and switching sample metadata. 85. A device comprising: means for receiving a file associated with multimedia data, the file including parameter set metadata identifying one or more parameter sets of the multimedia data and switching sample metadata defining a plurality of switching sample sets associated with the multimedia data ;and means for extracting parameter set metadata and switching sampling metadata from a file, the extracted parameter set metadata is then used to determine the relationship between one or more parameter sets and multiple parts of the multimedia data, and The extracted switching sample metadata is then used to find a surrogate for a particular sample.

Images