KR20130053444A - Signaling data for multiplexing video components - Google Patents

Abstract

The server may provide the client with information describing the characteristics of the audio and video components, independent of the audio and video components' own encoded samples. The client may use the information to select the components and then request the selected components, e.g., in accordance with the streaming network protocol. In one example, an apparatus for transmitting encapsulated video data is a processor configured to determine characteristics for components of a plurality of representations of video content, the characteristics comprising a frame rate, a profile indicator, a level indicator, Wherein the processor comprises at least one of dependencies between elements; And sending the properties to the client device, receiving the request for at least one of the components from the client device after transmitting the properties, and sending the requested components to the client device in response to the request And one or more interfaces configured.

Description

SIGNALING DATA FOR MULTIPLEXING VIDEO COMPONENTS < RTI ID = 0.0 >

This application claims the benefit of US Provisional Patent Application No. 61 / 364,747, filed July 15, 2010, and US Provisional Patent Application No. 61 / 366,436, filed July 21, 2010, both of which are incorporated herein by reference. Each of which is incorporated herein by reference in its entirety.

The disclosure relates to the storage and transmission of encoded video units.

Digital video capabilities include digital television, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital media players, Gaming devices, video game consoles, cellular or satellite radiotelephones, teleconferencing devices, and the like. Digital video devices may use video compression techniques such as MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264 / MPEG-4, Part 10, Standards defined by AVC (Advanced Video Coding), and video compression techniques described in extensions of these standards.

Video compression techniques perform spatial and / or temporal prediction to reduce or eliminate redundancy inherent in video sequences. In block-based video coding, a video frame or slice may be partitioned into macroblocks. Each macroblock may be further partitioned. Macroblocks in an intra-coded (I) frame or slice are encoded using spatial prediction for neighboring macroblocks. Macroblocks in inter-coded (P or B) frames or slices may use spatial prediction for neighboring macroblocks in the same frame or slice, or temporal prediction for macroblocks in different reference frames.

After the video data is encoded, the video data may be packetized for transmission or storage. The video data may be assembled into any of a variety of standards, such as video files conforming to the International Organization for Standardization (ISO) based media file format and its extensions, such as AVC.

Efforts have been made to develop new video coding standards based on H.264 / AVC. One such standard is the Scalable Video Coding (SVC) standard, which is a scalable extension to H.264 / AVC. Another standard is multi-view video coding (MVC), which became a multi-view extension to H.264 / AVC. The joint draft of MVC will be held in July 2008 in Hannover, Germany and will be available at http://wftp3.itu.int/av-arch/jvt-site/2008_07_Hannover/JVT-AB204. JVT-AB204, "Joint Draft 8.0 on Multiview Video Coding. " The version of the AVC standard is the 30th JVT meeting, which is held in February 2009, CH, Genoa and available at http://wftp3.itu.int/av-arch/jvt-site/2009_01_Geneva/JVT-AD007.zip, It is described in JVT-AD007, "Editors' draft revision to ITU-T Rec. H.264 | ISO / IEC 14496-10 Advanced Video Coding-in preparation for ITU-T SG 16 AAP Consent (in integrated form)". This document includes SVC and MVC in the AVC Specification.

In general, the present disclosure describes techniques for transmitting video data over, for example, a network streaming protocol, such as Hypertext Transfer Protocol (HTTP) streaming. In some cases, the video content may include a number of possible combinations of audio data and video data. For example, the content may include a plurality of possible audio tracks (e.g., in different languages such as English, Spanish, and French) and a plurality of audio tracks (e.g., at different bit rates, different frame rates, And / or < / RTI > encoded with various other characteristics). These tracks may be referred to as components such as audio components and video components. The combination of each of the components may form a unique presentation of the multimedia content and may be delivered as a service to the client. The techniques of the present disclosure allow a server to signal the characteristics of various representations, or multimedia components, into a single data structure. In this way, the client device may retrieve its data structure and select one of the representations to request from the server, e.g., in accordance with the streaming network protocol.

In one example, a method for transmitting encapsulated video data comprises transmitting to a client device characteristics for components of a plurality of representations of video content, the characteristics including a frame rate, a profile indicator, a level indicator, The dependencies between components and the number of target output views for a 3D representation; transmitting, after sending the properties, a request for at least one of the components to the client device And sending the requested components to the client device in response to the request.

In another example, an apparatus for transmitting encapsulated video data is a processor configured to determine characteristics for components of a plurality of representations of video content, wherein the characteristics are frame rate, profile indicator, level indicator, and Receiving a request for at least one of the components from the client device after transmitting the processor and the characteristics to the client device, the at least one of the dependencies between components, and transmitting the characteristics; And one or more interfaces configured to send the requested components to the client device in response to the request.

In another example, an apparatus for transmitting encapsulated video data is means for transmitting characteristics for components of a plurality of representations of video content to a client device, wherein the characteristics are frame rate, profile indicator, level indicator. Means for transmitting, means for receiving from the client device a request for at least one of the components after transmitting the characteristics, and at least one of the dependencies between the components, and the request. Means for responding to sending the requested components to the client device.

In another example, a computer program product, when executed, causes a processor of a source device to send encapsulated video data to at least one of a frame rate, a profile indicator, a level indicator, and dependencies between components To send to the client device the properties for the components of the plurality of representations of the video content and to receive the request for at least one of the components from the client device after transmitting the properties And causing the requested components to be transmitted to the client device in response to the request.

In another example, a method for receiving encapsulated video data comprises requesting a source device for characteristics of components of a plurality of representations of video content, the characteristics comprising a frame rate, a profile indicator, And at least one of dependencies between the components, selecting one or more of the components based on the characteristics, requesting samples of the selected components, and And decoding and presenting the samples after the samples are received.

In another example, an apparatus for receiving encapsulated video data is one or more interfaces configured to request a source device for characteristics of components of a plurality of representations of video content, the characteristics comprising a frame rate, a profile indicator, Selecting one or more of the components based on the interfaces and their characteristics, including at least one of dependencies between the components, the level indicator, and the components, And submit the requests to the source device for the samples of the source device.

In yet another example, an apparatus for receiving encapsulated video data comprises means for requesting a source device for characteristics of components of a plurality of representations of video content, the characteristics comprising a frame rate, a profile indicator, Means for selecting one or more of the components based on the characteristics, means for requesting samples of the selected components, and means for requesting samples of the selected components based on the at least one of the dependencies between the components. And means for decoding and presenting the samples after they are received.

In yet another example, a computer program product is programmed to cause a processor of a device to receive encapsulated video data, the program product comprising at least one of a frame rate, a profile indicator, a level indicator, Cause the source device to request characteristics of components of the plurality of representations of the video content, to select one or more of the components based on the characteristics, to request samples of the selected components, And causing the samples to be decoded and presented after the samples have been received.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

1 is a block diagram illustrating an exemplary system in which an audio / video (A / V) source device transmits audio and video data to an A / V destination device.
FIG. 2 is a block diagram illustrating components of an example encapsulation unit suitable for use with the A / V source device shown in FIG. 1.
3 is a conceptual diagram illustrating an exemplary component map box and an exemplary component placement box that may be used in the system of FIG.
4 is a conceptual diagram illustrating an exemplary timing interval for multiplexing exemplary video components and exemplary audio components in the system of FIG.
5 is a flow chart illustrating an exemplary method of providing a component map box and component arrangement boxes from a server to a client.

In general, the disclosure describes techniques for transmitting video content. Techniques of the present disclosure include transmitting video content using a streaming protocol, e.g., Hypertext Transfer Protocol (HTTP) streaming. Although HTTP is described for illustrative purposes, the techniques disclosed in this disclosure may be useful for other types of streaming. The video content may be encapsulated in video files in a specific file format, such as an ISO-based media file format or extensions thereof. The video content may also be encapsulated into an MPEG-2 transport stream. The content server may have different types of media data (eg, audio and video), and sets of different data for each type (eg, different languages such as English, Spanish, and German audio and / or MPEG-2, MPEG). -4, H.264 / AVC, or different encoding types for video such as H.265). The techniques of the present disclosure may be particularly useful for signaling how different types and sets of data of each type can be combined and multiplexed.

This disclosure refers to a collection of related multimedia data of a scene as “content”, which content may include a number of video and / or audio content components. The term "content component" or simply "component" refers to a single type of media, such as video or audio data. A component of the data may refer to a collection of tracks, sub-tracks, or tracks or sub-tracks of data. In general, a "track" may correspond to a sequence of associated encoded picture samples, while a sub-track may correspond to a subset of the encoded samples of the track. As an example, the content component may correspond to a video track, an audio track, or movie captions. An HTTP streaming server may deliver a set of content components to a client as a service to that client.

A service may correspond to a selection of one video content element from all video content elements available for that content and a selection of one audio content element from all available audio content elements for that content . For example, as a content stored in an HTTP server, a football matching program may include a plurality of video content components, e.g., different bit rates (512 kbps or 1 Mbps) or different frame rates, For example, English, Spanish, or Chinese. Thus, the service provided to the client may correspond to the selection of Spanish audio with one video component and one audio component, e.g., 512 kbps video. The combination of video component and audio component may also be referred to as a representation of content.

In HTTP streaming, as an example, a client device generates one or more requests for data in the form of HTTP Get requests or partial Get requests. The HTTP get request specifies the uniform resource locator (URL) or uniform resource name (URN) of the file. The HTTP partial retrieval request specifies the byte range of the file to be retrieved as well as the URL or URN of the file. The HTTP streaming server may respond to an HTTP acquisition request by outputting (e.g., sending) the file in the requested URL or URN, or in the requested byte range of the file, in the case of an HTTP partial fetch request. In order for the client to appropriately generate HTTP acquisition and partial acquisition requests, the server may be configured to select content components desired by the client and appropriately generate HTTP acquisition and / or partial acquisition requests for the components. May provide the client with properties of the elements as well as information regarding URLs and / or URNs of files corresponding to the elements.

The techniques of the present disclosure include signaling the characteristics of the content components, e.g., signaling the locations of data for the various content components. In this way, the client device may select a representation of the content and generate requests for combinations of different types of content components. For example, according to the example above, the user may choose to watch a 512 kbps video with Spanish audio. The viewer's client device may submits a request for these two components. That is, the client device may determine the locations for the data of 512 kbps video and Spanish audio using the signaled data from the server, and then generate a request for data corresponding to these content elements. In response to this request, the server may forward these two components as a service to the client device.

The ISO-based media file format is designed to include timed media information for presentation in a flexible, extensible format that facilitates media interchange, management, editing, and presentation. ISO-based media file format (ISO / IEC 14496-12: 2004) is defined in MPEG-4 Part-12 which defines the general structure for time-based media files. Support for H.264 / MPEG-4 AVC video compression has been defined in other file formats (such as AVC file format (ISO / IEC 14496-15), 3GPP file format, SVC file format, and MVC file format) Are used as a basis. The 3GPP file format and the MVC file format are extensions of the AVC file format. The ISO based media file format includes timing, structure, and media information for timed sequences of media data, such as audio-visual representations, for example. The file structure may be object oriented. Files can be decomposed very simply into basic objects, and the structure of objects from their type is implied.

Files conforming to the ISO-based media file format (and its extensions) may be formed as a series of objects, so-called "boxes ". The data in the ISO-based media file format may be included in the boxes, so that there is no need to include any other data in the file and there is no need for data to exist outside the boxes in the file. This includes any initial signature required by a particular file format. A "box" may be an object-oriented building block defined by a unique type identifier and length. In general, expressions are contained in a single file, and the media representation is self-contained. The movie container (movie box) may include metadata of the media, and the video and audio frames may be contained in the media data container and in other files.

In accordance with the teachings of the present disclosure, a server may provide a component map box that signals characteristics of various content components. The component map box may correspond to a data structure that may be stored in a file separate from the files storing the encoded samples for the various content components. The component map box may signal characteristics of the content component that are not conventionally signaled with the data structure stored externally to the files that actually contain the coded video samples for the video data. This data structure may also be signaled to a manifest file or a media presentation description of the HTTP stream, as in the component map box.

These characteristics may include, for example, frame rates, profile indicators, level indicators, and dependencies between components. The characteristics signaled by the component map box may also include relationships between three-dimensional properties for 3D video, such as the number of views and the views (e.g., two views that form a stereo pair). The component map box may signal these characteristics in addition to conventional signaled properties for the content component, such as bit rate and resolution for the content component. The component map box may also provide a service identifier (e.g., content_id value) that uniquely identifies the service of the content. Each component of the service may be associated with a service identifier.

The source device may be configured to provide a component map box for video content, regardless of how the content is encapsulated. That is, the source device may have video content in the AVC (Advanced Video Coding) file format, the Scalable Video Coding (SVC) file format, the Multiview Video Coding (MVC) file format, the Third Generation Partnership Project (3GPP) file format, or other files. Whether or not encapsulated according to formats, a component map box may be provided to client devices. The component map box may signal characteristics of the content components for a particular content. In some instances, each component may correspond to a video or audio track of a file, a track in a series of small files, track fragments, combinations of tracks (e.g., in SVC or MVC), or a subset of tracks .

In general, the component map box may be stored separately from the video data to be described. In some instances, the component map box may be included in a separate file or as part of a single movie file, e.g., mp4 or 3GP file, containing content components, or other file supporting the functionality described in this disclosure . The location of the component map box may vary depending on the type of file being encapsulated. Moreover, the component map box may be one or more extensions to the ISO-based media file format or extensions thereof. This data structure may also be signaled as a manifest file or media presentation description of HTTP streaming, as in the component map box.

By default, the component map box may be applied to the entire duration of the associated content. However, in some cases, the component map box may only be applied to a particular timing interval of content. In such a case, the server may provide a number of component map boxes, and the component map boxes may signal during each of the corresponding timing intervals. In some instances, when the server provides multiple component map boxes, the server may be configured in static mode, where the component map boxes are arranged contiguously in the same file in timing-interval order. In some examples, the server may be configured in dynamic mode, where component map boxes may be provided in separate files and / or in discrete locations from each other. The dynamic mode may provide advantages in live streaming, while the static mode may provide advantages over trying in a larger time span.

The present disclosure also provides a component arrangement box that may be included in each file to signal the relationships between the tracks and the various components of the file. For example, in a file that contains data for two or more tracks, the component arrangement box may define the relationship between the track identifiers for the tracks in the file and the component identifiers for the corresponding content components Signaling. In this way, the client device may first retrieve the component map box from the server device. The client device may then select one or more components of the presentation based on the characteristics signaled by the component map box. The client device may then retrieve the component arrangement boxes from the files storing the components described by the component map box. Using a component map box that may include segment information such as byte ranges of fragments for a particular component, the client may determine where fragments of the selected components are stored in the files. Based on this determination, the client may submits requests (e.g., HTTP acquisition or partial acquisition requests) for fragments of tracks or sub-tracks corresponding to the selected components.

In this way, instead of signaling information about how each file or each track is associated with the content component in the component map box, this information is stored in the component arrangement boxes associated with each of the files . The component map box may signal component identifiers (e.g., component_id values) of all components of the content, while the component arrangement box may contain component_id values of the components stored in the file corresponding to the component arrangement box and may signal the relationships between the content_id values associated with the component_id values. The component map box may also store, in some cases, segment information. In addition, the component map box may include a flag indicating whether the component map box includes segment information. The client device may be configured to assume that the media data of the representation is included in the dependent representations if the component map box does not contain segment information.

By assigning unique component_id values to each type of media, the server may ensure that the component_id value is unique to any video or audio component in the same service. Certain types of components may switch between. That is, the client may switch between the various video components, e.g., in response to varying network conditions or other factors. Clients do not need to request components of each available type. For example, the client may omit requesting subtitles for content that includes closed caption components. Moreover, in some cases, multiple components of the same media type may be requested, e.g., to support 3D video or picture in picture (PIP). The server may provide additional signaling to support certain functions, e.g., in-screen display.

For example, the server may provide a flag indicating whether the component includes a description of screen-in-screen data. If the flag indicates that the component includes screen-in-screen data, the component map box may provide an identifier of the representation to be displayed together to form an in-screen display, along with the current presentation. One representation may correspond to a larger image, while the other representation may correspond to a smaller image overlaid on the larger image.

As mentioned above, the server may provide a component arrangement box to each file containing encoded samples corresponding to one or more components. The component arrangement box may be provided in the header data of the file. The component arrangement box may indicate how the components and components included in the file are stored, for example, as tracks in the file. The component arrangement box may provide a mapping between the component identifier value and track identifier values of the corresponding track in the file.

The component map box may also signal dependencies between content components, where the signaled dependencies, along with the current content components for the decoding order of the content components within the access unit, . The signaled information about the dependencies on the current representation may include either or both of the current representation and / or representations that depend on the representations upon which the current representation depends. There may also be dependencies between content components in the time dimension. However, simply indicating the temporal_id values for each video component may not be sufficient because the temporal sub-layers in totally independent alternative video bitstreams do not necessarily have to have mappings of frame rates to each other . For example, one video component may have a frame rate of 24 fps, a temporal id of 0, a sub-layer of 12 fps (assuming two time layers), and another video The component may have a frame rate of 30 fps, a temporal id of 0, and a lower layer of 7.5 fps (assuming three time layers). Thus, the server may indicate a time layer difference when the dependencies of the two video components are signaled.

In general, the signaled characteristics of a component may include, for example, an average bit rate, a maximum bit rate (e.g., greater than one second), resolution, frame rate, dependency on other components, and / And these extensions may include the number of views targeted for output and identifiers for those views. Information about a series of media fragments forming a content element may also be signaled. The signaled information for each media fragment may include a byte offset of the media fragment, a decoding time of the first sample in the media fragment, a random access point and its decoding and presentation time in the fragment, and / (And therefore different URLs) of the < / RTI >

In some cases, fragments of audio data are temporally aligned with fragments of video data. The present disclosure provides techniques for multiplexing multiple content components based on a particular time interval. The component map box may provide a list of supported multiplexing intervals, or a range of multiplexing intervals. The multiplexing interval may be denoted as T and may represent the length of time of the multiplexed audio and video data. Assume that the next time interval requested is [n * T, (n + 1) * T]. The client device may determine whether there is any fragment in each content element having start time t in the range of (n * T) <= t <= ((n + 1) * T). If so, the client device may request the fragment. Fragments starting before n * T may be requested before the current multiplexing interval n * T, while fragments starting after the interval (n + 1) * T may be requested in the subsequent multiplexing interval. In this way, content components that do not have fragment boundaries that align with each other or at the requested multiplexing interval may nevertheless be multiplexed. Moreover, the multiplexing interval may change without interfering with the multiplexing of the content components during the service.

The client device may be configured to adapt to changing network conditions by changing the multiplexing interval. For example, when bandwidth becomes relatively more available, the client device may increase the multiplexing interval. On the other hand, when the bandwidth becomes relatively less available, the client device may reduce the multiplexing interval. The client device may further be configured to request multiplexed fragments based on a certain timing interval and instantaneous bit rate. The client device may calculate the instantaneous bit rate based on the number of intra-fragment bytes and the time duration of the fragment.

In some instances, the server may assign the same component identifier to two video files having two consecutive media representations, e.g., sequential timing information, to support time splicing. As mentioned above, in some cases, the representations may include content components stored in different files. Thus, a client device may need to commit a number of acquisitions or partial acquisitions to retrieve data during a particular time interval of the content. That is, a client may need to submit multiple retrieval or partial retrieval requests that refer to several files storing content elements for that representation. When multiple requests are required to acquire data that is to be multiplexed at some time interval, the client device sends a request to that request to ensure that no data is received between the desired media fragment data in the current time interval in another time interval May be formed into a pipeline.

In this way, media content with components in multiple files may be supported in a network streaming context, e.g., HTTP streaming. That is, the representation of the media content may include one component in one file and another component in a separate file. The server may signal the characteristics of the components in the different files into a single data structure, e.g., a component map box. This may allow the client to make requests for any target content element, or for any duration of the target content element.

The use of similar data structures in the component map boxes and component arrangement boxes of this disclosure may also provide other advantages. For example, in the different components, the two media tracks may have the same track identifier (track_id) value in each of the components. However, as mentioned above, the component map box may refer to separating components using component identifiers that are not the same as the track identifier value. Since each file may include a component arrangement box that maps component identifiers to track identifiers, the component map box may refer to the components using component identifiers that are independent of their track identifier values have. The component arrangement box may also provide an efficient mechanism for defining which file corresponds to which content, for example, when a content delivery network (CDN) server stores multiple files corresponding to many different content.

Moreover, the techniques of this disclosure may support clients with different network buffer sizes. That is, some clients may require buffers of a different size than other buffers, e.g., due to network conditions, client capabilities, and so on. Thus, in some cases, multiple types of components for a particular representation may need to be multiplexed at different time intervals. This disclosure considers the performance of transmissions in terms of the server signaling different possible multiplexing time intervals, and therefore in terms of variations in the size of the data requested, and therefore, for example, a round trip time between a client and a server using HTTP. Provide techniques to do this.

Moreover, in some cases, a content component in one file may depend on several other content components in one or more other files. This dependency may occur within an access unit. As an example, the video content component may correspond to a CIF SVC enhancement layer that relies on a common interface format (CIF) layer and a quarter common interface format (QCIF) layer. While both CIF and QCIF layers may be in one file, the 4CIF enhancement layer may be in another file. The techniques of the present disclosure allow a client to appropriately request data for the CIF, QCIF, and 4CIF layers so that the decoder of the client can receive samples from these layers in a suitable decoding order based on their dependencies .

In some instances, a dynamic server may be used to dynamically generate files that multiplex content elements together. For example, a dynamic server may support methods that conform to Common Gateway Interface (CGI) services to multiplex components together to make the data for the current time interval a continuous part of a dynamic file. The CGI is described in a Request for Comment 3875 of Comment 3875 available at http://tools.ietf.org/html/rfc3875. Using a service such as CGI, a server may dynamically generate a file containing a combination of several content components for presentation of the content.

The representation (motion sequence) may be included in several files. Timing and framing (position and size) information is typically in ISO-based media files, and auxiliary files may use essentially any format. This representation may be ' local ' to the system containing the representation or it may be provided via a network or other stream delivery mechanism.

The files may have a logical structure, a time structure, and a physical structure, and these structures are not required to be paired. The logical structure of the file may ultimately be a movie or video clip (potentially including both video and audio data) that includes a set of time-parallel tracks. The temporal structure of the file may be such that the tracks contain sequences of samples in time and those sequences are mapped to the entire movie's timeline by optional edit lists. The physical structure of the file may separate the data required for logic, time, and structural decomposition from the media data samples themselves. This structural information may be concentrated in the movie box, perhaps extended in time by the movie short boxes. The movie box may document the logic and timing relationships of the samples, and may also include pointers where they are located. Those pointers may point to, for example, the same file or another file referenced by the URL.

Each media stream may be included in a track specialized for its media type (audio, video, etc.) and may be further parameterized by a sample entry. The sample entry may include the 'name' of the exact media type (the type of decoder needed to decode the stream) and any parameterization of the decoder required. The name may also take the type of a 4-character code, e.g., "moov, " or" trak. &Quot; There are sample entry formats defined for media types used by MPEG-4 media, as well as other organizations using this file format family.

Support for meta-data generally takes two forms. First, timed meta-data is stored in the appropriate track and may be synchronized with the media data describing it as desired. Second, there may be general support for non-timed meta-data accompanying movies or individual tracks. Structural support is common and enables storage of meta data resources in a manner similar to the storage of media data, i.e. coded video images, in the file or elsewhere in another file. In addition, these resources may be named and protected.

The term "progressive download" is used to describe the transfer of digital media files from a server to a client, typically using the HTTP protocol. When initiated from a computer, the computer may begin playback of the media before the download is complete. One difference between streaming media and progressive download is how digital media data is received and stored by an end user device that accesses the digital media. Progressive Download Playback enabled media players depend on the local buffer of metadata and digital media files located in the header of unaffected files when downloaded from a web server. At a point where a specified amount of buffered data becomes available to the local playback device, the device may begin playing the media. This specified amount of buffered data may be embedded in the file by the creator of the content in the encoder settings and may be enforced by additional buffer settings enforced by the media player of the client computer.

Instead of providing a single movie box (moov box) that contains all the media data, including video and audio samples, in progressive downloading or HTTP streaming, movie fragments may be added to the samples contained within the movie box It is supported to include extra samples. In general, movie fragments contain samples for a certain time period. Using these movie fragments, the client can quickly find the desired time. The movie segment may include a file of consecutive bytes so that the client can issue a partial-acquisition request to retrieve the movie segment, in accordance with a streaming protocol such as HTTP streaming.

As an example, for 3GPP, HTTP / TCP / IP transmissions are supported for 3GPP files for download and progressive download. Moreover, using HTTP for video streaming may provide some benefits, and HTTP-based video streaming services are becoming popular. HTTP streaming may include certain advantages that existing Internet components and protocols may be used, so no new efforts are required to develop new techniques for transmitting video data over a network. Other transport protocols, such as the Real Time Protocol (RTP) payload format, require intermediate network devices, e.g., intermediate boxes, to recognize the media format and signaling context. In addition, HTTP streaming can be client-driven, avoiding control issues. Using HTTP also does not require new hardware or software implementations on a web server that implements HTTP 1.1. HTTP streaming also provides TCP-friendliness and firewall traversal. In HTTP streaming, the media representation may be a structured collection of data accessible to the client. The client may request and download the media data information to present the streaming service to the user.

A service is experienced as a representation of a movie by a user of the client, which is decoded and rendered by the client from the content elements delivered by the server. In HTTP streaming, instead of receiving the entire content in response to a single request, a client may request segments of content components. In this way, HTTP streaming may enable delivery of more flexible content. A segment may include a set of consecutive movie segments that can be requested by a single URL. For example, the segment may be an entire small file, which may include video and audio. As another example, a segment may correspond to one movie segment, which may include one video track segment and one audio track segment. As yet another example, a segment may correspond to several movie segments, any of which may have one video segment and one audio segment, and the movie segments may have a continuous decoding time.

A content delivery network (CDN), also referred to as a content distribution network, is located at various points in the network to maximize bandwidth for accessing the data by clients across the network, Or a system of computers including, but not limited to, A client may access a copy of the data near the client, in contrast to all clients accessing the same central server, thereby avoiding bottlenecks near the individual servers. Content types may include web objects, downloadable objects (media files, software, documents, etc.), applications, real-time media streams, and other components of Internet delivery (DNS, routes, , And database queries). There are many successful CDNs that depend only on the HTTP protocol, more specifically origin servers, proxies and caches based on HTTP 1.1.

In HTTP streaming, frequently used operations include fetching and partial fetching. The get operation retrieves the entire file associated with a given uniform resource locator (URL) or uniform resource name (URN). The partial acquisition operation receives the byte range as an input parameter and retrieves the consecutive number of bytes of the file corresponding to the received byte range. Thus, movie fragments may be provided for HTTP streaming because the partial acquisition operation may acquire one or more individual movie fragments. A movie segment may include multiple track fragments from different tracks.

In the context of HTTP streaming, a segment may be delivered as a response to an acquisition request or a partial acquisition request (in HTTP 1.1). In a CDN, computing devices such as proxies and caches may store segments in response to those requests. Thus, if the segment is requested by another client (or the same client) and the client has a path through this proxy device, the proxy device will not be able to retrieve the local copy of the segment without retrieving the segment from the originating server again. Can be delivered to. In HTTP streaming, if the proxy device supports HTTP 1.1, the byte ranges can be extracted as a response to requests, while being stored in the cache of the proxy device or being used as a local copy of the response to the request . Each content component may comprise sections of consecutive fragments, each of which may be requested by HTTP acquisition or partial acquisition sent by the client device. Such fragments of content components may be referred to as media fragments.

In addition to supporting multiple bit rates and multiple devices, there may be more than one media representation in HTTP streaming to accommodate different user preferences. The description of the representations may be described by a media presentation description (MPD) data structure when generated by a server and transmitted to a client, which may correspond to a component map box. That is, a conventional MPD data structure may include data corresponding to a component map box, as described in this disclosure. In other examples, the component map box may further include data similar to the MPD data structure, in addition to the data described in this disclosure for the component map box. Representations described may include content components, included in one or more movie files. When a static content server is used, the server may store movie files. If a dynamic content server is supported, the server may generate a dynamic file (content) in response to the received request. Dynamic content may then be generated by the server, but is transparent to computing devices such as proxies and caches. Thus, segments provided in response to requests for the dynamic content server may also be cached. The dynamic content server may have more complex implementations, and perhaps less storage is optimal on the server side or less cache will be efficient during delivery of the content.

In addition, the present disclosure also includes techniques for signaling at the MPD whether a particular representation (e.g., a combination of components) is an operating point of completion. That is, the server may provide a flag to the MPD to indicate to the client whether the presentation can be selected as a completed video operation point. The operating point may correspond to a subset of the MVC sub-bit stream, i.e., the MVC bit stream, which contains a subset of views at some time level and represents a bit stream valid for itself. An operating point represents a level of time and view scalability, and may only include NAL units required for a valid bitstream to indicate some subset of views at any time level. The operating point may be described by the view identifier values of the subset of views, and by the highest time identifier of the subset of views.

The MPD may also describe individual representations for multimedia content. For example, for each representation, the MPD may signal an expression identifier, a default attribute representation identifier, a profile and level indicator for the representation, a frame rate for the representation, a dependency group identifier, and a time identifier. The presentation identifier may provide a unique identifier of the associated presentation for the multimedia content. The default attribute representation identifier may include any or all of a profile and level indicator, a bandwidth, a width, a height, a frame rate, a dependency group identifier, a time identifier, and / or a frame packing type for 3D video Which may be used as default attributes for the current presentation, which are to be used. The frame rate identifier may define the frame rate of the video component (s) for the corresponding representation. The dependency group identifier may specify a dependency group to which the corresponding representation is assigned. Representations in a dependency group having a time identifier value may depend on the expressions in the same dependency group having low time identifier values.

For example, in 3D video representations corresponding to multi-view video, the component map box may describe the number of target views for output. That is, the component map box may include a value indicating the number of target output views for the representation. In some instances, the component map box may provide depth information for a single view with coded samples for a single view, such that the client device may construct a second view from the single view and depth information. The flag may be provided to indicate that the representation is a view plus depth representation. In some instances, multiple views each associated with depth information may be included in the representation. In this way, each of the views can be used as a basis for generating a stereo view pair, thus generating two views for each of the views of the representation. Thus, although multiple views may be included in the representation, two of the views do not necessarily form a stereo-view pair. In some instances, a flag may be included to indicate whether the representation is merely a dependent expression that can not form a valid representation of its corresponding multimedia content alone.

1 is a block diagram illustrating an exemplary system 10 in which an audio / video (A / V) source device 20 transmits audio and video data to an A / V destination device 40. As shown in FIG. The system 10 of FIG. 1 may be a remote video conferencing system, a server / client system, a broadcaster / receiver system, or a system in which video data is received from a source device, such as an A / V source device 20, Or to any other system that is transmitted to the device 40. In some examples, A / V source device 20 and A / V destination device 40 may perform a bidirectional information exchange. That is, the A / V source device 20 and the A / V destination device 40 may be capable of both (and transmitting and receiving) both audio and video data for encoding and decoding. In some instances, the audio encoder 26 may also include a voice encoder, also referred to as a vocoder.

In the example of FIG. 1, the A / V source device 20 includes an audio source 22 and a video source 24. The audio source 22 may include, for example, a microphone that generates electrical signals representative of the captured audio data encoded by the audio encoder 26. Alternatively, audio source 22 may include a storage medium that stores previously recorded audio data, an audio data generator, such as a computerized synthesizer, or any other source of audio data. Video source 24 includes a video camera that generates video data encoded by a video encoder 28, a storage medium encoded with previously recorded video data, a video data generation unit, or any other source of video data You may.

Raw audio and video data may include analog or digital data. The analog data may be digitized before being encoded by the audio encoder 26 and / or the video encoder 28. The audio source 22 may obtain audio data from the conversation participant while the conversation participant is speaking and the video source 24 may simultaneously acquire the video data of the conversation participant. In other examples, audio source 22 may include a computer readable storage medium including stored audio data, and video source 24 may include a computer readable storage medium including stored video data. In this way, the techniques described in this disclosure may be applied to live, streaming, real-time audio and video data or to archived, pre-recorded audio and video data. Moreover, these techniques may be applied to computer-generated audio and video data.

Audio frames corresponding to video frames are audio frames that contain audio data captured by audio source 22, concurrent with video data captured by video source 24, typically contained within video frames. For example, the audio source 22 captures audio data while the conversation participant typically generates audio data, and the video source 24 simultaneously captures audio data 22, Captures the video data of the conversation participant while doing so. Therefore, the audio frame may correspond in time to one or more specific video frames. Thus, an audio frame corresponding to a video frame generally corresponds to a situation where audio data and video data are captured at the same time, and audio and video frames respectively contain audio data and video data captured simultaneously.

In some instances, the audio encoder 26 may encode a time stamp in each encoded audio frame, indicating the time at which the audio data for the encoded audio frame was recorded, May encode a time stamp in each encoded video frame, indicating the time at which the video data for the encoded video frame was recorded. In these instances, the audio frame corresponding to the video frame may comprise an audio frame containing a time stamp and a video frame containing the same time stamp. The A / V source device 20 may also be configured to allow the audio encoder 26 and / or the video encoder 28 to generate time stamps or the audio source 22 and video source 24 to time- Lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt;

The audio source 22 may transmit data corresponding to the time at which the audio data was recorded to the audio encoder 26 and the video source 24 may transmit data corresponding to the time at which the video data was recorded to the video encoder 26. [ (28). In some examples, audio encoder 26 may encode a sequence identifier in the encoded audio data to indicate a relative time order of the encoded audio data without necessarily indicating the absolute time at which the audio data was recorded, and similarly, video Encoder 28 may also use sequence identifiers to indicate a relative temporal order of the encoded video unit. Similarly, in some instances, the sequence identifier may be mapped or otherwise correlated with a time stamp.

Techniques of the present disclosure generally relate to the transmission of encoded multimedia (e.g., audio and video) data, and the subsequent reception and subsequent interpretation and decoding of the transmitted multimedia data. In particular, the encapsulation unit 30 may generate component arrangement boxes for multimedia content, as well as component arrangement boxes for each file corresponding to the multimedia content. In some instances, the processor may execute instructions corresponding to the encapsulation unit 30. [ That is, instructions to perform functions attributable to encapsulation unit 30 may be stored on a computer readable medium and executed by a processor. Other processing circuitry may, in other examples, be configured to perform the functions that are also attributable to encapsulation unit 30. The component map box may be stored separately from the components of its content (eg, audio components, video components, or other components).

Thus, the destination device 40 may request a component map box for the multimedia content. Destination device 40 may determine components that request to perform playback of content based on user's preferences, network conditions, decoding and rendering capabilities of destination device 40, or other factors. You can also use the component map box.

The A / V source device 20 may provide a "service" to the A / V destination device 40. A service generally corresponds to a combination of one or more audio content components and a video content component, wherein the audio and video content components are a subset of the available content components of the overall content. One service may correspond to a stereo video having two views, while another service may correspond to four views, and another service may correspond to eight views. Generally, a service corresponds to a source device 20 that provides a combination (i.e., a subset) of available content components. The combination of content components is also referred to as a representation of the content.

The encapsulation unit 30 receives the encoded samples from the audio encoder 26 and the video encoder 28 and forms corresponding network abstraction layer (NAL) units from the encoded samples, And may take the type of packetized elementary stream (PES) packets. In the example of H.264 / AVC (Advanced Video Coding), coded video segments are organized into NAL units, which address applications such as video telephony, storage, broadcast, or streaming Providing a "network-friendly" video representation. NAL units may be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL units may include data from the core compression engine and may include block, macroblock, and / or slice level data. Other NAL units may be non-VCL NAL units. In some instances, a coded picture at one time instance, presented as a generally coded picture, may be included in an access unit, which may include one or more NAL units.

According to the teachings of the present disclosure, the encapsulation unit 30 may constitute a component map box that describes the characteristics of the content components. The encapsulation unit 30 may also configure component arrangement boxes for one or more video files. Encapsulation unit 30 may associate each component arrangement box with a corresponding video file, and may associate a component map box with a set of video files. In this way, there is a 1: 1 correspondence between the component arrangement boxes and the video files, and there may be a 1: N correspondence between the component map boxes and the video files.

As mentioned above, the component map box may describe the characteristics of the components common to the content. For example, the content may include audio components, video components, and other components, such as closed captions. Each type of component may be switchable to each other. For example, two video components may be switchable in that data from either of the two components may be retrieved without interfering with playback of the content. The various components may be encoded in different ways and in different qualities. For example, various video components may be encoded at different frame rates, bit rates, using different encoders (e.g., corresponding to different codecs) to provide different file types (e.g., H.264 / AVC or MPEG-2 transport stream (TS)), or they may be different from each other. However, for example, the selection of video components is generally independent of the selection of audio components. The characteristics of the components that are signaled by the component map box include the average bit rate, the maximum bit rate (e.g., for a playback time of one second for the component), resolution, frame rate, dependencies on other components, Extensions for various file types such as multi-view video, e.g., the number of views targeted for output and identifiers for each of the views.

The source device 20, which may operate as a server, such as an HTTP server, may store multiple representations of the same content for adaptation. Some expressions may include multiple content elements. The components may be stored in different files on the storage device (eg, one or more hard drives) of the source device 20, so that the representation may include data from different files. By signaling the characteristics of the various components, the encapsulation unit 30 may provide the destination device 40 with the ability to select one of each switchable component to render and play the corresponding content. That is, the destination device 40 retrieves the component map box from the source device 20 for the particular content, selects the components for the content corresponding to the particular representation of the content, Data for the elements may be retrieved from the source device 20, e.g., in accordance with a streaming protocol, such as HTTP streaming.

The destination device 40 may select the presentation based on network conditions, such as available bandwidth, and characteristics of the components. Moreover, the destination device 40 may adapt to changing network conditions using data signaled by the source device 20. [ That is, because the same type of components can be switched to each other, when the network conditions change, the destination device 40 may select a different type of different component that is more suitable for the newly determined network conditions.

The encapsulation unit 30 assigns the component identifier values to each of the components of the multimedia content. The component identifier values are unique to the components regardless of type. That is, for example, there should be no audio component and no video component with the same component identifier. The component identifiers also need not necessarily be associated with track identifiers in individual files. For example, the content may have two video components each stored in different files. Each of the files may be identified using the same track identifier, since the identifiers local to a particular file are unique, but not externally, to the extent of the file. However, because the techniques of the present disclosure involve providing the characteristics of components that may reside in multiple files, the present disclosure may be used to uniquely assign component identifiers that are not necessarily associated with the track identifiers .

The component map box also displays how the fragments are stored for each component / track in the file, e.g., where fragments originate, whether they contain random access points (and whether the random access points are instantaneous decoding refreshes IDR) or open decoding refresh (ODR) pictures), byte offsets for the beginning of each fragment, decoding times of the first samples in each fragment, decoding and presentation times for random access points , And a flag indicating whether a particular segment belongs to a new segment. Each segment can be taken out independently. For example, the encapsulation unit 30 may store each segment of a component such that each segment can be retrieved using a unique URL or URN.

Moreover, the encapsulation unit 30 may provide component arrangement boxes to each of the files providing mappings between component identifiers for content and track identifiers in a corresponding file. The encapsulation unit 30 may also signal dependencies between components of the same type. For example, some components may depend on other components of the same type that are correctly decoded. As an example, in a scalable video coding (SVC), a base layer may correspond to one component, and an enhancement layer for a base layer may correspond to another component. As another example, in multi-view video coding (MVC), one view may correspond to one component, and another view of the same scene may correspond to another component. As yet another example, samples of one component may be encoded for samples of another component. For example, in MVC, the components corresponding to different views where inter-view prediction is enabled may be.

In this way, the destination device 40 determines the dependencies among the components to properly decode and / or render the components and, in addition to the desired components, to the components that depend on the parent components You can also retrieve the parent components for. Encapsulation unit 30 may further signal the order of dependencies and / or the decoding order of the components so that destination device 40 can request data for the components in a suitable order. Furthermore, the encapsulation unit 30 may signal time layer differences between components having dependencies, such that the destination device 40 may properly align the samples of the components for decoding and / or rendering. For example, one video component may have a frame rate of 24 and temporal_id which is 0 sublayer of 12 fps, while another video component may have a frame rate of 30 and temporal_id which is 0 sublayer of 7.5 fps. .

Encapsulation unit 30 may signal several possible multiplexing intervals for combinations of components to form a representation. In this way, the destination device 40 is able to retrieve data for the various components over a sufficient time period so that data for the upcoming segments of the components can be retrieved while the previous segments of the components are being decoded and displayed To request, one of the possible multiplexing intervals may be selected. That is, the destination device 40 does not advance to the point where the buffer overflows but requests data for those components that are far enough ahead that there is no blocking (assuming there is no immediate change in network conditions) It is possible. If there is a change in the network conditions, then the destination device 40 may wait for transmission of more subsequent data, rather than switching all of the components, to ensure that a sufficient amount of data is fetched for decoding and rendering, You can also choose an interval. The encapsulation unit 30 may signal multiplexing intervals based on explicitly signaled intervals or ranges of intervals, and may signal these multiplexing intervals within the component map box.

In some instances, the source device 20 may receive requests defining a number of byte ranges. That is, destination device 40 may define multiple byte-ranges in one request to achieve multiplexing of the various components in the file. The destination device 40 may send multiple requests when the components are in multiple files, and any or all of the requests may define one or more byte ranges. As one example, destination device 40 may submit multiple HTTP acquisition or partial acquisition requests to multiple URLs or URNs, where any or all of the partial acquisition requests are URLs or URNs of those requests. Multiple byte ranges may be defined within. The source device 20 may respond by providing the requested data to the destination device 40. In some instances, the source device 20 may multiplex the components of the multiplex representation together, for example, in order to dynamically form a file that the source device 20 may then provide to the destination device 40, By implementing a gateway interface (CGI), it may support dynamic multiplexing.

The encapsulation unit 30 may also define the time duration of the content for which the component map box corresponds. By default, the destination device 40 may be configured to determine that the component map box applies to the entire content when no time duration is signaled. However, when signaled, destination device 40 may be configured to request multiple component map boxes for that content, each corresponding to a different time duration of the content. The encapsulation unit 30 may store the component map boxes together adjacent, or in separate locations.

In some cases, different parts of the component (e.g., segments) may be stored in separate files (e.g., URL or URN retrievable data structures). In this case, the same component identifier may be used in each file, e.g., to identify the component within the component arrangement box of the file. The files may have sequential timing information, i.e., timing information indicating that one of the files follows the other file immediately. The destination device 40 may generate requests for multiplexed fragments based on some timing interval and instantaneous bit rate. The destination device 40 may calculate the instantaneous bit rate based on the number of bytes in the fragments of the component.

As with most video coding standards, H.264 / AVC defines a decoding process for syntax, semantics, and error-free bitstreams, and any stream of these bitstreams may be associated with any profile or level . H.264 / AVC does not prescribe an encoder, but the encoder is responsible for ensuring that the generated bitstreams are standards-compliant for the decoder. In the context of a video coding standard, a "profile" corresponds to a subset of algorithms, features, or tools and constraints that apply to them. As defined by the H.264 standard, for example, a “profile” is a subset of the full bitstream syntax defined by the H.264 standard. The "level" corresponds to limitations of decoder resource consumption, such as, for example, decoder memory and calculations, related to the resolution, bit rate, and macroblock (MB) processing rate of pictures. A profile may be signaled with a profile_idc (profile indicator) value, but the level may be signaled with a level_idc (level indicator) value.

The H.264 standard allows for encoders and decoders, for example, in accordance with the values taken by the syntax elements in the bitstream, such as the specified size of the decoded pictures, within limits imposed by the syntax of a given profile It is still possible to require large fluctuations in performance. The H.264 standard further recognizes that, in many applications, it is neither practical nor economical to implement decoders that can handle all the hypothetical uses of the syntax within a particular profile. Thus, the H.264 standard defines a "level" as a specified set of constraints imposed by the values of syntax elements in the bitstream. These constraints may be simple restrictions on values. Alternatively, these constraints may take the type of constraints on the arithmetic combinations of values (e.g., image width times image height times times the number of images decoded per second). The H.264 standard further provides that individual implementations may support different levels for each supported profile.

A profile compliant decoder typically supports all the features defined in the profile. For example, as a coding feature, B-picture coding is not supported in the baseline profile of H.264 / AVC, but is supported in other profiles of H.264 / AVC. A level-compliant decoder should be able to decode any bitstream that does not require resources beyond the limits defined at that level. Definitions of profiles and levels may be helpful in interpreting ability. For example, during video transmission, the pair of profile and level definitions may be negotiated and agreed for the entire transmission session. More specifically, in H.264 / AVC, the level may be, for example, the number of macroblocks that need to be processed, the decoded picture buffer (DPB) size, the coded picture buffer (CPB) , The maximum number of motion vectors per two consecutive MBs, and whether the B-block has sub-macroblock partitions of less than 8 x 8 pixels. In this way, the decoder may determine whether or not the decoder can properly decode the bitstream.

The media representation may include a media description description (MPD), and the media description description may include descriptions of different alternative representations (e.g., video services with different qualities), such as codec information, A profile value, and a level value. The destination device 40 may retrieve the MPD of the media representation to determine how to access the movie fragments of the various representations. Movie fragments may be located in movie fragments boxes (moof boxes) of video files.

Video compression standards such as ITU-T H.261, H.262, H.263, MPEG-1, MPEG-2 and H.264 / MPEG-4 Part 10 use motion compensation time prediction to reduce time redundancy . The encoder uses motion compensated prediction from some previously encoded pictures (also referred to herein as frames) to predict the current coded pictures according to the motion vectors. In representative video coding, there are three main picture types. ("P-pictures" or "P-frames") and bidirectionally predicted pictures ("B-pictures" Pictures "or" B-frames "). The P-pictures use only the reference picture prior to the current picture in the time sequence. In the B-picture, each block of the B-picture may be predicted from one or two reference pictures. These reference pictures can be located before or after the current picture in chronological order.

According to the H.264 coding standard, as one example, B-pictures use two of the previously-coded reference pictures, i.e., list 0 and list 1. These two lists may each include past and / or future coded pictures in chronological order. The blocks in the B-picture can be divided into several methods: motion-compensated prediction from a list 0 reference picture, motion-compensated prediction from a list 1 reference picture, or motion from a combination of both a list 0 reference picture and a list 1 reference picture - May be predicted as one of the compensated predictions. To obtain a combination of both the List 0 reference picture and the List 1 reference picture, two motion compensated reference areas are obtained from the List 0 reference picture and the List 1 reference picture, respectively. Their combination may be used to predict the current block.

The ITU-T H.264 standard defines intra prediction in several block sizes, e.g., 16 x 16, 8 x 8, or 4 x 4 for luma components and 8 x 8 for chroma components. As well as in various block sizes, e.g., 16 x 16, 16 x 8, 8 x 16, 8 x 8, 8 x 4, 4 x 8 and 4 x 4 for luma components, Inter prediction is supported in scaled sizes. In the present disclosure, "N x N" and "N x N" are interchangeable to refer to the pixel dimensions of the block, e.g., 16 x 16 pixels or 16 x 16 pixels, . In general, a 16 x 16 block will have 16 pixels (y = 16) in the vertical direction and 16 pixels (x = 16) in the horizontal direction. Similarly, an N × N block generally has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. The pixels in the block may be arranged in rows and columns. The blocks may have a different number of pixels in the horizontal and vertical dimensions. That is, the blocks may include N x M pixels, where N does not necessarily have to equal M.

Block sizes less than 16x16 may be referred to as partitions of a 16x16 macroblock. The video blocks are either blocks of pixel data in the pixel domain, or into discrete video block data representing pixel differences between discrete cosine transform (DCT), integer transform, wavelet transform, or coded video blocks and predictive video blocks. It may conceptually include blocks of transform coefficients in the transform domain followed by application of a transform, such as a similar transform. In some cases, the video block may comprise blocks of quantized transform coefficients in the transform domain.

Smaller video blocks may provide better resolution and may be used for positions of video frames that include higher detail levels. In general, macroblocks and several partitions are often referred to as sub-blocks and may be considered as video blocks. In addition, the slice may be regarded as a plurality of video blocks, e.g., macroblocks and / or sub-blocks. Each slice may be an independently decodable unit of a video frame. Alternatively, the frames themselves may be decodable units, or other parts of the frame may be defined as decodable units. The term "coded unit" or "coding unit" refers to any independently decodable unit of a video frame, such as a whole frame, a slice of frames, a group of pictures (GOPs) May refer to another independently decodable unit defined in accordance with techniques.

The term macroblock refers to a data structure for encoding image and / or video data according to a two-dimensional pixel array comprising 16 x 16 pixels. Each pixel includes a chrominance component and a luminance component. Thus, the macroblock is divided into four luminance blocks each comprising a two-dimensional array of 8 x 8 pixels, two chrominance blocks each comprising a two-dimensional array of 16 x 16 pixels, and a coded block pattern (CBP). ), Encoding mode (eg, intra- (I), or inter- (P or B) encoding modes), partition size for partitions of the intra-encoded block (eg, 16 × 16, 16 × 8, 8 x 16, 8 x 8, 8 x 4, 4 x 8, or 4 x 4), or one or more motion vectors for the inter-encoded macroblock.

Each of the video encoder 28, the video decoder 48, the audio encoder 26, the audio decoder 46, the encapsulation unit 30 and the decapsulation unit 38 may be implemented as one or more microprocessors, Any of a variety of suitable processing circuits such as DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware or any combination thereof, If applicable, may be implemented. Each of video encoder 28 and video decoder 48 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined video encoder / decoder (codec). Similarly, each of the audio encoder 26 and the audio decoder 46 may be included in one or more encoders or decoders, or may be integrated as part of a combined codec. An apparatus comprising a video encoder 28, a video decoder 48, an audio encoder audio encoder 26, an audio decoder 46, an encapsulation unit 30, and / or an encapsulation unit 38 may be implemented within one or more integrated circuits , Microprocessors, and / or any combination of wireless communication devices, e.g., cellular telephones.

After the encapsulation unit 30 assembles the video file based on the received data, the encapsulation unit 30 passes the video file to the output interface 32 for output. In some instances, the encapsulation unit 30 may transmit the video file to the remote server via the output interface 32, rather than locally storing the video file or sending the video file directly to the destination device 40. [ Output interface 32 may be, for example, a transmitter, a transceiver, a device for writing data to a computer readable medium such as an optical drive, a magnetic media drive (e.g., a floppy drive), a universal serial bus Port, network interface, or other output interface. Output interface 32 outputs the video file to a computer readable medium 34, or other computer readable medium, such as, for example, a transmit signal, magnetic media, optical media, memory, flash drive. The output interface 32 may implement HTTP 1.1 to respond to HTTP acquisition and partial acquisition requests. In this way, source device 20 may operate as an HTTP streaming server.

Ultimately, the input interface 36 retrieves data from the computer readable medium 34. The input interface 36 may include, for example, an optical drive, a magnetic media drive, a USB port, a receiver, a transceiver, or other computer readable media interface. The input interface 36 may provide the data to the decapsulation unit 38. [ Decapsulation unit 38 decapsulates the elements of the video file to retrieve the encoded data and, depending on whether the encoded data is part of an audio or video component, convert the encoded data to audio decoder 46 or video. May send to decoder 48. The audio decoder 46 decodes the encoded audio data and sends the decoded audio data to the audio output 42 while the video decoder 48 decodes the encoded video unit to generate a plurality of views To the video output unit 44. The video output unit 44 outputs the decoded video unit to the video output unit 44. [

2 is a block diagram illustrating the components of an exemplary encapsulation unit 30. 2, the encapsulation unit 30 includes a video input interface 80, an audio input interface 82, a file creation unit 60, and a video file output interface 84. File generation unit 60 includes, in this example, component assembly unit 62, component map box constructor 64, and component array (arr't) box constructor 66.

Video input interface 80 and audio input interface 82 receive encoded video and audio data, respectively. Video input interface 80 and audio input interface 82 may receive encoded video and audio data as the data is encoded or may extract video and audio data encoded from computer readable media. Upon receiving the encoded video and audio data, the video input interface 80 and the audio input interface 82 deliver the encoded video and audio data to the file creation unit 60 for assembly to a video file.

The file creation unit 60 may correspond to a control unit including hardware, software, and / or firmware configured to perform functions and procedures attributable to the control unit. In general, the control unit may further perform functions attributed to the encapsulation unit 30. [ In the examples where the file generation unit 60 is embedded in software and / or firmware, the encapsulation unit 30 includes (component assembly unit 62, component map box constructor 64, and component arrangement box configuration. Computer-readable medium containing instructions for one or more processors associated with file generation unit 60, as well as a processing unit that executes the instructions. Each of the sub-units (in this example, the component assembly unit 62, the component map box constructor 64, and the component arrangement box constructor 66) May be implemented as units and / or software modules, and may be functionally integrated into additional sub-units or further separated.

The file generation unit 60 may be implemented as, for example, one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs) And may correspond to any suitable processing unit or processing circuit, such as a processor. The file creation unit 60 may include a processor for executing the instructions as well as instructions for any or all of the component assembly unit 62, the component map box constructor 64, May also include non-temporal computer-readable media for storing data.

Generally, the file creation unit 60 may generate one or more video files containing the received audio and video data. The component assembly unit 62 may generate components of the content from the received encoded video and audio samples. The component may correspond to the number of segments, and each of the segments may include one or more video segments. Each of the segments may be independently retrieved by a client device, e.g., destination device 40. [ For example, the file creation unit 60 may assign a unique URL or URN to a file containing a segment. In general, the component assembly unit 62 may ensure that encoded samples belonging to the same component are assembled into that component. The component assembly unit 62 may also assign unique component identifiers to each of the components of the content. The file generating unit 60 may include data for more than one component in one file, and one component may encompass a plurality of files. The file creation unit 60 may store the data for the component as a track in the video file.

The component map box constructor 64 may generate a component map box for the multimedia content according to the techniques of the present disclosure. For example, the component map box may signal the characteristics of the components of the content. These characteristics include the average bit rate of the component, the maximum bit rate of the component, the resolution and frame rate of the component (assuming the component is a video component), dependencies on other components, or other characteristics You may. When dependencies are signaled, the component map box constructor 64 may also define a time layer difference between components having a dependency relationship. The component map box may also signal a set of potential multiplexing intervals or ranges of multiplexing intervals available to the component. In some instances, the file creation unit 60 may store the component map box in a separate file from all other files that contain coded samples for that content. In other examples, file generation unit 60 may include a component map box in one header of the video files.

By default, the component map box is applied to the entire content. However, when the component map box is applied only to a portion of the content, the component map box constructor 64 may signal the time duration of the content to which the component map box is applied. Component map box configurator 64 may then generate multiple component map boxes for its content in static mode or dynamic mode. In static mode, the component map box constructor 64 groups all component map boxes together in an order corresponding to the time durations of the content for which the component map boxes correspond. In the dynamic mode, component map box configurator 64 may place each component map box in a different location, eg, in different files.

The component map box may also signal whether the media fragment belongs to a new segment of the component. Because each segment of an element includes a component identifier, even if the segments are stored in separate files, segments belonging to the same component can be identified. The component map box may further signal the timing information for the parts of the component in files with encoded samples for the component. Therefore, time splicing is naturally supported. For example, a client device, e.g., destination device 40, may determine that two separate files contain data for the same component, and the time order of the two files.

The component arrangement box constructor 66 may generate component arrangement boxes for each file generated by the file creation unit 60. [ In general, the component arrangement box constructor 66 may identify not only which components are included in the file, but also the correspondence between component identifiers and track identifiers for the file. In this way, the component arrangement box may provide a mapping between component identifiers for the content and track identifiers for the file. The track identifiers may correspond to a track of a file having encoded samples for a component specified in the mapping.

The component arrangement box may also indicate how the fragments of each component are stored in the file. For example, the component arrangement box constructor 66 may include byte ranges for component fragments in a file, byte offsets for a particular fragment, decoding time of a first sample in the media fragment, Whether the access point is present in the fragment and, if present, its decoding and presentation times and whether the random access point is an IDR or ODR picture.

After the file generating unit 60 generates the file, the file output interface 84 may output the file. In some instances, the file output interface 84 may store the file on a computer readable storage medium, such as a hard disk. In some instances, the file output interface 84 may send the file via the output interface 32 (FIG. 1) to another device configured to operate as a server, such as an HTTP streaming server implementing HTTP 1.1. In some examples, file output interface 84 may provide the file to local storage medium such that output interface 32 may provide the file to client devices, such as destination device 40, for example in response to HTTP streaming requests. Can also be stored in

3 is a conceptual diagram showing an exemplary component map box 100 and a component arrangement box 152A. In this example, component map box 100 includes video components 110 and audio components 140. It should be noted that component map box 110 itself includes the signaled characteristics for video components 110 and audio components 140. 2, the component map box 100 and the component arrangement boxes 152 are defined by the file creation unit 60, for example, by the component map box constructor 64 and the component arrangement And may be generated by the box composer 66, respectively. In this way, the encapsulation unit 30 may signal the characteristics of files including multimedia content and data for the multimedia content. For example, video components 110 include signaled properties for components 112, and audio components 140 include signaled properties for components 142. For example, As shown in this example, component 112A includes component properties 114A.

Component properties 114A, in this example, include bitrate information 116, resolution information 118, frame rate information 120, codec information 122, profile and level information 124, dependencies information. 126, segment information 128, multiplexing interval information 130, and 3D video information 132.

The bit rate information 116 may include either or both of an average bit rate and a maximum bit rate for the component 112A. The bit rate information 116 may also include flags indicating whether average and / or maximum bit rate information is signaled. For example, the bit rate information 116 may include an average bit rate flag and a maximum bit rate flag, where the average bit rate flag indicates whether the average bit rate is signaled for the component 112A, The bit rate flag indicates whether the maximum bit rate is signaled for component 112A. Bit rate information 116 may also include an average bit rate value that represents the average bit rate for component 112A. Similarly, the bit rate information 116 may include a maximum bit rate value that represents a maximum bit rate value during a time period, e.g., an interval of one second.

Resolution information 118 may describe the resolution of component 112A, e.g., in terms of pixel width and pixel height of the image. In some cases, resolution information 118 may not be explicitly signaled for component 112A. For example, element properties 114A may include default property flags indicating whether an element with index i has the same properties of the same content element with index i-1. If the flags indicate that the characteristics are the same, then the characteristics need not be signaled. The default characteristics may include, for example, available options such as resolution, frame rate, codec information, profile information, and level information, or other combinations of characteristics that can be signaled by a component map box, May correspond to a subset of the properties. In some instances, the individual flags are included for each potential component that indicates whether the corresponding property for that component is the same as the previous component.

In some instances, the frame rate information 120 may be specified as a default characteristic as described above. Alternatively, the frame rate information 120 may define a frame rate for the component 112A. The frame rate may be defined as 256 frames per second of the video component. The codec information 122 may also be specified as a default characteristic, as described above. Alternatively, the codec information 122 may define an encoder used to encode the component 112A. Similarly, the profile and level information 124 may be specified as default properties or explicitly defined as, for example, the profile indicator (profile_idc) and level indicator (level_idc) values.

The dependencies information 126 may indicate whether the component 112A depends on other components of the components 110. For example, The dependency information 126 includes information indicating the time identifier for the component 112A and the difference between the time identifier for the component 112A and the time identifier for the component on which the component 112A depends . &Lt; / RTI &gt;

Segment information 128 describes the segments of component 112A. Segments may be stored in files, such as files 150. [ In the example of FIG. 3, data for segments of component 112A may be stored in file 150A, specifically in video track 158, as described in greater detail below. In some cases, segments for component 112A may be stored in multiple files. Each segment may correspond to one or more fragments. For each segment, the segment information 128 includes information such as whether the fragment includes a random access point, the type for the random access point (e.g., IDR or ODR), whether the fragment corresponds to a new file (e.g., a new segment) , Byte offset for the start of the fragment, timing information (e.g., decoding and / or display time) for the first sample of the fragment, byte offset for the next fragment, byte offset for the random access point if present, and ODR RAP It may signal the number of samples to skip decoding at the beginning of the stream.

Multiplexing interval information 130 may define a set or range of multiplexing intervals for component 112A. 3D video information 132 may be included when component 112A is used to generate a 3D effect, e.g., by simultaneously or nearly simultaneously displaying views of two or more slightly different scenes. The 3D video information 132 includes information such as the number of views to be displayed, identifiers for components corresponding to the views, start time of the 3D representation for a particular basic video component, time duration of the 3D representation, (E.g., the target width and target height of the 3D representation when ultimately displayed), positioning information (e.g., horizontal offset and vertical offset in the display window), a window layer representing the layer of the decoded video component for presentation, And may include a transparent factor. In general, a low window layer value may indicate that the associated video component is rendered early and may be covered by a video component with a higher layer value. Transparency level information may be combined with window level information. When a component is combined with another component having a lower window layer value, each pixel in the other components may be weighted with a value of [Transparency Level] / 255, collocated pixel may be weighted with a value of (255 - [transparency level]) / 255.

Figure 3 illustrates the correspondence between the components 112 and 142 and the various files 150 including the data for the components 112 and 142. In this example, file 150A includes encoded samples for video component 112A as a type of video track 158 and encoded samples for audio component 142A as a type of audio track 160 . The file 150A also includes a component arrangement box 152A. As also shown in this example, the component placement box 152A includes a component to video track map 154 and a component to audio track map 156 do. The component to video track map 154 indicates that the component identifier for component 112A is mapped to the video track 158 of file 150A. Similarly, the component to audio track map 156 indicates that the component identifier for the component 142A is mapped to the audio track 160 of the file 150A.

In this example, component 112B corresponds to video track 162 of file 150B, and component 142B corresponds to audio track 164 of file 150C. The component placement box 152B may include a mapping between the component 112B and the video track 162 while the component placement box 152C includes the component 142B and the audio track 164, Lt; / RTI &gt; In this way, the client device retrieves the component map box 100 and the component arrangement boxes 152 to determine which components to request, and how to add encoded data to the components from the files 150 And determine whether to access it.

The following pseudo code is an exemplary implementation of a data structure for a component map box.

aligned (8) class ComponentMapBox extends FullBox ('cmmp', version, 0) {

unsigned int (32) box_length;

unsigned int (64) content_ID;

unsigned int (32) timescale;

unsigned int (8) video_component_count;

unsigned int (8) audio_component_count;

unsigned int (8) other_component_count;

bit (1) first_cmmp_flag; // Default 1

bit (1) more_cmmp_flag; // default 0

bit (2) cmmp_byte_range_idc;

bit (1) multi_video_present_flag;

bit (2) multiplex_interval_idc;

bit (1) duration_signalled_flag;

bit (1) dynamic_component_map_mode_flag;

bit (7) reserved_bit;

if (duration_signalled_flag) {

unsigned int (64) starting_time;

unsigned int (64) duration;

}

for (i = 1; i < = video_component_count; i ++) {

unsigned int (8) component_ID;

bit (1) average_bitrate_flag;

bit (1) maximum_bitrate_flag;

bit (1) default_characteristics_flag;

bit (2) resolution_idc;

bit (2) frame_rate_idc;

bit (2) codec_info_idc;

bit (2) profile_level_idc;

bit (1) dependency_flag;

bit (1) 3DVideo_flag;

bit (2) reserved_flag;

// Bit rate

if (average_bitrate_flag)

unsigned int (32) avgBitrate;

if (maximum_bitrate_flage)

unsigned int (32) maxbitrate;

// resolution

if (! default_characteristics_flag) {

if (resolution_idc == 1) {

unsigned int (16) width;

unsigned int (16) height;

}

else if (resolution_idc == 2)

unsigned int (8) same_cha_component_id;

// When resolution_idc is 0, resolution is undefined,

// When the value is 3, the component with the index of i-1

// have the same resolution

// Frame rate

if (frame_rate_idc == 1)

unsigned int (32) frame_rate;

else if (frame_rate_idc == 2)

usngined int (8) same_cha_component_id;

// When frame_rate_idc is 0, the frame rate is undefined,

// When the value is 3, it is the same as the component with the index of i-1

// Has a frame rate.

if (codec_info_idc == 1)

string [32] compressorname;

else if (codec_info_idc == 2)

unsingedn int (8) same_cha_component_id;

// profile_level

if (profile_level_idc == 1)

profile_level;

else if (profile_level_idc == 2)

unsigned int (8) same_cha_component_id;

}

if (dependency_flag) {

unsigned int (8) dependent_comp_count;

bit (1) temporal_scalability;

unsigned int (3) temporal_id;

bit (4) reserved;

for (j = 1; j <= dependent_comp_count; j ++) {

unsigned int (6) dependent_comp_id;

if (temporal_scalability)

unsigned int (2) delta_time_id;

}

}

if (3DV_flag) {

unsigned int (8) number_target_views;

}

// segments

if (cmmp_byte_range_idc> 0) {

unsigned int (16) entry_count_byte_range;

for (j = 1; i < = entry_count; j ++) {

int (2) contains_RAP;

int (1) RAP_type;

bit (1) new_file_start_flag;

int (4) reserved;

unsigned int (32) reference_offset;

unsigned int (32) reference_delta_time;

if (cmmp_byt_rage_idc> 0)

unsigned int (32) next_fragment_offset;

if (contain_RAP> 1) {

unsigned int (32) RAP_delta_time;

unsigned int (32) delta_offset;

}

if (contain_RAP> 0 && RAP_type! = 0) {

unsigned int (32) delta_DT_PT;

unsigned int (8) number_skip_samples;

}

}

}

if (multiplex_interval_idc == 1) {

unsigned int (8) entry_count;

for (j = 1; j < = entry_count; j ++)

unsigned int (32) multiplex_time_interval;

}

else if (multiplex_interval_idc == 2) {

unsigned int (32) min_muliplex_time_interval;

unsigned int (32) max_muliplex_time_interval;

}

}

if (multi_video_present_flag) {

unsigned int (8) multi_video_group_count;

i (i = 1; i < = multi_video_group_count; i ++) {

unsigned int (8) basic_video_component_id;

unsigned int (8) extra_video_component_count;

int (64) media_time;

int (64) duration;

for (j = 1; j < = extra_video_component_count; j ++)

unsigned int (8) component_id;

for (j = 0; j <= extra_video_component_count; j ++) {

unsigned int (16) target_width;

unsigned int (16) target_height;

unsigned int (16) horizontal_offset;

unsigned int (16) vertical_offset;

unsigned int (4) window_layer;

unsigned int (8) transparent_level;

}

}

}

for (i = 1; i <= audio_component_count; i ++) {

unsigned int (8) component_ID;

bit (1) average_bitrate_flag;

bit (1) maximum_bitrate_flag;

bit (2) codec_info_idc;

bit (2) profile_level_idc;

...

// similar to syntax table for video components

}

}

In this example, the pseudo code elements have the following meanings. In other instances, it should be understood that other variable names and semantics may be assigned to the elements of the component map box. In this example, box_length represents the length of the component map box in units of bytes. The content_ID specifies a unique identifier of the content provided by the streaming server.

timescale is an integer defining the time-scale for the entire service. This is the number of time units that pass within one second. For example, the time coordinate system measuring time in 1/60 seconds has a time scale of 60.

video_component_count specifies the number of alternative video components that the service corresponding to the component map box can provide. Any two video components defined in this box can be switched to each other. If there are video presentations made up of multiple representations, such presentations may include one component belonging to an alternative group of video components.

audio_component_count specifies the number of alternative audio components that the service corresponding to the component map box can provide. Any two audio components defined in this box can be switched to each other. other_component_count specifies the number of other components of the service corresponding to the component map box.

first_cmmp_flag indicates whether this component map box is the first box of the same type for its associated service. more_cmmp_flag indicates whether this component map box is the last type of the same type for its associated service.

A value of 0, cmmp_byte_range_idc, indicates that the byte range and timing information are not signaled in the component map box. Cmmp_byte_range_idc with a value greater than zero indicates that byte range and timing information is signaled in the component map box. A value of 1, cmmp_byte_range_idc, indicates that only the start byte offset of the segment of the component is signaled. Cmmp_byte_range_idc with a value of 2 indicates that both the start byte offset and the end byte offset of the segment of the component are signaled.

temporal_scalability indicates whether the current component depends on the next signaled content components, at least one of the following signaled content components having a low temporal_id. temporal_id represents the time identifier of the current component. The temporal_id value may be ignored if the temporal_scalability value is zero. delta_temporal_id indicates the difference between the highest time identifier value of all of the samples of the current component and the highest time identifier value of the dependent component.

A multi_video_present_flag having a value of one indicates that there are video presentations rendered from more than one decoded video component. For example, for a picture in picture, multi_video_present_flag may have a value of 0, which may indicate that no video presentation is rendered by more than one decoded video component.

A multiplex_interval_idc having a value of 0 indicates that the multiplexing interval is not signaled. A multiplex_interval_idc value of 1 indicates that the list of multiplexing intervals is signaled. A multiplex_interval_idc having a value of 2 indicates that the range of multiplexing intervals is signaled.

The duration_signalled_flag indicates whether the duration of the corresponding service of the component map box is signaled. If the duration is not signaled (e.g., when duration_signalled_flag has a value of zero), the current component map box is assumed to apply to the entire service.

dynamic_component_map_mode_flag indicates whether the dynamic mode is supported for the current component map box. A value of 1, dynamic_component_map_mode_flag, indicates static mode and follows the current component map box in the same file, if there is a next component map box of the same service. Dynamic_component_map_mode_flag with a value of 0 indicates a dynamic mode, so that the next component map box of the same service will later be sent to the client by different means. For example, the following component map box may be included in the movie box of the next file.

The starting_time indicates the start time of the service applied by the current component map box. The duration represents the duration of the content to which the current component map box applies.

The component_ID is a unique identifier of the component. Average_bitrate_flag indicates whether the average bit rate is signaled for its associated component. maximum_bitrate_flag indicates whether the maximum bit rate is signaled for the associated component. The bit rate value 116 of FIG. 3 may correspond to either or both of average_bitrate_flag and maximum_bitrate_flag in the pseudo code. default_characteristics_flag indicates whether the current component having the index of i has the same values as the component having the index of i-1, for the following properties: resolution, frame rate, codec information, profile and level.

The resolution_idc / frame_rate_idc / codec_info_idc / profile_level_idc values set to zero indicate that the resolution, frame rate, codec information, profile, and / or level (respectively) of the associated video component are not signaled. The resolution_idc may correspond to the resolution value 118 of FIG. The frame_rate_idc may correspond to the frame rate value 120 of FIG. The codec_info_idc may correspond to the codec information (info) value 122 in Fig. profile_level_idc may correspond to the profile / level value 124 of FIG. 3. These values, which are set to 1, indicate that the resolution, frame rate, codec information, profile, or level (respectively) of the associated video component is the same as the video component with the index of i-1. Any of these values set to 2 indicates that each value is the same as one particular video component signaled using the value "same_cha_component_id ".

dependency_flag indicates whether the dependency of the current video component is signaled. When dependency_flag indicates that the component depends on other video components, the components upon which the current component depends may also be signaled. That is, if the dependency is signaled, the corresponding video component depends on the signaled video components. The dependency_flag value, in conjunction with the signaled video components upon which the current component depends, may correspond to the dependencies value 126 of FIG.

3DVideo_flag indicates whether the current video component is related to another video content providing MVC or 3D representation. number_target_views specifies the number of target views when, for example, decoding 3D video components coded with MVC (multi-view video coding). The entry_count_byte_range specifies the number of signaled fragments for the associated component. 3DVideo_flag, number_target_views, and entry_count_byte_range may generally correspond to the 3D video information value 132 of FIG.

avgbitrate represents the average bit rate of the associated component. maxbitrate represents the maximum bit rate of the associated component calculated at random seconds interval. The width and height, in units of luma pixels, represent the resolution of the decoded video component.

The same_resl_component_id indicates a component identifier of a video component having the same specific characteristics (resolution or frame rate or codec information, or profile and level) of the associated video component.

frame_rate represents the frame rate of the video component in frames per 256 seconds. The compressorname is a 4-byte value representing the brand of the codec, for example, "avc1 ". It has the same meanings as the major_brand of the file type box. The profile_level indicates the profile and level required to decode the current video component. dependent_comp_count indicates the number of video components that are dependent on the associated video components.

The dependent_comp_id specifies one component identifier of the video components on which the associated video component depends. At the same time instances, the samples in the different content elements may be reordered in ascending order of the index of the content elements. That is, a sample with an index of j may be placed earlier than a sample with an index of j + 1, and the sample of the current content element may be the last sample at that time instance.

The contains_RAP indicates whether the fragments of the component include a random access point. The contains_RAP is set to zero if the fragment does not contain any random access points. The contains_RAP is set to 1 if the fragment includes the random access point as the first sample in the fragment. The contains_RAP is set to 2 if the random access point is not the first sample of the fragment. RAP_type specifies the type of random access points (RAPs) included in the referenced track of the movie segment. RAP_type is set to 0 if the random access point is an instantaneous decoder refresh (IDR) image. RAP_type is set to 1 if the random access point is an open GOP random access point, e.g., an Open Decoder Refresh (ODR) image.

The flag new_file_start_flag indicates whether the fragment is the first fragment of the corresponding component in the file. This implies that the current fragment is in the new file. This signaling may be advantageous when relatively small size files are used in the server or when time splicing is used.

reference_offset indicates the offset to the start byte of the fragment in the file containing the fragment. reference_delta_time represents the decoding time of the associated fragment. next_fragment_offset indicates the start byte offset of the next fragment of the associated video component in the file containing the fragment. RAP_delta_time represents the decoding time difference between the first IDR random access point and the first sample of the fragment. delta_offset represents the byte offset difference between the byte offset of the first sample of the fragment and the byte offset of the random access point.

delta_DT_PT represents the difference between the decoding time and the presentation time for RAP, which is the ODR (Open GOP Random Access Point). number_skip_samples represents the number of samples having the presentation time preceding the ODR and the decomposition time after the ODR, which may be the first RAP of the referenced track of the movie segment. Note that the decoder may skip decoding of these samples when the decoder receives the stream starting at ODR. includes_RAP, RAP_type, new_file_start_flag, reference_offset, refrence_delta_time, next_fragment_offset, RAP_delta_time, delta_offset, delta_DT_PT, and number_skip_samples may generally correspond to the segment information 128.

multiplex_time_interval indicates the multiplexing interval in units of a timescale for the associated video component. Video components are typically associated with multiplexing interval information, but multiplexing interval information may also be signaled to audio components. The multiplex_time_interval may correspond to the multiplex interval value 130 of FIG. min_muliplex_time_interval and max_muliplex_time_interval indicate the range of the multiplexing interval in units of time scale for the associated video components. multi_video_group_count defines the number of video presentations to be displayed as a combination of multiple decoded video components.

basic_video_component_id specifies the component identifier of the base video component. Other signaling components that are signaled are considered as one alternative video presentation with the basic video component. For example, assume that the previous loop of "for video_component_count of video components" includes seven video components with CIDs 0, 1, 2, 3, 4, 5, It is additionally assumed that there is a basic_video_component_id with a number of 3 and two extra video components 5 and 6. Then, only the next group of presentations are alternatives to {0}, {1}, {2}, {3, 5, 6} and {4}.

media_time represents the start time of the multi-video representation with the basic video component having the identifier of basic_video_component_id. The duration defines the duration of a multi-video presentation having a primary video component with an identifier of basic_video_component_id. target_width and target_height define the target resolution of the video component in this multi-video presentation. Note that if this is not the same as the original resolution of the video, the destination device may perform scaling.

The horizontal_offset and vertical_offset define the offsets at the horizontal and vertical offsets in the display window. window_layer represents a layer of decoded video components for presentation. A lower layer value may indicate that the associated video component is rendered early and may be covered by a video component with a higher layer value. The decoded video components may be rendered in ascending order of window_layer values.

transparent_level indicates the transparency factor used when this decoded video is combined with window_layer lower than the current video component. For each pixel, the existing pixel may be weighted to a value of transparent_level / 255, and the pixels arranged together in the current decoded video component may be weighted to a value of (255 - transparent_level) / 255.

The following pseudocode is an exemplary implementation of a data structure for a component array box.

aligned (8) class ComponentArrangeBox extends FullBox ('cmar', version, 0) {

unsigned int (64) content_ID;

unsigned int (8) component_count;

bit (1) track_map_flag;

bit (1) sub_fragment_flag;

bit (1) agg_fragment_flag;

for (i = 1; i <= component_count; i ++) {

unsigned int (8) component_ID;

if (track_map_flag)

unsigned int (32) track_id;

}

if (sub_fragment_flag) {

unsigned int (8) major_component_count;

for (i = 1; i <= major_component_count; i ++) {

unsigned int (8) full_component_ID;

unsigned int (8) sub_set_component_count;

for (j = 1; j <sub_set_component_count; j ++)

unsigned int (8) sub_set_component_ID;

}

}

if (agg_fragment_flag) {

unsigned int (8) aggregated_component_count;

for (i = 1; i <= aggregated_component_count; i ++) {

unsigned int (8) aggr_component_id;

for (j = 1; j < = dependent_component_count; j ++)

unsigned int (8) depenedent_component_ID;

}

}

}

In this example, the semantics of the pseudo code elements are as follows. In other instances, it should be understood that other variable names and semantics may be assigned to the elements of the component arrangement box.

component_count specifies the number of components in the current file. track_map_flag indicates whether the mapping of tracks in this file to components of the service having the service ID of content_ID is signaled.

sub_fragment_flag indicates whether mapping of lower tracks in this file to elements is signaled. agg_fragment_flag indicates whether the mapping of track sets in this file to components is signaled. major_component_count represents the number of major components that contain all samples in the file.

The component_ID values represent the major components stored in the file, approximately, the identifiers of the first samples of each component in each movie segment. The track_id indicates an identifier of the track corresponding to the component having the component identifier of the component_ID.

sub_set_component_count represents the number of subcomponents forming the entire set of components having component_id of full_component_ID. The sub_set_component_ID specifies the component_id values of the subcomponents that form the complete set of components having the component_id of the full_component_ID. Any two subcomponents of the same component do not have nested samples.

In some examples, aggregated_component_count represents the number of content components aggregated from other components in the file. In some examples, aggregated_component_count represents the number of dependent components required to aggregate the aggregated content component with the component identifier of aggr_component_id. aggr_component_id specifies the component identifier of the aggregated component. depenedent_component_ID specifies the component id of the component used to aggregate the component with id of aggr_component_id.

Table 1 below shows another exemplary set of syntax objects that conform to the techniques of the present disclosure. The "element or attribute name" column describes the name of the syntax object. The "type" column describes whether the syntax object is an element or an attribute. The "element count" column describes the cardinality of the syntax object, i.e., the number of instances of the syntax object in the instance of the data structure corresponding to Table 1. The "optionality" column describes whether the syntax object is optional, in this example, "M" indicates mandatory, "O" indicates optional, "OD" indicates optional by default , "CM" indicates conditional compulsion. The "Description" column describes the semantics of the corresponding syntax object.

In the example of FIG. 3, the client device may request a component map box 100, which includes information about the characteristics of video components 110 and audio components 140 . For example, component 112A is described by component properties 114A. Similarly, each of the other components is described by component property information similar to component properties 114A. The client device may also extract component arrangement boxes 152, which may include component identifiers and tracks of audio and video data, such as video tracks 158 and 162, RTI ID = 0.0 &gt; 160, &lt; / RTI &gt; In this way, the component map box 100 is stored separately from the files 150, which include coded samples of audio and video data. The client device uses the data in the component map box 100 and the component arrangement boxes 152 to select a representation of the content and to render segments of the selected components in accordance with a network streaming protocol such as HTTP streaming You can also request it.

4 is a conceptual diagram illustrating an exemplary timing interval 190 for multiplexing the video component 180 and audio component 184. In this example, the representation includes a video component 180 and an audio component 184. The video component 180 includes video segments 182A-182D (video segments 182) while the audio component 184 includes audio fragments 186A-186C (audio fragments 186) ). The video fragments 182 may comprise encoded video samples while the audio fragments 186 may comprise encoded audio samples.

Video fragments 182 and audio fragments 186 may be arranged in decoding time order within video component 180 and audio component 184, respectively. Axis 188 in FIG. 4 represents decoding time information for video component 180 and audio component 184. In this example, the decoding time increases from left to right, as indicated by axis 188. Thus, the client device may decode the video fragment 182A, for example, prior to the video fragment 182B.

4 also shows an exemplary timing interval 190 of one second. According to the teachings of the present disclosure, the component map box for the video component 180 and the audio component 184 indicates that the timing interval 190 is one of the potential sets of timing intervals, Lt; RTI ID = 0.0 &gt; 190 &lt; / RTI &gt; The client device uses this information to avoid buffer overflows as well as to ensure that a sufficient amount of data that can be decoded before being streamed through the network is buffered, Element 180 and the audio component 184. &lt; RTI ID = 0.0 &gt;

As noted above, the disclosure is directed to network streaming situations, wherein the client continuously requests data from the server and decodes and renders the data as the data is retrieved. For example, during decoding and rendering of data in video segment 182A and audio segment 186A, the client device may request video segment 182B and audio segment 186B. As shown in the example of FIG. 4, video fragments 182 and audio fragments 186 are not necessarily time aligned. Thus, the client device may use the timing interval information to determine when to request data for subsequent fragments of the video component 180 and the audio component 184.

In general, the client device may be configured to retrieve fragments having a start decoding time within the next timing interval. If the component includes a fragment having a start decoding time within the next timing interval, the client may request the fragment. Otherwise, the client may skip the request for data from the component until a subsequent timing interval.

In the example of FIG. 4, the timing interval 190 is equal to one second. Decoding time values of video fragments 182 of video component 180 are, in this example, N-1 seconds for video fragment 182A, N + 1.2 seconds for video fragment 182B, and video fragment 182C. N + 2.1 seconds for N) and N + 3.3 for video fragment 182D. The decoding time values of the audio fragments 186 of the audio component 184 in this example are N.-2 seconds for the audio fragment 186A, N + 1.3 seconds for the audio fragment 186B, / RTI &gt; may be N + 3.2 seconds relative to the signal 186C.

As an example, assume that the next upcoming local decoding time at the destination device 40 is N + 2 seconds. Thus, the destination device 40 may determine which component fragments have a decoding time between N + 2 and N + 3 seconds, i.e., a local decoding time plus a timing interval 190. Fragments of components having start decoding times between N + 2 and N + 3 seconds may correspond to the next fragments requested. In this example, the video fragment 182C has a decoding time between N + 2 and N + 3 seconds. Thus, the destination device 40 may submit a request to retrieve the video fragment 182C, e.g., an HTTP partial retrieval request that specifies a byte range of the video fragment 182C. Since none of the audio fragments 186 has a decoding time between N + 2 and N + 3, the destination device 40 will still not submit the request for any fragments of the audio fragments 186 will be.

As another example, when the upcoming local decoding time is N + 3 seconds, destination device 40 may submits requests for video fragment 182D and audio fragment 186C. That is, video fragment 182D and audio fragment 186C have decoding times between N + 3 seconds and N + 4 seconds. Thus, the destination device 40 may submits requests for the video fragment 182D and the audio fragment 186C.

As yet another example, it is assumed that the timing interval 190 was 2 seconds instead. Destination device 40 may first determine that video segments 182B and 182C and audio fragment 186B have decoding times between N + 1 and N + 3 seconds if the local decoding time was N + 1 seconds have. Thus, the destination device 40 may submits requests for video fragments 182B and 182C and audio fragment 186B.

5 is a flowchart illustrating an exemplary method of providing a component map box and component arrangement boxes from a server to a client. Figure 5 also illustrates an exemplary method of selecting component elements forming a presentation and utilizing component map boxes and component arrangement boxes to request encoded samples of the selected components. It should be understood that while the source device 20 and destination device 40 of FIG. 1 are generally described, other devices may implement the techniques of FIG. For example, any server and client configured to communicate via a streaming protocol may implement these techniques.

First, the source device 20 may receive encoded video samples (200). The source device 20 may also receive encoded audio samples. The received samples correspond to common content. The received samples may correspond to various components of the content. Source device 20 may determine to which component the samples correspond. The source device 20 may also store samples in one or more files as components of the content (202). The source device 20 may arrange the samples into types of fragments so that the components may include one or more fragments and fragments of the same components may be stored in separate files.

After storing the components of the content in one or more files, the source device 20 may generate 204 the component arrangement boxes for each of the files. As described above, the component arrangement boxes may provide a mapping between component identifiers and track identifiers of a file. The source device 20 may also generate a component map box describing all of the components of the content (206). As described above, the component map box may include, for example, bit rates, frame rates, resolution, codec information, profile and level information, dependencies between components, segment information, multiplexing intervals, Segment information describing byte ranges of fragments within components, and / or 3D video information.

In some instances, the source device 20 may store component arrangement boxes, as well as files containing encoded video samples and component map boxes locally within the source device 20. In other examples, the source device 20 may send its files and the component map box to a separate server device provided to clients via a streaming network protocol. In the example of FIG. 5, assume that the source device 20 stores its files and implements a streaming network protocol, such as HTTP streaming.

Thus, the destination device 40 may request the source device 20 for a component map box and component arrangement boxes (208). For example, the destination device 40 may send a request for a component map box and component arrangement boxes to the source device 20 in accordance with HTTP streaming, e.g., a head request sent at the URL associated with the content have. In response to receiving the request 210, the source device 20 may provide a component map box and component arrangement boxes to the destination device 40 (212).

After receiving 214 the component map box and the component arrangement boxes, the destination device 40 may determine the components to be requested based on the data contained in the component map box. For example, the destination device 40 may determine whether the destination device 40 can decode and render certain components based on resolution, frame rate, codec information, profile and level information, and 3D video information It is possible. The destination device 40 may also determine current network conditions, such as available bandwidth, and select components based on the current network conditions. For example, the destination device 40 may select relatively low bit rates when less bandwidth is available, or relatively higher bit rates when more bandwidth is available. As another example, the destination device 40 may select the multiplexing interval based on the current network conditions, and may change the multiplexing interval based on possible multiplexing intervals as indicated by box, It is possible. If the selected component depends on another component, the destination device 40 may request both the selected component and the component upon which the selected component depends.

After selecting the components to request, the destination device 40 may determine 218 the files storing the data for the selected components based on the received component array boxes. For example, the destination device 40 may analyze its component arrangement boxes to determine whether the file has a mapping between the component identifier of the selected component and the track of the file. If so, the destination device 40 may request data from the file, e.g., according to a streaming network protocol (220). The requests may include HTTP acquisition or partial retrieval requests for URLs or URNs of the files, possibly defining byte ranges of the files, where the byte ranges may correspond to fragments of the components stored by the files have.

Destination device 40 may submit a number of requests to retrieve sequential portions of the components based on the selected multiplexing interval. That is, initially, destination device 40 may first request fragments from each of its selected components. The destination device 40 may then determine, for the next multiplexing interval, whether there is a fragment starting within the next multiplexing interval for each component, and if so, request the fragment (s). In this way, the destination device 40 may request fragments based on the multiplexing interval from the components. Destination device 40 may also periodically reevaluate network conditions, eg, by changing the multiplexing interval or by requesting data from different components, to perform adaptation to changing network conditions. In any case, in response to the requests, the source device 20 may output the requested data to the destination device 40 (222).

In one or more examples, the techniques described may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium, and executed by a hardware-based processing unit. The computer readable medium may also include a computer readable storage medium that facilitates transfer of a computer program from one location to another location, for example, in accordance with a communication protocol, And the like. In this way, the computer readable medium may generally correspond to (1) a non-transitory type of computer readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure . The computer program product may comprise a computer readable medium.

By way of example, and not limitation, such computer readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or desired program code. Or any other medium that can be used to store data in the form of devices or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer readable medium. For example, if commands are sent from a website, server, or other remote source using coaxial cable, fiber optic cable, double winding, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and microwave, Coaxial cable, fiber optic cable, double winding, DSL, or wireless technologies such as infrared, wireless, and microwave are included in the definition of the medium. However, it should be understood that the computer-readable storage medium and the data storage medium do not include connections, carrier waves, signals, or other temporal media, but instead are transmitted to a non-temporal, type of storage medium. A disk and a disc as used herein include a compact disk (CD), a laser disk, an optical disk, a digital versatile disk (DVD), a floppy disk and a Blu-ray disk, Discs usually reproduce data magnetically, while discs optically reproduce data with a laser. Combinations of the foregoing should also be included within the scope of computer readable media.

The instructions may be implemented in one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit Lt; / RTI &gt; Thus, the term "processor" when used herein may refer to any of the structures described above or any other structure suitable for implementation of the techniques described herein. Further, in some aspects, the functions described herein may be provided in dedicated software and / or software modules configured for encoding and decoding, or may be included in a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

The techniques of the present disclosure may be implemented in a wide variety of devices or devices, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chipset). The functional aspects of the various components, modules, or devices configured to perform the techniques disclosed by the units are described in this disclosure, but do not necessarily require realization by different hardware units. More precisely, as described above, several units may be coupled to a codec hardware unit or provided by a collection of interacting hardware units, including one or more processors as described above, along with suitable software and / or firmware have.

Several examples have been described. These and other examples are within the scope of the following claims.

Claims (54)

CLAIMS 1. A method of transmitting encapsulated video data,
Transmitting characteristics for components of the plurality of representations of video content to a client device, the characteristics comprising at least one of a frame rate, a profile indicator, a level indicator, and dependencies between the components. Transmitting the characteristics;
Receiving a request for at least one of the components from the client device after transmitting the properties; And
Transmitting the requested components to the client device in response to the request. The method of claim 1,
At least two of the components are stored in separate files,
Wherein transmitting the properties comprises transmitting a data structure comprising properties for each of the at least two components of the components. The method of claim 1,
Storing the properties for the components in a file separate from one or more files that store encoded samples for the components,
The method of claim 1,
Receiving a first request for the file in which the properties are stored; And
And responsive to the first request, transmitting the file independently of the one or more files storing the encoded samples,
Wherein the request for at least one of the video components comprises a second different request. The method of claim 1,
Storing properties for each of the components in a single data structure, the data structure being distinct from the components;
Assigning an identifier to the data structure to associate the data structure with the multimedia content comprising the plurality of representations; And
Further comprising assigning unique identifiers to the representations of the multimedia content,
Wherein the transmitting the characteristics comprises transmitting the data structure. The method of claim 1,
Wherein transmitting the properties further comprises transmitting component identifier values for the components,
At least one component identifier value of the component identifier values transmits encapsulated video data different from a track identifier value for a component corresponding to the at least one component identifier value of the component identifier values. Way. The method of claim 5, wherein
Further comprising transmitting information indicative of a correspondence between component identifier values for the components and track identifier values for the components in one or more files that store encoded samples for the components And transmitting the encapsulated video data. The method according to claim 6,
For each of the components of the one or more files, byte offsets for fragments in the component, decoding times of first samples in the fragments, random access points in the fragments, and the fragment Transmitting information indicative of whether they belong to a new segment of the component. The method of claim 1,
Wherein transmitting the properties comprises transmitting information indicating that the set of components is switchable to each other,
Wherein the request defines at least one of the set of components. The method of claim 1,
The step of transmitting the properties includes transmitting information indicating an ordering of the dependencies between the components and the dependencies between the components on the decoding order of the components in the access unit &Lt; / RTI &gt; The method of claim 1, further comprising transmitting the encapsulated video data. The method of claim 1,
Wherein transmitting the properties comprises transmitting information indicative of a time layer difference between dependencies between the components and a second component dependent on the first component and the first component , And transmitting the encapsulated video data. The method of claim 1,
Transmitting the characteristics comprises transmitting information indicative of a number of target views for output for one or more of the plurality of representations. The method of claim 1,
Wherein the transmitting of the characteristics comprises transmitting information indicating possible multiplexing intervals for a combination of two or more of the components,
The request defining fragments of any one of the two or more components of the components having decoding times within a common interval of the multiplexing intervals. The method of claim 1,
The characteristics comprising a first set of properties,
Transmitting the characteristics comprises transmitting information indicating a first time duration of the components to which the first set of characteristics corresponds;
The method comprises:
Further comprising transmitting a second set of properties for the components and a second time duration of the corresponding components of the second set of characteristics. An apparatus for transmitting encapsulated video data,
A processor configured to determine characteristics for components of a plurality of representations of video content, the characteristics comprising at least one of a frame rate, a profile indicator, a level indicator, and dependencies between the components; The processor; And
Receiving a request for at least one of the components from the client device after sending the properties to a client device, receiving the request for at least one of the components from the client device, The device comprising one or more interfaces configured to transmit to the device. 15. The method of claim 14,
Wherein the characteristics further comprise component identifier values for the components,
Wherein at least one component identifier value of the component identifier values is different from a track identifier value for a component corresponding to the at least one component identifier value of the component identifier values,
The properties include information indicative of a correspondence between component identifier values for the components and track identifier values for the components in one or more files storing encoded samples for the components. Device for transmitting encapsulated video data. The method of claim 15,
Characterized in that for each of the components of the one or more files, byte offsets for the fragments in the component, decoding times of the first samples in the fragments, random access points And information indicative of whether or not the segment belongs to a new segment of the component. 15. The method of claim 14,
The characteristics comprising information indicating an order of dependencies between the components and an order of dependencies between the components on the decoding order of the components in the access unit, the device transmitting the encapsulated video data . 15. The method of claim 14,
Wherein the properties include information indicating a time layer difference between dependencies between the components and a second component that relies on the first component and the first component to transmit the encapsulated video data Device. 15. The method of claim 14,
Wherein the properties include information representing a number of target views for output for one or more representations of the plurality of representations. 15. The method of claim 14,
The characteristics comprising information indicating possible multiplexing intervals for a combination of two or more of the components,
The request defining fragments of any of the two or more components of the components having decoding times within a common interval of the multiplexing intervals. 15. The method of claim 14,
The characteristics comprising a first set of properties,
The one or more interfaces are configured to transmit information indicative of a first time duration of the components to which the first set of characteristics corresponds;
The processor is further configured to generate a second set of properties for the components and a second set of characteristics corresponding to the second time duration of the components,
And the one or more interfaces are configured to transmit the second set of characteristics. 15. The method of claim 14,
The apparatus comprises:
integrated circuit;
A microprocessor; And
A wireless communication device comprising the processor
Gt; encapsulated &lt; / RTI &gt; video data. An apparatus for transmitting encapsulated video data,
Means for transmitting characteristics for components of a plurality of representations of video content to a client device, the characteristics comprising at least one of a frame rate, a profile indicator, a level indicator, and dependencies between the components. Means for transmitting the properties;
Means for receiving a request for at least one of the components from the client device after transmitting the properties; And
And means for sending the requested components to the client device in response to the request. 24. The method of claim 23,
Wherein the means for transmitting the characteristics comprises:
Means for transmitting component identifier values for the components, wherein the component identifier value of at least one of the component identifier values includes a component identifier value corresponding to the at least one component identifier value of the component identifier values Means for transmitting the component identifier values different from a track identifier value for the component identifier;
Means for transmitting information indicative of a correspondence between component identifier values for the components and track identifier values for the components in one or more files storing encoded samples for the components; And
For each of the components of the one or more files, byte offsets for fragments within the component, decoding times of the first samples in the fragments, random access points in the fragments, And means for transmitting information indicative of whether the segment belongs to a new segment of the component. 24. The method of claim 23,
The means for transmitting the characteristics comprises means for transmitting information indicative of the dependencies between the components and the order of the dependencies between the components with respect to the decoding order of the components in an access unit, Device for transmitting encapsulated video data. 24. The method of claim 23,
Wherein the means for transmitting the properties comprises means for transmitting information indicative of a time layer difference between dependencies between the components and a second component dependent on the first component and the first component , And transmits the encapsulated video data. 24. The method of claim 23,
Wherein the means for transmitting the characteristics comprises means for transmitting information indicating possible multiplexing intervals for a combination of two or more of the components,
The request defining fragments of any of the two or more components of the components having decoding times within a common interval of the multiplexing intervals. 24. The method of claim 23,
The characteristics comprising a first set of properties,
Wherein the means for transmitting the characteristics comprises means for transmitting information indicating a first time duration of the corresponding components of the first set of characteristics,
The apparatus comprises:
Further comprising means for transmitting a second set of properties for the components and a second time duration of a corresponding second component of the features. When executed, the processor of the source device transmitting the encoded video data,
Enabling to transmit characteristics for components of a plurality of representations of video content to a client device, the characteristics comprising at least one of a frame rate, a profile indicator, a level indicator, and dependencies between the components. Send the characteristics to a client device;
Receive a request for at least one of the components from the client device after transmitting the properties;
And a computer readable storage medium having stored thereon instructions for transmitting the requested components to the client device in response to the request. 30. The method of claim 29,
The instructions that cause the processor to transmit the characteristics include:
The processor,
Wherein the at least one component identifier value of at least one of the component identifier values comprises at least one of a component identifier value corresponding to the at least one component identifier value of the component identifier values, To transmit component identifier values for the components different from track identifier values for the component;
To transmit information indicative of a correspondence between component identifier values for the components and track identifier values for the components in one or more files that store encoded samples for the components;
For each of the components of the one or more files, byte offsets for fragments within the component, decoding times of the first samples in the fragments, random access points in the fragments, Further comprising instructions for causing the computer to transmit information indicative of whether a fragment belongs to a new segment of the component. 30. The method of claim 29,
The instructions that cause the processor to transmit the characteristics include:
Wherein the instructions cause the processor to perform the steps of: determining dependencies between the components; ordering dependencies between the components on a decoding order of the components in an access unit; And instructions to cause the computer to: transmit information indicative of a time layer difference between the second component. 30. The method of claim 29,
The instructions that cause the processor to transmit the characteristics include:
And cause the processor to transmit information indicating a number of target views for outputting one or more representations of the plurality of representations. 30. The method of claim 29,
The instructions that cause the processor to transmit the characteristics include instructions that cause the processor to transmit information indicative of possible multiplexing intervals for a combination of two or more of the components,
The request defining fragments of any one of the two or more components of the components having decoding times within a common interval of the multiplexing intervals. 30. The method of claim 29,
The characteristics comprising a first set of properties,
Sending the characteristics includes instructions that cause the processor to send information indicating a first time duration of the corresponding components of the first set of characteristics;
The computer program product,
And transmitting the second set of characteristics for the components and the second time duration of the components corresponding to the second set of characteristics. A method for receiving encapsulated video data,
Requesting a source device for characteristics of components of a plurality of representations of video content, the characteristics comprising at least one of a frame rate, a profile indicator, a level indicator, and dependencies between the components; Requesting the source device for the characteristics;
Selecting one or more of the components based on the characteristics;
Requesting samples of the selected components; And
And decoding and presenting the samples after the samples are received. 36. The method of claim 35,
Receiving information indicative of a correspondence between component identifier values for the selected components and track identifier values for the components in one or more files that store encoded samples for the components; And
Byte offsets for the fragments within each of the selected components, decoding times of the first samples in the fragments, random access points in the fragments, and the new segments of each component &Lt; / RTI &gt; further comprising the step of:
Wherein the step of requesting the samples further comprises the steps of: determining, based on the byte offsets, the decoding times, the random access points, and indications of whether the fragments belong to new segments, And requesting samples from the tracks of the one or more files corresponding to the track identifier values corresponding to the component identifier values. 36. The method of claim 35,
Receiving information indicating that at least one of the selected components depends on another component; And
Requesting samples of the component upon which one of the selected components depends. 36. The method of claim 35,
Wherein requesting the samples of the selected components comprises:
Determining a next multiplexing interval;
Determining, among the selected components, components having fragments starting at the next multiplexing interval; And
And requesting fragments starting at the next multiplexing interval from the determined ones of the selected components. 36. The method of claim 35,
The characteristics comprising a first set of properties,
The method comprises:
Receiving information indicative of a first time duration of the corresponding components of the first set of characteristics;
Requesting a second set of properties for the components, the second set of properties corresponding to a second time duration of the corresponding components; And
And requesting samples from the components corresponding to the second time duration based on the second set of properties. &Lt; Desc / Clms Page number 19 &gt; An apparatus for receiving encapsulated video data,
One or more interfaces configured to request a source device for characteristics of components of a plurality of representations of video content, the characteristics being at least one of a frame rate, a profile indicator, a level indicator, and dependencies between the components. The one or more interfaces, including one; And
A processor configured to select one or more of the components based on the characteristics and to submit the one or more interfaces to submit requests for samples of the selected components to the source device; Wherein the encapsulated video data is received by the device. 41. The method of claim 40,
The processor comprising:
Receiving information indicative of a correspondence between component identifier values for the selected components and track identifier values for the components in one or more files storing encoded samples for the components,
Byte offsets for the fragments within each of the selected components, decoding times of the first samples in the fragments, random access points in the fragments, and the new segments of each component The information indicating the presence or absence of the information,
Corresponding to said component identifier values for said selected components, based on said byte offsets, said decoding times, said random access points, and indications of whether said fragments belong to new segments, And configure the requests for the samples from tracks of the one or more files corresponding to track identifier values. 41. The method of claim 40,
The processor comprising:
Receiving information indicating that at least one of the selected components is dependent on another component,
Wherein the processor is configured to request samples of the component upon which one of the selected components depends. 41. The method of claim 40,
In order to generate the requests for the samples of the selected components,
Determine the next multiplexing interval,
Determining components of the selected components having fragments starting at the next multiplexing interval,
And request fragments starting at the next multiplexing interval from the determined ones of the selected components. 41. The method of claim 40,
The characteristics comprising a first set of properties,
The processor comprising:
Wherein the first set of properties receives information indicative of a first time duration of the corresponding component,
The second set of properties requesting a second set of properties for the components corresponding to a second time duration of the corresponding components,
And to request samples from the components corresponding to the second time duration based on the second set of characteristics. An apparatus for receiving encapsulated video data,
Means for requesting a source device characteristics for components of a plurality of representations of video content, the characteristics comprising at least one of a frame rate, a profile indicator, a level indicator, and dependencies between the components; Means for requesting a source device for said characteristics;
Means for selecting one or more of the components based on the characteristics;
Means for requesting samples of the selected components; And
Means for decoding and presenting the samples after the samples have been received. 46. The method of claim 45,
Means for receiving information indicative of a correspondence between component identifier values for the selected components and track identifier values for the components in one or more files that store encoded samples for the components; And
Byte offsets for the fragments within each of the selected components, decoding times of the first samples in the fragments, random access points in the fragments, and the new segments of each component Means for receiving information indicative of whether the user belongs to the group;
Wherein the means for requesting the samples further comprises means for determining, based on the byte offsets, the decoding times, the random access points, and indications of whether the fragments belong to new segments, And means for requesting samples from the tracks of the one or more files corresponding to the track identifier values corresponding to the component identifier values. 46. The method of claim 45,
Means for receiving information indicating that at least one of the selected components is dependent on another component; And
And means for requesting samples of the component upon which one of the selected components depends. 46. The method of claim 45,
Wherein the means for requesting samples of the selected components comprises:
Means for determining a next multiplexing interval;
Means for determining, among said selected components, components having fragments starting at said next multiplexing interval; And
And means for requesting fragments starting at the next multiplexing interval from the determined ones of the selected components. 46. The method of claim 45,
The characteristics comprising a first set of properties,
The apparatus comprises:
Means for receiving information indicative of a first time duration of the components to which the first set of characteristics corresponds;
Means for requesting a second set of properties for the components, the second set of properties corresponding to a second time duration of the corresponding components; And
And means for requesting samples from the components corresponding to the second time duration based on the second set of characteristics. When executed, causes the processor of the device to receive the encapsulated video data,
Requesting a source device for characteristics of components of a plurality of representations of video content, the characteristics comprising at least one of a frame rate, a profile indicator, a level indicator, and dependencies between the components; Request the source device for the characteristics;
Select one or more of the components based on the characteristics;
Request samples of the selected components;
The instructions comprising instructions for decoding and presenting the samples after the samples are received. 51. The method of claim 50,
The processor,
Receive information indicative of a correspondence between component identifier values for the selected components and track identifier values for the components in one or more files that store encoded samples for the components;
Byte offsets for the fragments within each of the selected components, decoding times of the first samples in the fragments, random access points in the fragments, and the new segments of each component The method comprising the steps of: receiving information indicating an indication of whether or not a user belongs to the group;
The instructions for causing the processor to request the samples include instructions for causing the processor to cause the processor to determine whether the byte offsets, the decoding times, the random access points, and indications of whether the fragments belong to new segments Instructions for requesting samples from the tracks of the one or more files corresponding to the track identifier values corresponding to the component identifier values for the selected components Computer program products. 51. The method of claim 50,
The processor,
Receive information indicating that at least one of the selected components is dependent on another component;
And instructions for requesting samples of the component upon which one of the selected components depends. 51. The method of claim 50,
Instructions for causing the processor to request samples of the selected components,
The processor,
Determine a next multiplexing interval;
Determine, among the selected components, components having fragments starting at the next multiplexing interval;
Instructions for requesting fragments starting at the next multiplexing interval from the determined ones of the selected components. 51. The method of claim 50,
The characteristics comprising a first set of properties,
The processor,
Cause the first set of characteristics to receive information indicative of a first time duration of the corresponding component;
Cause a second set of properties to request a second set of properties for the components corresponding to a second time duration of the corresponding components;
And based on the second set of characteristics, instructions for requesting samples from the components corresponding to the second time duration.

Images