TW202420830A - Identifying and marking video data units for network transport of video data - Google Patents

Abstract

An example device for retrieving media data includes a memory; and a processing system comprising one or more processors implemented in circuitry, the processing system being configured to:receive a packet including a packet header and a payload including at least a portion of a frame of video data, the packet header being separate from the payload; extract, from the packet header, a video frame identifier for the frame of video data; and process the payload according to the video frame identifier.

Description

Identifies and tags video data units for network transmission of video data

This patent application claims the benefit of U.S. Provisional Application No. 63/381,902, filed on November 1, 2022, the entire contents of which are incorporated herein by reference.

This case is about the transmission of encoded video data.

Digital video capabilities may be incorporated into a variety of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones, video teleconferencing equipment, etc. Digital video devices implement video compression techniques (such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 (also known as High Efficiency Video Coding (HEVC)), and extensions of such standards) to more efficiently send and receive digital video information.

Video compression techniques perform spatial prediction and/or temporal prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video frame or slice may be partitioned into macroblocks. Each macroblock may be further partitioned. Intra-frame coding (I) Macroblocks in a frame or slice are encoded using spatial prediction relative to neighboring macroblocks. Inter-frame coding (P or B) Macroblocks in a frame or slice may use spatial prediction relative to neighboring macroblocks in the same frame or slice or temporal prediction relative to other reference frames.

After the video data has been encoded, it can be packaged for transmission or storage. The video data can be assembled into video files that conform to any of a number of standards, such as the International Standards Organization (ISO) base media file format and its extensions, such as AVC.

Generally, techniques are described herein that relate to signaling characteristics of media data contained in a network packet in a packet header. Such characteristics may include, for example, an identifier of the media data (e.g., a slice of a picture and/or an identifier of the picture). The identifier may indicate or relate to whether the media data may be discarded under certain circumstances, such as depending on a random access technique used to begin receiving a bit stream that includes the media data. For example, the packet may be discarded if the media data depends on earlier media data of the bit stream not being received. This may occur if the media data is included in a Progressive Decoder Refresh (GDR) picture used for random access, if the media data is included in a Random Access Skippable Preamble (RASL) picture, or other such instances.

In one example, a method for receiving video data includes: receiving a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separated from the payload; extracting a video frame identifier of the frame of video data from the packet header; and processing the payload based on the video frame identifier.

In another example, a device for receiving video data includes: a memory configured to store the video data; and one or more processors implemented in a circuit system and configured to: receive a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separated from the payload; extract a video frame identifier of the frame of video data from the packet header; and process the payload based on the video frame identifier.

In another example, an apparatus for receiving video data includes: a component for receiving a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separated from the payload; a component for extracting a video frame identifier of the frame of video data from the packet header; and a component for processing the payload based on the video frame identifier.

In another example, a computer-readable storage medium has instructions stored thereon that, when executed, cause a processor to perform the following operations: receive a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separated from the payload; extract a video frame identifier of the frame of video data from the packet header; and process the payload based on the video frame identifier.

The details of one or more embodiments are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will become apparent from the description and drawings, and from the claims.

In general, the present invention describes technologies related to sending and receiving extended reality (XR) media data, such as augmented reality (AR), mixed reality (MR), and/or virtual reality (VR). For example, a media communication session between two network devices (such as a server device and a client device or two user equipment (UE) devices) may include audio data, video data, and/or XR data. Therefore, a user may participate in an XR communication session while communicating with one or more other users. The XR communication session may correspond to an XR-based telecommunication session, a game, etc.

Various issues related to XR and media (XRM) services are currently being studied. Two issues include packetization of protocol data unit (PDU) sets and the use of distinguishable PDU set processing, which generally involves enhancing PDU set processing at the fifth generation (5G) user plane function (UPF) to optimize the XRM consumer experience. A PDU set may include a plurality of PDUs, and each PDU may include data for a common presentation time. For example, a PDU set may include a PDU that includes data for a frame of video data and/or graphics XR data for computer generated graphics. Thus, a PDU set may correspond to a video frame, and each PDU may be a slice of a video frame or a network abstraction layer (NAL) unit.

In the 5G system (5GS), the interface between the application domain and the 5GS can be based on quality of service (QoS) flows. A QoS flow is the finest granularity of QoS differentiation within a PDU communication period. A QoS flow identifier (QFI) can be used to identify a QoS flow in the 5GS. User plane traffic with the same QFI within a PDU communication period can receive the same traffic forwarding treatment.

Each PDU may correspond to a packet transmitted, for example, over a computer-based network. Real-time Transport Protocol (RTP) or other protocols may be used to transmit PDUs. RTP is typically carried over the Uniform Datagram Protocol (UDP). Thus, packets may be delivered out of order, and UDP does not provide packet delivery guarantees. Furthermore, because the payload data may be encrypted or inaccessible to network elements, network-level packet processing typically does not have permission to access packet payload data (which may include video coding layer (VCL) data).

Thus, according to the techniques of the present invention, certain video information may be included in a packet header outside of the payload so that the packet may be processed by a network device that cannot access the payload data. Such data may include, for example, identification information of a frame or a portion of a frame of video data. The identification may specifically identify the frame, such as using a picture order count (POC) value that indicates the display order of the frame. As another example, the frame number may indicate a coding order value of the frame, which may be different from the display order. The identification information may also (supplementarily or alternatively) include data representing a coding layer, such as a time layer identifier, which typically corresponds to the number of possible reference frames that the frame can use for prediction and whether a subsequent frame can use the frame as a reference frame.

In this manner, a network device or network element of a user device (e.g., user equipment) can determine, for each received packet, identifier information of the media data included in the payload of the packet. Thus, the network device or network element can determine, for example, whether all packets of a frame have been received, whether a reference frame of a frame has been received, whether a subsequent frame that uses the frame for reference can be decoded (due to whether the frame has been received), etc. In this manner, the network device or network element can determine, for example, whether to provide video data in the payload of a packet to a video decoder, whether to retrieve a missing reference frame, whether to discard one or more sets of video data without sending the video data to a video decoder, etc.

The technology of the present invention can be applied to video files that conform to video data encapsulated according to any of the ISO base media file format, the scalable video coding (SVC) file format, the advanced video coding (AVC) file format, the third generation partnership project (3GPP) file format and/or the multi-view video coding (MVC) file format or other similar video file formats.

FIG. 1 is a block diagram illustrating an example system 10 implementing techniques for streaming media data via a network. In this example, system 10 includes content preparation device 20, server device 60, and client device 40. Client device 40 and server device 60 are communicatively coupled via network 74, which may include the Internet. In some examples, content preparation device 20 and server device 60 may also be coupled via network 74 or another network, or may be directly communicatively coupled. In some examples, content preparation device 20 and server device 60 may include the same device.

In the example of FIG. 1 , content preparation device 20 includes an audio source 22 and a video source 24. Audio source 22 may include, for example, a microphone that generates electrical signals representing captured audio data to be encoded by audio encoder 26. Alternatively, audio source 22 may include a storage medium storing previously recorded audio data, an audio data generator (such as a computerized synthesizer), or any other source of audio data. Video source 24 may include a video camera that generates video data to be encoded by video transcoder 28, a storage medium encoded with previously recorded video data, a video data generation unit (such as a computer graphics source), or any other source of video data. In all examples, the content preparation device 20 does not necessarily need to be communicatively coupled to the server device 60, but can store the multimedia content to a separate medium that is read by the server device 60.

The original audio and video data may include analog or digital data. The analog data may be digitized before being encoded by the audio encoder 26 and/or the video transcoder 28. The audio source 22 may obtain audio data from the speaking participant while the speaking participant is speaking, and the video source 24 may obtain video data of the speaking participant at the same time. In other examples, the audio source 22 may include a computer-readable storage medium containing stored audio data, and the video source 24 may include a computer-readable storage medium containing stored video data. In this way, the technology described in this case may be applied to live, streaming, real-time audio and video data or to archived, pre-recorded audio and video data.

An audio frame corresponding to a video frame is typically an audio frame containing audio data that is captured (or generated) by audio source 22 simultaneously with video data captured (or generated) by video source 24 contained within the video frame. For example, when a speaking participant typically generates audio data by speaking, audio source 22 captures audio data and video source 24 captures video data of the speaking participant simultaneously (i.e., while audio source 22 is capturing audio data). Thus, an audio frame may correspond in time to one or more particular video frames. Therefore, an audio frame corresponding to a video frame generally corresponds to a situation where audio data and video data are captured simultaneously, and for this situation, the audio frame and the video frame include the audio data and video data captured simultaneously, respectively.

In some examples, the audio encoder 26 may encode a timestamp in each encoded audio frame, the timestamp indicating the time when the audio data of the encoded audio frame was recorded, and similarly, the video transcoder 28 may encode a timestamp in each encoded video frame, the timestamp indicating the time when the video data of the encoded video frame was recorded. In such examples, the audio frames corresponding to the video frames may include audio frames containing timestamps and video frames containing the same timestamps. The content preparation device 20 may include an internal clock, and the audio encoder 26 and/or the video transcoder 28 may generate timestamps according to the internal clock, or the audio source 22 and the video source 24 may use the internal clock to associate audio and video data with timestamps, respectively.

In some examples, audio source 22 may send data corresponding to the time at which the audio data was recorded to audio encoder 26, and video source 24 may send data corresponding to the time at which the video data was recorded to video transcoder 28. In some examples, audio encoder 26 may encode a sequence identifier in the encoded audio data to indicate a relative temporal order of the encoded audio data, but not necessarily an absolute time at which the audio data was recorded, and similarly, video transcoder 28 may also use a sequence identifier to indicate a relative temporal order of the encoded video data. Similarly, in some examples, the sequence identifier may be mapped or associated with a timestamp.

The audio encoder 26 typically produces a coded audio data stream, while the video transcoder 28 produces a coded video data stream. Each individual data stream (whether audio or video) can be referred to as an elementary stream. An elementary stream is a single digitally encoded (possibly compressed) component of a media presentation. For example, the encoded video or audio portion of a media presentation can be an elementary stream. Before the elementary stream is encapsulated in a video file, the elementary stream can be converted into a packetized elementary stream (PES). Within the same media presentation, a stream ID can be used to distinguish a PES packet belonging to one elementary stream from another. The basic data unit of an elementary stream is a packetized elementary stream (PES) packet. Therefore, coded video data typically corresponds to an elementary video stream. Similarly, audio data corresponds to one or more corresponding elementary streams.

In the example of FIG. 1 , the encapsulation unit 30 of the content preparation device 20 receives an elementary stream including encoded video data from the video transcoder 28, and receives an elementary stream including encoded audio data from the audio encoder 26. In some examples, the video transcoder 28 and the audio encoder 26 may each include a packetizer for forming PES packets from the encoded data. In other examples, the video transcoder 28 and the audio encoder 26 may each be connected to a corresponding packetizer interface for forming PES packets from the encoded data. In still other examples, the encapsulation unit 30 may include a packetizer for forming PES packets from the encoded audio and video data.

The video transcoder 28 can encode the video data of the multimedia content in various ways to generate different representations of the multimedia content at various bit rates and with various characteristics (such as pixel resolution, frame playback rate, compliance with various coding standards, compliance with various profiles and/or profile levels of various coding standards, representations with one or more views (such as for two-dimensional or three-dimensional playback), or other such characteristics). As used in the present case, the representation may include audio data, video data, text data (such as for hidden subtitles), or one of other such data. The representation may include elementary streams, such as an audio elementary stream or a video elementary stream. Each PES packet may include a stream_id that identifies the elementary stream to which the PES packet belongs. The encapsulation unit 30 is responsible for assembling the elementary streams into media data that can be streamed.

The encapsulation unit 30 receives PES packets of the elementary streams of the media presentation from the audio encoder 26 and the video transcoder 28, and forms corresponding network abstraction layer (NAL) units from the PES packets. The encoded video segments can be organized into NAL units, which provide a "network-friendly" video representation, thereby addressing applications such as video telephony, storage, broadcasting, or data streaming. NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL units can contain the core compression engine and can include block, macroblock and/or slice level data. Other NAL units can be non-VCL NAL units. In some examples, a coded picture in a temporal instance (usually presented as a primary coded picture) may be contained in an access unit, which may include one or more NAL units.

Non-VCL NAL units can include parameter set NAL units and SEI NAL units, among others. Parameter sets can contain sequence-level header information (in a sequence parameter set (SPS)) and infrequently changing picture-level header information (in a picture parameter set (PPS)). With parameter sets (such as PPS and SPS), infrequently changing information does not need to be repeated for each sequence or picture; therefore, coding efficiency can be improved. In addition, the use of parameter sets enables out-of-band transmission of important header information, thereby avoiding the need for redundant transmission for error tolerance. In instances of out-of-band transmission, parameter set NAL units can be sent on a different channel from other NAL units (such as SEI NAL units).

Supplementary Enhancement Information (SEI) may contain information that is not necessary for decoding coded picture samples from VCL NAL units, but may assist procedures related to decoding, display, error resilience, and other purposes. SEI messages may be included in non-VCL NAL units. SEI messages are a regular part of some standard specifications and are therefore not always mandatory for standard-compliant decoder implementations. SEI messages may be sequence-level SEI messages or picture-level SEI messages. Some sequence-level information may be included in SEI messages, such as the scalability information SEI messages in instances of SVC and the view scalability information SEI messages in MVC. These instance SEI messages may convey information about, for example, the extraction of operation points and the characteristics of the operation points.

The server device 60 includes a real-time transport protocol (RTP) sending unit 70 and a network interface 72. In some examples, the server device 60 may include a plurality of network interfaces. In addition, any or all features of the server device 60 may be implemented on other devices of the content delivery network (such as routers, bridges, proxy devices, switches, or other devices). In some examples, a relay device of the content delivery network may cache data of the multimedia content 64 and include components that are substantially consistent with the components of the server device 60. In general, the network interface 72 is configured to send and receive data via the network 74.

The RTP sending unit 70 is configured to deliver media data to the client device 40 via the network 74 according to RTP, which is standardized in the Request for Comments (RFC) 3550 of the Internet Engineering Task Force (IETF). The RTP sending unit 70 can also implement protocols related to RTP, such as the RTP Control Protocol (RTCP), the Real-Time Streaming Protocol (RTSP), the Session Initiation Protocol (SIP) and/or the Session Description Protocol (SDP). The RTP sending unit 70 can send the media data via the network interface 72, which can implement the Uniform Datagram Protocol (UDP) and/or the Internet Protocol (IP). Thus, in some examples, server device 60 may use network 74 to send media data via UDP, via RTP, and via RTSP.

The RTP send unit 70 may receive an RTSP describe request from, for example, the client device 40. The RTSP describe request may include data indicating what types of data are supported by the client device 40. The RTP send unit 70 may respond to the client device 40 with data indicating a media stream, such as the media content 64, which may be sent to the client device 40 along with a corresponding network location identifier, such as a uniform resource locator (URL) or a uniform resource name (URN).

The RTP send unit 70 may then receive an RTSP setup request from the client device 40. The RTSP setup request may generally indicate how the media stream will be transmitted. The RTSP setup request may include a network location identifier of the requested media data (e.g., media content 64) and a transmission descriptor, such as a local port for receiving RTP data and control data (e.g., RTCP data) on the client device 40. The RTP send unit 70 may then reply to the RTSP setup request with data indicating a port of the server device 60 over which the RTP data and control data will be sent. The RTP send unit 70 may then receive an RTSP play request to cause the media stream to be "played," i.e., sent to the user client device 40 over the network 74. The RTP sending unit 70 may also receive an RTSP teardown request to end a streaming communication period, in response to which the RTP sending unit 70 may stop sending media data to the client device 40 for the corresponding communication period.

Likewise, the RTP receiving unit 52 may initiate a media stream by initially sending an RTSP describe request to the server device 60. The RTSP describe request may indicate the data types supported by the client device 40. The RTP receiving unit 52 may then receive a reply from the server device 60 specifying available media streams, such as media content 64, which may be sent to the client device 40 along with a corresponding network location identifier, such as a uniform resource locator (URL) or uniform resource name (URN).

The RTP receiving unit 52 may then generate an RTSP setup request and send the RTSP setup request to the server device 60. As mentioned above, the RTSP setup request may include a network location identifier of the requested media data (e.g., media content 64) and a transport descriptor, such as a local port for receiving RTP data and control data (e.g., RTCP data) on the client device 40. In response, the RTP receiving unit 52 may receive an acknowledgment from the server device 60 (including the port of the server device 60 that the server device 60 will use to send the media data and control data).

After the media stream communication period is established between the server device 60 and the client device 40, the RTP sending unit 70 of the server device 60 can send media data (e.g., media data packets) to the user client device 40 according to the media stream communication period. The server device 60 and the client device 40 can exchange control data (e.g., RTCP data) indicating, for example, reception statistics of the client device 40, so that the server device 60 can perform congestion control or otherwise diagnose and resolve transmission failures.

Network interface 54 may receive media of the selected media presentation and provide the media of the selected media presentation to RTP receiving unit 52, which in turn may provide the media data to decapsulation unit 50. Decapsulation unit 50 may decapsulate elements of the video file into constituent PES streams, depacketize the PES streams to retrieve the encoded data, and send the encoded data to audio decoder 46 or video decoder 48 depending on whether the encoded data is part of an audio stream or part of a video stream (e.g., as indicated by the PES packet headers of the streams). The audio decoder 46 decodes the encoded audio data and sends the decoded audio data to the audio output 42, while the video decoder 48 decodes the encoded video data and sends the decoded video data, which may include a plurality of views of the stream, to the video output 44.

The video transcoder 28, the video decoder 48, the audio encoder 26, the audio decoder 46, the packaging unit 30, the RTP receiving unit 52, and the decapsulation unit 50 can each be implemented as any of a variety of suitable processing circuit systems, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), individual logic circuit systems, software, hardware, firmware, or any combination thereof, as appropriate. Each of the video transcoder 28 and the video decoder 48 can be included in one or more encoders or decoders, and any of the encoders or decoders can be integrated as part of a combined video transcoder/decoder (transcoder). Likewise, each of the audio encoder 26 and the audio decoder 46 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined transcoder. A device including the video transcoder 28, the video decoder 48, the audio encoder 26, the audio decoder 46, the packaging unit 30, the RTP receiving unit 52, and/or the decapsulation unit 50 may include an integrated circuit, a microprocessor, and/or a wireless communication device, such as a cellular phone.

Client device 40, server device 60, and/or content preparation device 20 may be configured to operate according to the techniques of the present invention. For example, the present invention describes these techniques with respect to client device 40 and server device 60. However, it should be understood that content preparation device 20 may be configured to perform these techniques instead of server device 60 (or in addition to server device 60).

The encapsulation unit 30 may form a NAL unit including a header identifying the program to which the NAL unit belongs and a payload, such as audio data, video data, or data describing the transport or program stream to which the NAL unit corresponds. For example, in H.264/AVC, the NAL unit includes a 1-byte header and a payload of variable size. A NAL unit including video data in its payload may include video data at various levels of detail. For example, a NAL unit may include a block of video data, a plurality of blocks, a slice of video data, or an entire picture of video data. The encapsulation unit 30 may receive encoded video data in the form of PES packets of elementary streams from the video transcoder 28. The encapsulation unit 30 may associate each elementary stream with a corresponding program.

The encapsulation unit 30 may also assemble access units from a plurality of NAL units. In general, an access unit may include one or more NAL units for representing a frame of video data and audio data corresponding to the frame (when such audio data is available). An access unit typically includes all NAL units of an output time instance, such as all audio and video data of a time instance. For example, if each view has a picture playback rate of 20 frames per second (fps), each time instance may correspond to a time interval of 0.05 seconds. During the time interval, specific frames of all views of the same access unit (same time instance) may be presented simultaneously. In an example, in a time instance, an access unit may include a coded picture, which may be presented as a primary coded picture.

Thus, an access unit may include all audio and video frames of a common time instance, e.g., all views corresponding to time X. The coded pictures of a particular view are also referred to herein as "view components." That is, a view component may include coded pictures (or frames) of a particular view at a particular time. Thus, an access unit may be defined as including all view components of a common time instance. The decoding order of access units does not necessarily need to be the same as the output or display order.

After the packaging unit 30 has assembled the NAL units and/or access units into a video file based on the received data, the packaging unit 30 transmits the video file to the output interface 32 for output. In some examples, the packaging unit 30 may store the video file locally or send the video file to a remote server via the output interface 32, rather than directly sending the video file to the client device 40. The output interface 32 may include, for example, a transmitter, a transceiver, a device for writing data to a computer-readable medium, such as, for example, an optical disk drive, a magnetic media drive (e.g., a floppy disk drive), a universal serial bus (USB) port, a network interface, or other output interface. The output interface 32 outputs the video file to a computer readable medium, such as, for example, a transmission signal, magnetic media, optical media, memory, flash memory drive or other computer readable media.

Network interface 54 may receive NAL units or access units via network 74 and provide the NAL units or access units to decapsulation unit 50 via RTP receiving unit 52. Decapsulation unit 50 may decapsulate elements of the video file into constituent PES streams, depacketize the PES streams to retrieve the encoded data, and send the encoded data to audio decoder 46 or video decoder 48 depending on whether the encoded data is part of an audio stream or part of a video stream (e.g., as indicated by a PES packet header of the stream). The audio decoder 46 decodes the encoded audio data and sends the decoded audio data to the audio output 42, while the video decoder 48 decodes the encoded video data and sends the decoded video data, which may include a plurality of views of the stream, to the video output 44.

FIG2 is a conceptual diagram illustrating an example architecture for extended reality (XR) transport delivery. FIG2 illustrates an application/service layer 100, a user plane function (UPF) 102, an access network (AN) 110, and a client device 120. The client device 120 may correspond to the client device 40 of FIG1 and generally include components similar to those of the client device 40. The UPF 102 and the AN 110 may correspond to network devices within the network 74 of FIG1.

In this example, UPF 102 includes a packet detection unit 104 and packet detection rules 106, AN 110 includes a quality of service (QoS) 112 of an AN resource mapping unit and a radio interface plane 114, and client device 120 includes QoS rules 122, QoS 124 of an AN resource mapping unit, and a radio interface plane 126.

The client device 120 can send and receive media data, such as in the form of audio, video, and/or extended reality (XR) data. Data sent from the client device 120 to another device (such as another client device or the server device 60 of FIG. 1 ) is sent via an uplink stream, and data received by the client device 120 is received via a downlink stream. In a downlink (DL) extended reality and media (XRM) service stream, the UPF 102 can classify incoming data packets based on a set of packet filters of packet detection rules 106. The UPF 102 can communicate the classification of user plane traffic belonging to a QoS stream via a QoS flow identifier (QFI) marking. The QoS 112 of the AN resource mapping unit of the AN 110 constrains the QoS stream to the AN resources (i.e., the data radio bearer). For uplink (UL) XRM service streams, the client device 120 (which may be a user equipment (UE)) performs a PDU set identification procedure similar to that of the UPF 102. The client device 120 may mark the identified PDU set to the AN.

3GPP TR 23.700-60 reports candidate solutions for identifying PDU and PDU set boundaries by matching RTP/SRTP headers, header extensions, and payloads. New parameters such as PDU set sequence number, PDU set identifier, and PDU set type are proposed to identify PDUs and PDU sets. Candidate solutions also propose to mark PDU set priority, relevance, or importance by matching relevant parameters in RTP header extensions or payloads such as video network abstraction layer (NAL) type, time ID (TID), and layer ID (LID). The most important PDU or PDU set can be assigned to QoS streams with higher QoS requirements, while less important PDUs or PDU sets can be discarded during network congestion or when their associated PDUs or PDU sets are not fully delivered. It is also proposed that the AN can discard a PDU set if some PDUs in the PDU set are not received.

PDUs may be mapped to video data packets. An encoder or application function (AF) may be configured to identify video data packet characteristics such as grouping type and correlation and identifiers of data packets such as picture order counter (POC) values for video packets. Identifiers are typically embedded within the media data packet and are not exposed to the RTP header or header extension. During data encapsulation, filtering, and mapping to a QoS stream, the link between a specific video data packet and the corresponding characteristic may be lost, particularly for out-of-order delivery since the packet order may be disrupted. Because many RTP packets may share the same characteristics, adding fields for attributes or characteristics for each packet to the RTP header or header extension may increase management overhead. It may be beneficial to add a video data packet identifier to the RTP header or header extension as an index into a list of data packet characteristics.

PDU sets may also have tagged priority values. In some cases, a picture type, a time identifier (TID), or a layer identifier (LID) may be used to identify priority between PDU sets, and to mark PDU set priority or importance. However, multiple priority meanings may increase the complexity for relay devices to derive the final overall priority. Furthermore, the encoder may assign the same TID to all frames, and the importance of frames with the same TID may also vary. It generally makes sense to mark pictures that are coded within a frame as the highest priority, and pictures that are not used for reference as the lowest priority. However, there are cases where pictures are coded within a frame but are not used for reference (e.g., all-intraframe mode). A single indication may be needed to indicate the priority of pictures at the codec level for checking.

A PDU set may also have tagged dependency information. In most video coding schemes, picture dependencies are managed via reference picture management. Reference pictures are stored in the decoded picture buffer (DPB) for inter-frame prediction until the reference picture is marked as "unused for reference" and removed from the DPB.

ITU-T H.266/Versatile Video Coding (VVC) specifies two reference picture lists (RPLs), which are called list0 and list1. The predefined candidate RPLs are signaled in the sequence parameter set (SPS). The index of the reference predefined candidate RPLs is signaled in the picture header (PH) if all slices of a picture have the same RPL or in the slice header (SH). New RPLs can also be signaled directly in the picture header (PH) and slice header (SH). A reference picture can be a short-term reference picture, a long-term reference picture or an inter-layer reference picture, and if the reference picture is used for inter-layer prediction, it is marked with a picture order count (POC) and a layer ID. The reference picture used for inter-frame prediction of the current picture is the active reference picture of the current picture.

ITU-T H.265/High Efficiency Video Coding (HEVC) specifies reference pictures in a Reference Picture Set (RPS). The RPS can be signaled in the Sequence Parameter Set (SPS) or in the Slice Header (SH).

ITU-T H.264/Advanced Video Coding (AVC) specifies reference pictures based on two marking mechanisms: an implicit sliding window procedure and an explicit memory management control operation procedure. Typically, a maximum of 16 unique reference pictures are allowed for a given decoded picture buffer size.

AOMedia Video 1 (AV1) allows up to 7 reference frames and specifies the frame marking function in the frame header open bitstream unit (OBU). Reference picture indication can be present in the frame header OBU or in an additional header for each tile.

Due to different reference picture management designs in different codecs, it is complex for UPF 102 to export the relevance of each PDU set in a codec agnostic manner. The present invention recognizes that it would be beneficial to explicitly indicate picture relevance to support existing codecs at specific NAL units or SEI messages.

In addition, the reference picture tag is based on the POC value, which indicates the picture output order, while the frames or PDU sets in the bitstream are in coding order, so that the POC values may not be consecutive for adjacent frames in the bitstream. Mapping the POC value to the PDU set sequence number (SN) using existing schemes is not simple.

H.266 supports sub-picture partitioning, where each sub-picture can be decoded independently of other sub-pictures in the same frame. Even though all sub-pictures can share the same reference picture, a frame can contain a mix of intra-frame coded sub-pictures and inter-frame coded sub-pictures. Sub-picture identification and marking may require additional attributes to facilitate proper PDU processing. For example, when a sub-picture is independently coded and some PDUs in a PDU set are lost, the remaining PDUs can continue to be delivered. When a sub-picture is not independently coded and some PDUs are lost, the remaining PDUs can be discarded.

AV1 supports tile lists, which contain tile data associated with a frame, and each tile can be decoded independently. Tile lists allow the decoder to process a subset of tiles and display the corresponding part of the frame without fully decoding all tiles in the frame.

FIG3 is a conceptual diagram illustrating a series of progressive decoder refresh (GDR) frames of video data. Generally, when random access is performed (i.e., streaming transmission is started at a point in the video other than the start of the video), the stream is accessed starting from a stream access point. The stream access point may be fully intraframe predictive coded so that the entire frame can be decoded and replay can start from the stream access point. Such frames may be referred to as transient decoder refresh (IDR) pictures. However, fully intraframe predictive coded frames have a relatively high bit rate.

Therefore, instead of having a single IDR stream access point, a progressive decoder refresh (GDR) stream access point can be used. In general, a GDR stream access point includes a sequence of frames that include portions that are intra-frame prediction coded, while other portions are inter-frame prediction coded. When randomly accessed from a GDR stream access point, the inter-frame prediction coded portions may be decodable or non-decodable based on which side of the intra-frame prediction portion the inter-frame prediction portion appears on.

GDR enables the encoder to smooth the bit rate of the bit stream by distributing intra-frame coded slices or blocks across multiple pictures, as opposed to intra-frame coding of the entire picture, thereby significantly reducing end-to-end latency, which is especially important for ultra-low latency applications. When the decoding process starts with the decoding of a GDR picture, some areas of the picture cannot be decoded correctly. After decoding a number of additional pictures, called a recovery cycle, the entire picture at the recovery point and all subsequent pictures in output order will be decoded correctly. Figure 3 illustrates an example of such a GDR picture recovery cycle, where the clean areas and intra-frame predicted areas are the areas that can be decoded correctly, and the dirty areas are the areas that cannot be decoded correctly for random access. Therefore, PDUs of dirty areas that cannot be decoded correctly can be discarded when random access occurs at the associated GDR picture. The identification and marking of PDUs and PDU sets for GDR pictures has not been addressed in the candidate solutions.

More specifically, in the example of FIG. 3 , GDR frames 140A to 140D are illustrated. GDR frame 140A includes an intra-frame prediction coding area 142A and an undecodable inter-frame prediction coding area 144A. GDR frame 140B includes a decodable inter-frame prediction coding area 146B, an intra-frame prediction coding area 142B, and an undecodable inter-frame prediction coding area 144B. GDR frame 140C includes a decodable inter-frame prediction coding area 146C, an intra-frame prediction coding area 142C, and an undecodable inter-frame prediction coding area 144C. GDR frame 140D includes a decodable inter-frame prediction coding area 146D and an intra-frame prediction coding area 142D.

Because the undecodable inter-frame prediction coded areas 144A to 144C can refer to a reference frame preceding the GDR frame 140A in the coding order, the undecodable inter-frame prediction coded areas 144A to 144C are undecodable when random access is performed starting from the GDR frame 140A. The decodable inter-frame prediction areas 146B to 146D are decodable because they can only be predicted from a reference frame starting from the GDR frame 140A and can only be predicted from a decodable inter-frame prediction area or an intra-frame prediction area.

In H.266/VVC, when PictureOutputFlag is equal to 0, the picture is not output. For example, when the picture is a Random Access Skippable Leading (RASL) picture and the NoOutputBeforeRecoveryFlag of the associated Intra-frame Random Access Point (IRAP) picture is equal to 1, the picture is not output; when the picture is a Progressive Decoding Refresh (GDR) picture with NoOutputBeforeRecoveryFlag equal to 1 or a recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, the picture is not output; when the ph_pic_output_flag of the picture is equal to 0, the picture is not output. The value of NoOutputBeforeRecoveryFlag can be set by an external component or by the associated picture position in the bitstream. If the PDU set is not output and is not used as a reference for subsequent picture prediction, the PDU set may be discarded.

FIG4 is a conceptual diagram illustrating an example set of identification information for a protocol data unit (PDU). This case describes techniques that can use high-level syntax design in a video transcoder to facilitate PDU and PDU set identification and marking. An identifier of a video frame (e.g., POC value in AVC/HEVC/VVC, display_frame_id or current_frame_id in AV1) can be added to an RTP header extension and further marked in a PDU set or PDU. The application function (AF) can use the video frame identifier to indicate video frame characteristics such as frame type, priority, and relevance to the UPF 102 ( FIG2 ) via the policy control function (PCF) and the communication management function (SMF). UPF 102 may classify video packets for QoS marking by examining the frame identifier of the data packet and linking it to the corresponding characteristics for QoS marking. FIG4 illustrates an example of using the video frame ID as an index to link a PDU set to a table or list containing the associated video data characteristics.

In some instances, a slice or tile identifier (e.g., slice segment address in HEVC, slice_address in VVC, tile count in AV1) indicating a specific slice/tile within a frame may be signaled in an RTP header extension along with the frame identifier. The slice/tile identifier may be mapped to a PDU attribute field. The AF may indicate video slice/tile characteristics to the UPF 102 using the frame identifier and the slice/tile identifier. The UPF 102 may classify video packets for QoS marking by inspecting the frame identifier and slice/tile identifier of the data packet and linking these values to the corresponding characteristics for QoS marking.

In some examples, a frame marking an RTP header extension may include priority marking information. For example, a frame marking an RTP header extension may include data indicating whether the corresponding frame is intra-frame predictive coded (I frame), a discardable frame (D), a time ID (TID) and/or a layer ID (LID) of the frame. Any or all of this data may represent a PDU set priority marking. This priority attribute data may be signaled in a network abstraction layer (NAL) unit header, a specific NAL unit, or a supplemental enhancement information (SEI) message used for PDU set priority marking checking.

In some examples, a NAL unit priority indication nuh_priority can be signaled in a NAL unit header or header extension. The priority indication can be a 3-bit code that specifies the priority of the associated NAL unit, which takes a value between 1 and 7, where 1 represents the highest priority and 7 represents the lowest priority. Table 1 below is an example of an indication of NAL unit priority in a NAL unit header extension. In Table 1, "[addition: ""]" indicates an addition relative to an existing NAL unit syntax (e.g., of ITU-T H.266), and "[removal: ""]" indicates a deletion relative to an existing NAL unit syntax. Table 1 nal_unit_header() { Descriptor forbidden_zero_bit f(1) [Removed: " nuh_reserved_zero_bit "] u(1) nuh_layer_id u(6) nal_unit_type u(5) nuh_temporal_id_plus1 u(3) [Added: " nuh_extention_flag "] u(1) [Add: “if( nuh_extension_flag ) {“] [Add: " nuh_priority "] u(3) [Add: " nuh_reserved_zero_5bits "] u(5) } }

The semantics of the added syntactic elements in the example of Table 1 may be as follows:

nuh_extension_flag equal to 0 specifies that the nuh_priority and muh_reserved_zero_5bits syntax elements are not present in the NAL unit header syntax structure. nuh_extension_flag equal to 1 specifies that the nuh_priority and muh_reserved_zero_5bits syntax elements may be present in the NAL unit header syntax structure.

nuh_priority specifies the NAL unit priority. nuh_priority equal to 1 indicates the highest priority, while nuh_priority equal to 7 indicates the lowest priority.

As another example, the picture priority indication aud_priority may be signaled in the AU delimiter (AUD), as shown in Table 2: Table 2 access_unit_delimiter_rbsp( ) { Descriptor aud_irap_or_gdr_flag u(1) aud_pic_type u(3) [Added: " aud_priority "] u(3) rbsp_trailing_bits( ) }

The semantics of the added syntactic elements in the example of Table 2 may be as follows:

aud_priority specifies the priority of the access unit (AU) containing the AU delimiter. aud_priority equal to 1 indicates the highest priority, while aud_priority equal to 7 indicates the lowest priority.

As yet another example, the picture priority indication may be signaled in the picture header (PH), as shown in Table 3: Table 3 picture_header_structure() { Descriptor ph_gdr_or_irap_pic_flag u(1) [Removed: " ph_non_ref_pic_flag "] u(1) [Add: " ph_priority "] u(3) if( ph_gdr_or_irap_pic_flag ) ph_gdr_pic_flag u(1) ph_inter_slice_allowed_flag u(1)

The semantics of the added syntactic elements in the example of Table 3 may be as follows:

ph_priority specifies the priority of the current picture. A ph_priority of 1 means the highest priority, while a ph_priority of 7 means the lowest priority. The lowest priority means that the current picture is never used as a reference picture.

In yet another example, the picture priority indication may specify the priority between pictures that share the same TID and LID of the current picture. In this case, the PDU set priority flag may be derived from ph_priority, TID and LID.

In some examples, a slice priority indication may be signaled in a slice header (SH) to indicate the priority between slices within the same picture or between slices of a picture sharing the same TID and LID.

In some examples, picture or slice priority can be signaled in a SEI message to indicate picture priority or slice priority using a slice address.

In some instances, the frame priority indication may be signaled in a specific OBU of the AV1 codec, such as a frame header OBU or a metadata OBU. The tile priority indication may be signaled in a tile group OBU or a tile list OBU. Table 4 is an example of the syntax of a frame header OBU with a frame priority indication: Table 4 uncompressed_header() { Type if ( frame_id_numbers_present_flag ) {     idLen = ( additional_frame_id_length_minus_1 + delta_frame_id_length_minus_2 + 3 ) } allFrames = (1 << NUM_REF_FRAMES) - 1 if ( reduced_still_picture_header ) { … } Otherwise{ show_existing_frame f(1) … frame_type f(2) [Add: “frame_priority”] f(3) …

The frame_priority syntax element may be signaled or mapped to other protocols such as the RTP header extension or the GPRS Tunneling Protocol User Plane (GTP-U) extension header.

Deriving PDU set dependencies from a reference picture list is typically very complex. Likewise, there is an additional cost required to map POC values to PDU set sequence numbers (SNs). Therefore, according to the present invention, the distance between the current picture and the active reference picture in the bitstream (e.g., POC distance) may be marked in a specific NAL unit or SEI message. In the bitstream, all active reference pictures precede the current picture. Depending on the PDU set boundary, the distance may be in units of access units (AUs) or picture units (PUs).

Table 5 below is an example syntax structure indicating a list of active reference pictures associated with the current picture in a bitstream. The list includes short-term active reference pictures, long-term active reference pictures, and inter-layer active reference pictures. Table 5 act_ref_pic_list_struct( ) { Descriptor num_st_act_ref_pos u(8) for( i = 0; i <num_st_act_ref_pos; i++) { st_act_ref_delta_pos [i] u(16) } num_lt_act_ref_pos u(8) for( i = 0; i <num_lt_act_ref_pos; i++) { lt_act_ref_delta_pos [i] u(32) } num_il_act_ref_pos u(8) for( i = 0; i <num_il_act_ref_pos; i++) { il_act_ref_delta_pos [i] u(8) } }

The semantics of the syntactic elements in the example of Table 5 may be as follows:

num_st_act_ref_pos specifies the number of short-term activity reference picture positions in the syntax structure.

st_act_ref_delta_pos specifies the distance between the current picture and the active short-term reference picture in access units (AU).

For a single-layer bitstream, an AU contains pictures. Assuming the sequence number (SN) of the current PDU set is N, the SN of the PDU set containing the i-th short-term active reference picture is (N - st_act_ref_delta_pos[ i ]).

In another example, st_act_ref_delta_pos can specify the distance between the current PU and the active short-term reference picture in units of pictures.

num_lt_act_ref_pos specifies the number of long-term activity reference picture positions in the syntax structure.

lt_act_ref_delta_pos specifies the distance in AU between the current picture and the active short-term reference picture.

For a single-layer bitstream, an AU contains pictures. Assuming the SN of the current picture or PDU set is N, the SN of the PDU set containing the i-th long-term active reference picture is (N - st_act_ref_delta_pos[ i ]).

In some implementations, lt_act_ref_delta_pos can specify the distance in pictures between the current picture and the active long-term reference picture.

num_il_act_ref_pos specifies the number of inter-layer activity reference picture positions in the syntax structure.

lt_act_ref_delta_pos specifies the delta between the current picture and the reference picture of the active layer.

Assuming that the SN of the current picture or PDU set is N, and each PDU set contains pictures, the SN of the PDU set containing the i-th inter-layer active reference picture is (N - il_act_ref_delta_pos[ i ]).

In some examples, when each PDU set contains pictures instead of AUs, the data lengths of st_act_ref_delta_pos, lt_act_ref_delta_pos, and il_act_ref_delta_pos may be extended to accommodate the number of layers.

Table 6 below illustrates an example of a simplified activity reference picture list structure: Table 6 act_ref_pic_list_struct( ) { Descriptor num_act_ref_pos u(8) for( i = 0; i <num_st_act_ref_pos; i++) { act_ref_delta_pos [i] u(32) } num_il_act_ref_pos u(8) for( i = 0; i <num_il_act_ref_pos; i++) { il_act_ref_delta_pos [i] u(8) } }

The semantics of the syntactic elements of the example in Table 6 may be as follows:

num_act_ref_pos specifies the number of act reference picture positions in the syntax structure.

act_ref_delta_pos specifies the distance between the current picture and the active reference picture in access units (AU). Assuming the SN of a PDU set is N, the SN of the PDU set containing the i-th active reference picture is (N - act_ref_delta_pos[ i ]).

num_il_act_ref_pos specifies the number of inter-layer activity reference picture positions in the syntax structure.

il_act_ref_delta_pos specifies the distance between the current picture and the active reference picture in picture units (PUs). Assuming the SN of a PDU set is N, the SN of the PDU set containing the i-th inter-layer active reference picture is (N - il_act_ref_delta_pos[ i ]).

The proposed active reference picture location syntax structure can be carried in the AU delimiter (AUD) or in the SEI message.

In some examples, the reference picture distance in the above syntax structure can be replaced by a sequence number indicating the AU or PU position in the bitstream. This sequence number increases by one for each new AU or PU added to the bitstream. The sequence number can be mapped to the PDU set sequence number to facilitate export frame correlation.

In some examples, the position of a reference picture in the decoding order may be signaled in a specific metadata type OBU (e.g. metadata_itut_t35) or a new metadata type OBU in AV1 to indicate the relevance of the current picture in the bitstream.

In some examples, the proposed active reference picture list syntax element may be signaled or mapped to other protocols such as RTP or GTP-U extension headers.

In some instances, an active reference picture list structure may be carried in an active reference picture location list SEI message. A persistence flag is proposed in the SEI message to indicate whether the active reference picture list applies to the current picture only or to the current picture and all subsequent pictures of the current layer. Persistence may be canceled when the proposed cancel flag is set or a new coded layer video sequence (CLVS) for the current layer starts.

A standalone PDU flag can be signaled to indicate whether the current PDU can be decoded without other PDUs of the same PDU set. A standalone PDU can be transmitted even if other PDUs in the same PDU set are missing. A standalone PDU can be a slice consisting of an independent sub-picture or a slice consisting of motion constrained tile sets (MCTS) in an H265 bitstream.

In some examples, a syntax element may be signaled in a specific video codec NAL unit to indicate that all slice boundaries in a CLVS (when signaled in an SPS) or associated pictures (when signaled in a PPS, PH, or AUD) are considered picture boundaries and there is no loop filtering across slice boundaries. Each slice may be decoded independently without reference to samples of other slices in the same picture.

In some instances, a syntax element can be signaled in a slice header (SH) to indicate whether the current slice can be decoded independently without prediction based on other samples in the same frame.

Since each tile of AV1 can be decoded independently, if the PDU is an AV1 tile and there is a tile list OBU, the independent PDU flag can be set for each PDU.

For HEVC bitstreams, a PDU set of a random access skippable leading (RASL) picture can be marked as a discarded picture in the frame flag or PDU set flag when the NoRaslOutputFlag of the associated intra-frame random access point (IRAP) picture is equal to 1.

For VVC bitstreams, a PDU set of a RASL picture with pps_mixed_nalu_types_in_pic_flag equal to 0 can be marked as a discarded picture in the frame flag or PDU set flag when the NoOutputBeforeRecoveryFlag of the associated IRAP picture is equal to 1.

Since detecting discardable pictures is not straightforward, syntax elements may be signaled to indicate whether the associated picture may be discarded for transmission or decoding in specific NAL units (such as AUD, PH, or SH) or SEI messages.

For VVC bitstreams, a RASL picture with pps_mixed_nalu_types_in_pic_flag equal to 1 may contain one or more RASL sub-pictures and one or more random access decodable leading (RADL) sub-pictures. RADL sub-pictures may be used as active reference sub-pictures, which shall not be discarded. Although RASL sub-pictures or slices of a RASL picture may be discarded, the associated PDUs may be marked as discardable PDUs.

For VVC bitstreams, slices of GDR pictures that cannot be decoded correctly may be discarded when the NoOutputBeforeRecoveryFlag of a GDR picture is equal to 1 or the NoOutputBeforeRecoveryFlag of a recovery picture of a GDR picture is equal to 1. The corresponding slices that cannot be decoded correctly may be marked in the RTP header extension so that the UPF 102 may discard the associated PDUs during congestion time.

For VVC, the picture header syntax element ph_non_ref_pic_flag indicates that the current picture has never been used as a reference picture. This syntax element can be used for discardable marking. For AV1 transcoders, the discardable or non-reference indication can be signaled in the frame header OBU uncompressed header or tile group OBU to indicate whether the associated frame is not used as a reference picture for subsequent pictures and can be discarded without affecting the decoding process. Table 7 below is an example of such a frame header OBU uncompressed header syntax with a discardable frame indication, which is indicated by the tag: "[add: ""]". Table 7 uncompressed_header() { Type if ( frame_id_numbers_present_flag ) {     idLen = ( additional_frame_id_length_minus_1 + delta_frame_id_length_minus_2 + 3 ) } allFrames = (1 << NUM_REF_FRAMES) - 1 if ( reduced_still_picture_header ) { … } Otherwise{ show_existing_frame f(1) … frame_type f(2) FrameIsIntra = (frame_type == INTRA_ONLY_FRAME || frame_type == KEY_FRAME) … [Add: "if ( !FrameIsIntra )"] [Add: “frame_discardable”] f(1) …

Since the discard of a slice or frame can be decided on the fly, an indicator can be signaled in a SEI message or in an overwritable field of a specific NAL unit or metadata type OBU to indicate whether the associated NAL unit can be discarded.

In some examples, a slice may be marked as a discardable slice when any of the following conditions is true: the current picture is a RASL picture and the current slice NAL unit type is RASL; the current picture is a GDR picture and the current slice is a P or B slice (unidirectional inter-frame prediction or bidirectional inter-frame prediction); or the current picture is a recovery picture of a GDR picture and the current slice is a P or B slice following a previous I slice in the same picture.

FIG5 is a block diagram illustrating elements of an example video file 150. As previously described, video files according to the ISO base media file format and its extensions store data in a series of objects referred to as "boxes." In the example of FIG5 , the video file 150 includes a file type (FTYP) box 152, a movie (MOOV) box 154, a segment index (sidx) box 162, a movie fragment (MOOF) box 164, and a movie fragment random access (MFRA) box 166. Although FIG5 represents an example of a video file, it should be understood that other media files according to the ISO base media file format and its extensions may include other types of media data (e.g., audio data, timed text data, etc.) structured in a similar manner to the data of the video file 150.

The file type (FTYP) box 152 generally describes the file type of the video file 150. The file type box 152 may include information identifying specifications describing the best use of the video file 150. The file type box 152 may alternatively be placed before the MOOV box 154, the movie clip box 164, and/or the MFRA box 166.

5 , the MOOV box 154 includes a movie header (MVHD) box 156, a track (TRAK) box 158, and one or more movie extension (MVEX) boxes 160. In general, the MVHD box 156 may describe general characteristics of the video file 150. For example, the MVHD box 156 may include data describing the time when the video file 150 was initially created, the time when the video file 150 was last modified, the time stamp of the video file 150, the playback duration of the video file 150, or other data generally describing the video file 150.

TRAK box 158 may include data for a track of video file 150. TRAK box 158 may include a track header (TKHD) box that describes characteristics of the track corresponding to TRAK box 158. In some examples, TRAK box 158 may include an encoded video picture, while in other examples, the encoded video picture of the track may be included in movie clip 164, which may be referenced by data in TRAK box 158 and/or sidx box 162.

In some examples, the video file 150 may include more than one track. Thus, the MOOV box 154 may include a number of TRAK boxes equal to the number of tracks in the video file 150. The TRAK box 158 may describe characteristics of a corresponding track of the video file 150. For example, the TRAK box 158 may describe temporal and/or spatial information of the corresponding track. When the encapsulation unit 30 ( FIG. 1 ) includes a parameter set track in a video file such as the video file 150, a TRAK box similar to the TRAK box 158 of the MOOV box 154 may describe characteristics of the parameter set track. The encapsulation unit 30 may signal the presence of a sequence-level SEI message in a parameter set track within a TRAK box describing the parameter set track.

The MVEX box 160 may describe characteristics of the corresponding movie segment 164, for example, to signal that the video file 150 includes the movie segment 164 in addition to the video data (if any) included in the MOOV box 154. In the context of streaming video data, the encoded video pictures may be included in the movie segment 164 instead of the MOOV box 154. Thus, all encoded video samples may be included in the movie segment 164 instead of the MOOV box 154.

The MOOV box 154 may include a number of MVEX boxes 160 equal to the number of movie clips 164 in the video file 150. Each of the MVEX boxes 160 may describe characteristics of a corresponding one of the movie clips 164. For example, each MVEX box may include a Movie Extended Header (MEHD) box that describes the time duration of the corresponding one of the movie clips 164.

As mentioned above, encapsulation unit 30 may store sequence data sets in video samples that do not include actual coded video data. Video samples may typically correspond to an access unit, which is a representation of a coded picture for a particular instance in time. In the context of AVC, a coded picture includes one or more VCL NAL units that contain information, such as SEI information, for all picture elements used to construct the access unit and other associated non-VCL NAL units. Thus, encapsulation unit 30 may include a sequence data set in one of the movie segments 164 that may include sequence-level SEI information. Packaging unit 30 may also signal the presence of a sequence data set and/or sequence-level SEI information as present in one of the movie segments 164 within one of the MVEX boxes 160 that corresponds to one of the movie segments 164 .

SIDX box 162 is an optional element of video file 150. That is, a video file conforming to the 3GPP file format or other such file formats does not necessarily include SIDX box 162. According to an example of the 3GPP file format, a SIDX box may be used to identify a sub-segment within a segment (e.g., a segment contained within video file 150). The 3GPP file format defines a sub-segment as "a self-contained collection of one or more consecutive movie segment boxes with corresponding media data frames, and a media data frame containing data referenced by a movie segment box must follow the movie segment box and precede the next movie segment box containing information about the same track." The 3GPP file format also indicates that the SIDX box "contains a series of references to subsegments within the (sub)segment recorded by the box. The referenced subsegments are contiguous in presentation time. Similarly, the bytes referenced by the Segment Index box are always contiguous within a segment. The referenced size provides a count of the number of bytes in the referenced material."

The SIDX box 162 generally provides information representing one or more sub-segments in the segment included in the video file 150. For example, such information may include the replay time at which the sub-segment begins and/or ends, the byte offset of the sub-segment, whether the sub-segment includes (e.g., begins at) a stream access point (SAP), the type of SAP (e.g., whether the SAP is a transient decoder refresh (IDR) picture, a clean random access (CRA) picture, a broken link access (BLA) picture, etc.), the location of the SAP in the sub-segment (in terms of replay time and/or byte offset), etc.

Movie segment 164 may include one or more encoded video pictures. In some examples, movie segment 164 may include one or more groups of pictures (GOPs), each of which may include a number of encoded video pictures, such as frames or pictures. In addition, as previously described, in some examples, movie segment 164 may include a sequence data set. Each of movie segment 164 may include a movie segment header box (MFHD, not shown in FIG. 5 ). The MFHD box may describe the characteristics of the corresponding movie segment, such as the sequence number of the movie segment. Movie segment 164 may be included in the order of the sequence numbers in video file 150.

MFRA box 166 may describe a random access point within movie segment 164 of video file 150. This may facilitate trick modes, such as seeking to a specific time position (i.e., replay time) within a segment encapsulated by video file 150. In some examples, MFRA box 166 is generally optional and need not be included in a video file. Likewise, a client device (such as client device 40) does not necessarily need to reference MFRA box 166 to correctly decode and display the video data of video file 150. MFRA block 166 may include a number of track fragment random access (TFRA) blocks (not shown) equal to the number of tracks of video file 150, or in some examples, equal to the number of media tracks (e.g., non-cue tracks) of video file 150.

In some examples, the movie segment 164 may include one or more stream access points (SAPs), such as IDR pictures. Similarly, the MFRA box 166 may provide an indication of the location of the SAP within the video file 150. Therefore, a time sub-sequence of the video file 150 may be formed by the SAPs of the video file 150. The time sub-sequence may also include other pictures, such as P frames and/or B frames that depend on the SAPs. The frames and/or slices of the time sub-sequence may be arranged within the segment so that the frames/slices of the time sub-sequence that depend on other frames/slices of the sub-sequence can be decoded correctly. For example, in a layered arrangement of data, data used for prediction of other data may also be included in the time sub-sequence.

FIG6 is a flow chart illustrating an example method of sending a packet including media data according to the present invention. Initially, a server device (such as server device 60 of FIG1 ) may receive media data (200), for example, from content preparation device 20 ( FIG1 ). The media data may be packaged media data or encoded media data. The media data may include at least a portion of a frame/picture of video data, such as a slice or tile.

The server device 60 may also receive data indicating whether the media data is discardable (202) and an identifier of the media data (204). The identifier may be, for example, a frame number, a picture order count (POC) value, and/or other such identifiers. The identifier may also indicate, for example, a time identifier (TID), a layer identifier (LID), etc. The identifier may also indicate a specific slice, tile, or other portion of a frame or picture represented by the media data.

The server device 60 may then encapsulate the media data into a packet (206). Alternatively, in some examples, the received media data may already be encapsulated in a network packet. In any case, according to the method of FIG. 6 , the server device 60 may add data to an RTP header extension of the packet that indicates the identifier (208). The identifier itself may indicate whether the media data of the packet is discardable, or the server device 60 may also add data to the RTP header extension that indicates whether the media data of the packet is discardable. For example, the media data may be discardable if the media data is predicted based on reference media data that has not been sent to the client device, for example because the reference media data is before the random access point accessed by the client device. The server device 60 may then send the packet to the client device.

FIG. 7 is a flow chart illustrating an example method including receiving a packet including media data according to the present invention. In this example, for example, the client device 40 of FIG. 1 may initially receive a packet including media data (250). That is, the packet may include a packet header and a payload separated from the packet. The payload may correspond to application layer data of the packet (e.g., media data formatted according to the ISO base media file format), as explained above with respect to FIG. 5. For example, the payload may include XR data, audio data, and/or video data of a PDU set.

According to the technology of the present case, the packet header may include an RTP header extension. Therefore, in addition to other data, the RTP header extension may include an identifier of the media data of the packet. The client device 40 can extract the identifier of the media data from the RTP header extension (252). The user client device 40 can then use the identifier to determine whether the media data is discardable (254). For example, the client device 40 can determine that the identifier includes a POC value of a picture of the media data, a slice or tile identifier of the picture, a TID and/or LID of the picture. The client device 40 can also determine whether the bit stream including the media data is randomly accessed from a stream access point other than the beginning of the bit stream. If the bit stream is accessed randomly, the client device 40 can also determine whether the stream access point follows one or more reference pictures of the packetized media data, so that the reference pictures will not be received. If the reference pictures have not been received, the client device 40 can determine that the media data is discardable.

In response to determining that the media data is discardable (the "yes" branch of 256), client device 40 may discard the media data without sending the media data to, for example, video decoder 48 (FIG. 1) (258). In response to determining that the media data is not discardable (the "no" branch of 256), client device 40 may forward the media data to video decoder 48 (260).

Although explained with respect to the client device 40 of Figure 1, the method of Figure 7 may also be performed by, for example, the client device 120 of Figure 2. Likewise, a similar method may be performed by the UPF 102 of Figure 2. In an example where the method of Figure 7 is performed by the UPF 102 or another device, when the media data is sent to a video decoder, it may be assumed that the video decoder forms part of the client device 120, such that sending the media data to the video decoder includes sending packets to the client device 120.

In this manner, the method of FIG. 7 represents an example of a method comprising: receiving a packet comprising a packet header and a payload, the payload comprising at least a portion of a frame of video data, the packet header being separated from the payload; extracting a video frame identifier of the frame of video data from the packet header; and processing the payload based on the video frame identifier.

Various examples of the techniques of this case are outlined in the following clauses.

Clause 1: A method for receiving video data, the method comprising: receiving a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separate from the payload; extracting a video frame identifier of the frame of video data from the packet header; and processing the payload based on the video frame identifier.

Clause 2: A method according to clause 1, wherein the video frame identifier comprises a picture order count (POC) value.

Clause 3: A method according to clause 1, wherein the video frame identifier comprises a display frame identifier or a current frame identifier.

Clause 4: A method according to any of clauses 1 to 3, wherein at least a portion of the frame comprises a slice of the frame, and wherein the video frame identifier comprises a slice identifier of the slice.

Clause 5: A method according to any of clauses 1 to 4, wherein at least a portion of the frame comprises a tile of the frame, and wherein the video frame identifier comprises a tile identifier of the tile.

Clause 6: A method according to any one of clauses 1 to 5, the method also comprising: using the video frame identifier to determine one or more of a frame type of the frame, a priority of the frame, or relevance information of the frame.

Clause 7: A method according to any of clauses 1 to 6, wherein at least a portion of the frames of video data comprises a protocol data unit (PDU) in a set of PDUs.

Clause 8: A method according to any one of clauses 1 to 7, the method also comprising: extracting one or more of the following from a packet header: a network abstraction layer (NAL) unit type of at least a portion of a frame, a time identifier (TID) of at least a portion of a frame, a layer identifier (LID) of at least a portion of a frame, data indicating whether at least a portion of a frame is predictively coded within a frame, or data indicating whether at least a portion is discardable.

Clause 9: A method according to any one of clauses 1 to 8, the method also comprising: processing a network abstraction layer (NAL) unit header of a frame, the NAL unit header comprising data indicating a priority value of the frame.

Clause 10: A method according to any one of clauses 1 to 9, the method also comprising: processing an access unit delimiter (AUD) of an access unit corresponding to a frame, the AUD comprising data representing a priority value of the access unit.

Clause 11: A method according to any one of clauses 1 to 10, the method also comprising: processing a picture header of a frame, the picture header comprising data representing a priority value of the frame.

Clause 12: A method according to any one of clauses 1 to 8, the method also comprising: processing an open bitstream unit (OBU) of at least a portion of the frame, the OBU comprising data representing a priority value of at least a portion of the frame.

Clause 13: A method according to clause 12, wherein the OBU comprises one of a frame header OBU or a metadata OBU.

Clause 14: A method according to any one of clauses 1 to 13, the method also comprising: receiving data indicating a picture distance between a frame and a reference frame in a coding order of the frames.

Clause 15: A method according to clause 14, wherein receiving data indicating a picture distance comprises: receiving a network abstraction layer (NAL) unit including the data or a supplemental enhancement information (SEI) message including the data.

Clause 16: A method according to any one of clauses 14 and 15, wherein receiving data indicating a picture distance between a frame and a reference frame of the frame comprises: receiving data indicating a picture distance between the frame and each active reference frame in coding order.

Clause 17: A method according to any one of clauses 1 to 16, the method also comprising: receiving information indicating whether a portion of a frame of video data can be decoded without other portions of the frame.

Clause 18: A method according to clause 17, wherein the portion of the frame comprises a protocol data unit (PDU), wherein the frame corresponds to a PDU set comprising the PDU, and wherein the information indicates whether the PDU can be decoded without other PDUs in the PDU set.

Clause 19: A method according to any one of clauses 1 to 18, the method also comprising: receiving information indicating whether loop filtering is to be performed across one or more boundaries between a portion of the frame and one or more other portions of the frame.

Clause 20: A method according to any one of clauses 1 to 19, the method also comprising: receiving information indicating whether a portion of the frame is independently encoded without prediction based on other portions of the frame.

Clause 21: A method according to any one of clauses 1 to 20, the method also comprising: receiving data indicating that the frame is a discardable frame.

Clause 22: A method according to clause 21, wherein receiving data indicating that a frame is a discardable frame comprises: receiving the frame using a network abstraction layer (NAL) unit, an access unit delimiter, a picture header, a slice header, a supplemental enhancement information (SEI) message, or a frame header open bitstream unit (OBU).

Clause 23: A method according to any of clauses 1 to 22, wherein processing the payload comprises: determining that the frame is a progressive decoder refresh (GDR) frame and determining that at least a portion of the frame is independently encoded; and providing at least a portion of the frame to a video decoder in response to determining that at least a portion of the frame is independently encoded.

Clause 24: A method according to any of clauses 1 to 22, wherein processing the payload comprises: determining that the frame is a progressive decoder refresh (GDR) frame and determining that at least a portion of the frame is encoded relative to a reference frame; determining that the GDR frame is the first frame in sequence to be retrieved for a bit stream comprising video data such that a reference frame has not yet been retrieved; and in response to the reference frame not having been retrieved, discarding at least a portion of the video frame.

Clause 25: An apparatus for retrieving media data, the apparatus comprising one or more components for performing a method according to any of clauses 1 to 24.

Clause 26: Apparatus according to clause 25, wherein one or more components comprise one or more processors implemented in a circuit system.

Clause 27: A device according to clause 25, which also comprises a memory configured to store video data.

Clause 28: Apparatus according to clause 25, wherein the apparatus comprises at least one of: an integrated circuit; a microprocessor; and a wireless communication device.

Clause 29: An apparatus for receiving media data, the apparatus comprising: means for receiving a packet comprising a packet header and a payload, the payload comprising at least a portion of a frame of video data, the packet header being separate from the payload; means for extracting a video frame identifier of the frame of video data from the packet header; and means for processing the payload based on the video frame identifier.

Clause 30: A method for receiving video data, the method comprising: receiving a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separated from the payload; extracting a video frame identifier of the frame of video data from the packet header; and processing the payload based on the video frame identifier.

Clause 31: A method according to clause 30, wherein the video frame identifier comprises a picture order count (POC) value.

Clause 32: A method according to clause 30, wherein the video frame identifier comprises a display frame identifier or a current frame identifier.

Clause 33: A method according to clause 30, wherein at least a portion of the frame comprises a slice of the frame, and wherein the video frame identifier comprises a slice identifier of the slice.

Clause 34: A method according to clause 30, wherein at least a portion of the frame comprises a tile of the frame, and wherein the video frame identifier comprises a tile identifier of the tile.

Clause 35: A method according to clause 30, the method also comprising: using the video frame identifier to determine one or more of a frame type of the frame, a priority of the frame, or relevance information of the frame.

Clause 36: A method according to clause 30, wherein at least a portion of the frames of video data comprises protocol data units (PDUs) in a set of PDUs.

Clause 37: A method according to clause 30, the method also comprising: extracting one or more of the following from a packet header: a network abstraction layer (NAL) unit type of at least a portion of a frame, a time identifier (TID) of at least a portion of a frame, a layer identifier (LID) of at least a portion of a frame, data indicating whether at least a portion of a frame is predictively coded within a frame, or data indicating whether at least a portion is discardable.

Clause 38: A method according to clause 30, the method also comprising: processing a network abstraction layer (NAL) unit header of a frame, the NAL unit header comprising data indicating a priority value of the frame.

Clause 39: A method according to clause 30, the method also comprising: processing an access unit delimiter (AUD) of an access unit corresponding to a frame, the AUD comprising data representing a priority value of the access unit.

Clause 40: A method according to clause 30, the method also comprising: processing a picture header of a frame, the picture header comprising data representing a priority value of the frame.

Clause 41: The method of clause 30, further comprising: processing an open bitstream unit (OBU) of at least a portion of the frame, the OBU comprising data representing a priority value of at least a portion of the frame.

Clause 42: A method according to clause 41, wherein the OBU comprises one of a frame header OBU or a metadata OBU.

Clause 43: A method according to clause 30, the method also comprising: receiving data indicating a picture distance between a frame and a reference frame of the frame.

Clause 44: A method according to clause 43, wherein receiving data indicating a picture distance comprises: receiving a network abstraction layer (NAL) unit including the data or a supplemental enhancement information (SEI) message including the data.

Clause 45: A method according to clause 43, wherein receiving data indicating a picture distance between a frame and a reference frame of the frame comprises: receiving data indicating a picture distance between a frame and each active reference frame.

Clause 46: A method according to clause 30, the method also comprising: receiving information indicating whether a portion of a frame of video data can be decoded without other portions of the frame.

Clause 47: A method according to clause 46, wherein the portion of the frame comprises a protocol data unit (PDU), wherein the frame corresponds to a PDU set comprising the PDU, and wherein the information indicates whether the PDU can be decoded without other PDUs in the PDU set.

Clause 48: A method according to clause 30, the method also comprising: receiving information indicating whether loop filtering is to be performed across one or more boundaries between a portion of the frame and one or more other portions of the frame.

Clause 49: A method according to clause 30, the method also comprising: receiving information indicating whether a portion of a frame is independently encoded without prediction based on other portions of the frame.

Clause 50: A method according to clause 30, the method also comprising: receiving data indicating that the frame is a discardable frame.

Clause 51: A method according to clause 50, wherein receiving data indicating that a frame is a discardable frame comprises: receiving the frame in a network abstraction layer (NAL) unit, an access unit delimiter, a picture header, a slice header, a supplemental enhancement information (SEI) message, or a frame header open bitstream unit (OBU).

Clause 52: A method according to clause 30, wherein processing the payload comprises: determining that the frame is a progressive decoder refresh (GDR) frame and determining that at least a portion of the frame is independently encoded; and providing at least a portion of the frame to a video decoder in response to determining that at least a portion of the frame is independently encoded.

Clause 53: A method according to clause 30, wherein processing the payload comprises: determining that the frame is a progressive decoder refresh (GDR) frame and determining that at least a portion of the frame is encoded relative to a reference frame; determining that the GDR frame is a first frame in sequence to be retrieved for a bit stream comprising video data such that a reference frame has not been retrieved; and in response to the reference frame not being retrieved, discarding at least a portion of the video frame.

Clause 54: A method for receiving video data, the method comprising: receiving a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separated from the payload; extracting a video frame identifier of the frame of video data from the packet header; and processing the payload based on the video frame identifier.

Clause 55: A method according to clause 54, wherein the video frame identifier comprises a picture order count (POC) value.

Clause 56: A method according to clause 54, wherein the video frame identifier comprises a display frame identifier or a current frame identifier.

Clause 57: A method according to clause 54, wherein at least a portion of the frame comprises a slice of the frame, and wherein the video frame identifier comprises a slice identifier of the slice.

Clause 58: A method according to clause 54, wherein at least a portion of the frame comprises a tile of the frame, and wherein the video frame identifier comprises a tile identifier of the tile.

Clause 59: A method according to clause 54, the method also comprising: using the video frame identifier to determine one or more of a frame type of the frame, a priority of the frame, or relevance information of the frame.

Clause 60: A method according to clause 54, wherein at least a portion of the frames of video data comprises a protocol data unit (PDU) in a set of PDUs.

Clause 61: A method according to clause 54, the method also comprising: extracting one or more of the following from a packet header: a network abstraction layer (NAL) unit type of at least a portion of a frame, a time identifier (TID) of at least a portion of a frame, a layer identifier (LID) of at least a portion of a frame, data indicating whether at least a portion of a frame is predictively coded within a frame, or data indicating whether at least a portion is discardable.

Clause 62: A method according to clause 54, the method also comprising: processing a network abstraction layer (NAL) unit header of a frame, the NAL unit header comprising data indicating a priority value of the frame.

Clause 63: The method according to clause 54, the method also comprising: processing an access unit delimiter (AUD) of an access unit corresponding to a frame, the AUD comprising data representing a priority value of the access unit.

Clause 64: A method according to clause 54, the method also comprising: processing a picture header of a frame, the picture header comprising data representing a priority value of the frame.

Clause 65: The method of clause 54, further comprising: processing an open bitstream unit (OBU) of at least a portion of the frame, the OBU comprising data representing a priority value of at least a portion of the frame.

Clause 66: A method according to clause 65, wherein the OBU comprises one of a frame header OBU or a metadata OBU.

Clause 67: A method according to clause 54, the method also comprising: receiving data indicating a picture distance between a frame and a reference frame of the frame.

Clause 68: A method according to clause 67, wherein receiving data indicating a picture distance comprises: receiving a network abstraction layer (NAL) unit including the data or a supplemental enhancement information (SEI) message including the data.

Clause 69: A method according to clause 67, wherein receiving data indicating a picture distance between a frame and a reference frame of the frame comprises: receiving data indicating a picture distance between a frame and each active reference frame.

Clause 70: A method according to clause 54, the method also comprising: receiving information indicating whether a portion of a frame of video data can be decoded without other portions of the frame.

Clause 71: A method according to clause 70, wherein the portion of the frame comprises a protocol data unit (PDU), wherein the frame corresponds to a PDU set comprising the PDU, and wherein the information indicates whether the PDU can be decoded without other PDUs in the PDU set.

Clause 72: The method of clause 54, further comprising: receiving information indicating whether loop filtering is to be performed across one or more boundaries between a portion of the frame and one or more other portions of the frame.

Clause 73: A method according to clause 54, the method also comprising: receiving information indicating whether a portion of a frame is independently encoded without prediction based on other portions of the frame.

Clause 74: The method according to clause 54, the method also comprising: receiving data indicating that the frame is a discardable frame.

Clause 75: A method according to clause 74, wherein receiving data indicating that a frame is a discardable frame comprises: receiving the frame in a network abstraction layer (NAL) unit, an access unit delimiter, a picture header, a slice header, a supplemental enhancement information (SEI) message, or a frame header open bitstream unit (OBU).

Clause 76: A method according to clause 54, wherein processing the payload comprises: determining that the frame is a progressive decoder refresh (GDR) frame and determining that at least a portion of the frame is independently encoded; and in response to determining that at least a portion of the frame is independently encoded, providing at least a portion of the frame to a video decoder.

Clause 77: A method according to clause 54, wherein processing the payload comprises: determining that the frame is a progressive decoder refresh (GDR) frame and determining that at least a portion of the frame is encoded relative to a reference frame; determining that the GDR frame is a first frame in a sequence to be retrieved for a bit stream comprising video data such that a reference frame has not been retrieved; and in response to the reference frame not being retrieved, discarding at least a portion of the video frame.

Clause 78: A device for retrieving media data, the device comprising: a memory; and a processing system, the processing system comprising one or more processors implemented in a circuit system, the processing system being configured to: receive a packet comprising a packet header and a payload, the payload comprising at least a portion of a frame of video data, the packet header being separated from the payload; extract a video frame identifier of the frame of video data from the packet header; and process the payload based on the video frame identifier.

Clause 79: Apparatus according to clause 78, wherein the video frame identifier comprises at least one of a picture order count (POC) value, a display frame identifier, or a current frame identifier.

Clause 80: Apparatus according to clause 78, wherein at least a portion of the frame comprises a slice of the frame or a tile of the frame, and wherein the video frame identifier comprises an identifier of the slice of the frame or the tile of the frame.

Clause 81: An apparatus according to clause 78, wherein the processing system is configured to use the video frame identifier to determine one or more of a frame type of the frame, a priority of the frame, or relevance information of the frame.

Clause 82: Apparatus according to clause 78, wherein at least a portion of the frames of the video data comprises a protocol data unit (PDU) in a set of PDUs.

Clause 83: An apparatus according to clause 78, wherein the processing system is also configured to extract one or more of the following from the packet header: a network abstraction layer (NAL) unit type of at least a portion of the frame, a time identifier (TID) of at least a portion of the frame, a layer identifier (LID) of at least a portion of the frame, data indicating whether at least a portion of the frame is predictively coded within the frame, or data indicating whether at least a portion is discardable.

Clause 84: Apparatus according to clause 78, wherein the processing system is also configured to process a network abstraction layer (NAL) unit header of a frame, the NAL unit header comprising data indicating a priority value of the frame.

Clause 85: Apparatus according to clause 78, wherein the processing system is also configured to process an access unit delimiter (AUD) of an access unit corresponding to a frame, the AUD comprising data representing a priority value of the access unit.

Clause 86: Apparatus according to clause 78, wherein the processing system is also configured to process a picture header of a frame, the picture header comprising data representing a priority value of the frame.

Clause 87: Apparatus according to clause 78, wherein the processing system is also configured to process an open bitstream unit (OBU) of at least a portion of the frame, the OBU comprising data representing a priority value of at least a portion of the frame.

Clause 88: Apparatus according to clause 78, wherein the processing system is also configured to receive data indicating a picture distance between a frame and a reference frame of the frame, the data being included in a network abstraction layer (NAL) unit or a supplemental enhancement information (SEI) message.

Clause 89: Apparatus according to clause 78, wherein the processing system is also configured to receive information indicating whether portions of a frame of video data can be decoded without other portions of the frame.

Clause 90: Apparatus according to clause 78, wherein the processing system is also configured to receive information indicating whether loop filtering is to be performed across one or more boundaries between a portion of the frame and one or more other portions of the frame.

Clause 91: Apparatus according to clause 78, wherein, to process the payload, the processing system is configured to: determine that the frame is a progressive decoder refresh (GDR) frame and determine that at least a portion of the frame is independently encoded; and in response to determining that at least a portion of the frame is independently encoded, provide at least a portion of the frame to a video decoder.

Clause 92: An apparatus according to clause 78, wherein, to process the payload, the processing system is configured to: determine that a frame is a progressive decoder refresh (GDR) frame and determine that at least a portion of the frame is encoded relative to a reference frame; determine that the GDR frame is a first frame in a sequence to be retrieved for a bit stream comprising video data such that a reference frame has not yet been retrieved; and in response to the reference frame not having been retrieved, discard at least a portion of the video frame.

Clause 93: An apparatus for receiving video data, the apparatus comprising: means for receiving a packet comprising a packet header and a payload, the payload comprising at least a portion of a frame of video data, the packet header being separate from the payload; means for extracting a video frame identifier of the frame of video data from the packet header; and means for processing the payload based on the video frame identifier.

Clause 94: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receive a packet comprising a packet header and a payload, the payload comprising at least a portion of a frame of video data, the packet header being separate from the payload; extracting a video frame identifier of the frame of video data from the packet header; and processing the payload based on the video frame identifier.

In one or more examples, the functions described may be implemented using hardware, software, firmware, or any combination thereof. If implemented using software, the functions may be stored on or sent via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media that corresponds to tangible media (such as data storage media) or communication media that includes any media that facilitates, for example, transferring a computer program from one place to another according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) non-transitory, tangible computer-readable storage media or (2) communications media (such as signals or carrier waves). Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementing the techniques described herein. A computer program product may include computer-readable media.

By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Moreover, any connection is properly referred to as a computer-readable medium. For example, if coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave) are used to send instructions from a website, server, or other remote source, the coaxial cable, fiber optic cable, twisted pair cable, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other transient media, but rather refer to non-transient tangible storage media. As used herein, magnetic disks and optical discs include compact disks (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where magnetic disks typically copy data magnetically and optical discs use lasers to copy data optically. Combinations of the above should also be included in the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or individual logic circuit systems. Therefore, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding or included in a combined transcoder. Likewise, these techniques may be implemented entirely in one or more circuits or logic components.

The technology of the present case can be implemented in a variety of devices or apparatuses, including wireless phones, integrated circuits (ICs) or IC collections (such as chipsets). Various components, modules or units are described in the present case to emphasize the functional aspects of the device configured to perform the disclosed technology, but they do not necessarily need to be implemented by different hardware units. On the contrary, as mentioned above, various units can be combined in a transcoder hardware unit in combination with suitable software and/or firmware or provided by a collection of interactively operating hardware units (including one or more processors as mentioned above).

Various examples have been described. These and other examples are within the scope of the following claims.

10: System 20: Content preparation equipment 22: Audio source 24: Video source 26: Audio encoder 28: Video transcoder 30: Encapsulation unit 32: Output interface 40: Client equipment 42: Audio output port 44: Video output port 46: Audio decoder 48: Video decoder 50: Decapsulation unit 52: RTP receiving unit 54: Network interface 60: Server equipment 64: Media content 70: RTP sending unit 72: Network interface 74: Network 100: Application/service layer 102: User plane function (UPF) 104: Detection unit 106: Packet detection rules 110: Access Network (AN) 112: Quality of Service (QoS) 114: Radio Interface 120: Client Equipment 122: QoS Rules 124: Quality of Service (QoS) 126: Radio Interface 140A: GDR Frame 140B: GDR Frame 140C: GDR Frame 140D: GDR Frame 142A: Intra-frame Predicted Coding Area 142B: Intra-frame Predicted Coding Area 142C: Intra-frame Predicted Coding Area 142D: Intra-frame Predicted Coding Area 144A: Inter-frame Predicted Coding Area 144B: Inter-frame Predicted Coding Area 144C: Inter-frame Predicted Coding Area 146B: Inter-frame Predicted Area 146C: Inter-frame prediction area 146D: Inter-frame prediction area 150: Video file 152: File type (FTYP) box 154: Movie (MOOV) box 156: Movie header (MVHD) box 158: Track (TRAK) box 160: Movie extension (MVEX) box 162: Segment index (sidx) box 164: Movie fragment (MOOF) box 166: Movie fragment random access (MFRA) box 200: Block 202: Block 204: Block 206: Block 208: Block 210: Block 250: Block 252: Block 254: Block 256: Block 258: Block 260: Block

FIG. 1 is a block diagram illustrating an example system implementing techniques for streaming media data via a network data stream.

FIG. 2 is a conceptual diagram illustrating an example architecture for extended reality (XR) transport delivery.

FIG3 is a conceptual diagram illustrating a series of progressive decoder refresh (GDR) frames of video data.

FIG. 4 is a conceptual diagram illustrating an example set of identification information of a protocol data unit (PDU).

FIG5 is a block diagram illustrating elements of an example video file.

Figure 6 is a flow chart illustrating an example method of sending a packet including media data according to the technology of the present case.

Figure 7 is a flow chart illustrating an example method of receiving a packet including media data according to the technology of the present case.

Domestic storage information (please note in the order of storage institution, date, and number) None Foreign storage information (please note in the order of storage country, institution, date, and number) None

250:Block

252: Block

254: Block

256: Block

258: Block

260: Block

Claims (41)

A method for receiving video data, the method comprising the following steps: receiving a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separate from the payload; extracting a video frame identifier of the frame of video data from the packet header; and processing the payload according to the video frame identifier. The method of claim 1, wherein the video frame identifier comprises a picture order count (POC) value. The method of claim 1, wherein the video frame identifier comprises a display frame identifier or a current frame identifier. The method of claim 1, wherein the at least a portion of the frame includes all slices of the frame, and wherein the video frame identifier includes all slice identifiers of the slice. The method of claim 1, wherein the at least a portion of the frame comprises a tile of the frame, and wherein the video frame identifier comprises a tile identifier of the tile. According to the method of claim 1, the method also includes the following step: using the video frame identifier to determine one or more of a frame type of the frame, a priority of the frame, or relevance information of the frame. The method of claim 1, wherein the at least a portion of the frame of video data comprises a protocol data unit (PDU) from a set of PDUs. According to the method of claim 1, the method also includes the following steps: extracting one or more of the following items from the packet header: a network abstraction layer (NAL) unit type of at least a portion of the frame, a time identifier (TID) of at least a portion of the frame, a layer identifier (LID) of at least a portion of the frame, data indicating whether the at least a portion of the frame is predictively encoded within the frame, or data indicating whether the at least a portion is discardable. According to the method of claim 1, the method also includes the following step: processing a network abstraction layer (NAL) unit header of the frame, the NAL unit header including data indicating a priority value of the frame. According to the method of claim 1, the method also includes the following step: processing an access unit delimiter (AUD) of an access unit corresponding to the frame, the AUD including data representing a priority value of the access unit. According to the method of claim 1, the method also includes the following steps: processing a picture header of the frame, the picture header including data representing a priority value of the frame. According to the method of claim 1, the method also includes the following steps: processing an open bitstream unit (OBU) of at least a portion of the frame, the OBU including data representing a priority value of the at least a portion of the frame. The method of claim 12, wherein the OBU comprises one of a frame header OBU or a metadata OBU. According to the method of claim 1, the method also includes the following step: receiving data indicating a picture distance between the frame and a reference frame of the frame. The method of claim 14, wherein receiving the data indicating the distance of the picture comprises the following steps: receiving a network abstraction layer (NAL) unit including the data or a supplemental enhancement information (SEI) message including the data. The method of claim 14, wherein receiving the data indicating the picture distance between the frame and the reference frame of the frame comprises the following steps: receiving data indicating the picture distance between the frame and each active reference frame. According to the method of claim 1, the method also includes the following step: receiving information indicating whether the portion of the frame of video data can be decoded without other portions of the frame. The method of claim 17, wherein the portion of the frame includes a protocol data unit (PDU), wherein the frame corresponds to a PDU set that includes the PDU, and wherein the information indicates whether the PDU can be decoded without other PDUs in the PDU set. According to the method of claim 1, the method also includes the following step: receiving information indicating whether loop filtering is to be performed across one or more boundaries between the portion of the frame and one or more other portions of the frame. According to the method of claim 1, the method also includes the following step: receiving information indicating whether the part of the frame is independently encoded without prediction based on other parts of the frame. The method of claim 1, further comprising the step of receiving data indicating that the frame is a discardable frame. The method of claim 21, wherein receiving the data indicating that the frame is the discardable frame comprises the following steps: receiving the frame with a network abstraction layer (NAL) unit, an access unit delimiter, a picture header, a slice header, a supplemental enhancement information (SEI) message, or a frame header open bitstream unit (OBU). The method of claim 1, wherein processing the payload comprises the steps of: determining that the frame is a progressive decoder refresh (GDR) frame, and determining that the at least a portion of the frame is independently encoded; and in response to determining that the at least a portion of the frame is independently encoded, providing the at least a portion of the frame to a video decoder. The method of claim 1, wherein processing the payload comprises the steps of: determining that the frame is a progressive decoder refresh (GDR) frame, and determining that the at least a portion of the frame is encoded relative to a reference frame; determining that the GDR frame is a sequential first frame to be retrieved for a bit stream comprising the video data such that the reference frame has not been retrieved; and in response to the reference frame not being retrieved, discarding the at least a portion of the frame of video data. A device for retrieving media data, the device comprising: a memory; and a processing system, the processing system comprising one or more processors implemented in a circuit system, the processing system being configured to: receive a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separate from the payload; extract a video frame identifier of the frame of video data from the packet header; and process the payload according to the video frame identifier. The apparatus of claim 25, wherein the video frame identifier comprises at least one of a picture order count (POC) value, a display frame identifier, or a current frame identifier. The apparatus of claim 25, wherein the at least a portion of the frame comprises a slice of the frame or a tile of the frame, and wherein the video frame identifier comprises an identifier of the slice of the frame or the tile of the frame. The apparatus of claim 25, wherein the processing system is configured to use the video frame identifier to determine one or more of a frame type of the frame, a priority of the frame, or relevance information of the frame. The apparatus of claim 25, wherein the at least a portion of the frame of video data comprises a protocol data unit (PDU) from a set of PDUs. A device according to claim 25, wherein the processing system is also configured to extract one or more of the following items from the packet header: a network abstraction layer (NAL) unit type of at least a portion of the frame, a time identifier (TID) of at least a portion of the frame, a layer identifier (LID) of at least a portion of the frame, data indicating whether the at least a portion of the frame is predictively encoded within the frame, or data indicating whether the at least a portion is discardable. The apparatus of claim 25, wherein the processing system is also configured to process a network abstraction layer (NAL) unit header of the frame, the NAL unit header comprising data indicating a priority value of the frame. The apparatus of claim 25, wherein the processing system is also configured to process an access unit delimiter (AUD) of an access unit corresponding to the frame, the AUD comprising data representing a priority value of the access unit. The apparatus of claim 25, wherein the processing system is also configured to process a picture header of the frame, the picture header comprising data representing a priority value of the frame. The apparatus of claim 25, wherein the processing system is also configured to process an open bitstream unit (OBU) of the at least a portion of the frame, the OBU comprising data representing a priority value of the at least a portion of the frame. The apparatus of claim 25, wherein the processing system is also configured to receive data indicating a picture distance between the frame and a reference frame of the frame, the data being included in a network abstraction layer (NAL) unit or a supplemental enhancement information (SEI) message. The apparatus of claim 25, wherein the processing system is also configured to receive information indicating whether the portion of the frame of video data can be decoded without other portions of the frame. The apparatus of claim 25, wherein the processing system is also configured to receive information indicating whether loop filtering is to be performed across one or more boundaries between the portion of the frame and one or more other portions of the frame. The apparatus of claim 25, wherein to process the payload, the processing system is configured to: determine that the frame is a progressive decoder refresh (GDR) frame and determine that the at least a portion of the frame is independently encoded; and in response to determining that the at least a portion of the frame is independently encoded, provide the at least a portion of the frame to a video decoder. The apparatus of claim 25, wherein to process the payload, the processing system is configured to: determine that the frame is a progressive decoder refresh (GDR) frame and determine that the at least a portion of the frame is encoded relative to a reference frame; determine that the GDR frame is a sequential first frame retrieved for a bit stream comprising the video data such that the reference frame has not been retrieved; and in response to the reference frame not being retrieved, discard the at least a portion of the frame of video data. A device for receiving video data, the device comprising: means for receiving a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separate from the payload; means for extracting a video frame identifier of the frame of video data from the packet header; and means for processing the payload based on the video frame identifier. A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to: receive a packet including a packet header and a payload, the payload including at least a portion of a frame of video data, the packet header being separate from the payload; extracting a video frame identifier for the frame of video data from the packet header; and processing the payload based on the video frame identifier.

Images