BR112017017281B1 - SIGNAGE OF OPERATING POINTS FOR DRIVING HEVC EXTENSIONS - Google Patents

Abstract

SIGNAGE OF OPERATION POINTS FOR DRIVING HEVC EXTENSIONS. This is a first descriptor that describes an operating point. The second descriptor is a hierarchy descriptor or a hierarchy extension descriptor. The second descriptor has a hierarchy layer index value equal to a value of the second syntax element. A first value of a first syntax element in the first descriptor specifies that an elementary stream indicated by a second syntax element in the first descriptor, when not present in a list of elementary streams, is to be added to the list, and an elementary stream indicated by an index on the second descriptor, when not present in the list, must be added to the list. In response to determining that the first syntax element has a second value different from the first value, add the elementary stream indicated by the second syntax element, when not present in the list, to the list, but not the elementary stream indicated by the index in the second descriptor.

Description

[001] The present application claims the benefit of the provisional patent application in U.S. 62/115,089, filed on February 11, 2015, the contents of which are incorporated by reference in their entirety.

FIELD OF TECHNIQUE

[002] The present disclosure relates to video coding and, more particularly, to transporting encoded video data.

BACKGROUND

[003] Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, desktop computers, tablet type, e-book type readers, digital cameras, digital recording devices, digital media players, electronic gaming devices, electronic gaming consoles, cellular or satellite radio telephones, then called "smart telephones", teleconferencing device video, video streaming device and the like. Digital video devices implement video coding techniques such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Video Coding. Advanced Video Coding (AVC), the High Efficiency Video Coding (HEVC) standard, and extensions of such standards. Video devices can transmit, receive, encode, decode and/or store digital video information more efficiently by implementing such video coding techniques.

[004] Video coding techniques include spatial prediction (intra-image) and/or temporal prediction (inter-image) to reduce or remove inherent redundancy in video sequences. For block-based video encoding, a video slice (e.g., a video frame or a portion of a video frame) can be partitioned into video blocks that can also be called tree blocks, units of coding (CUs) and/or coding nodes. Video blocks in an intracoded (I) slice of an image are encoded using a spatial prediction with respect to reference samples in neighboring blocks in the same image. Video blocks in an intercoded slice (P or B) of an image can use spatial prediction with respect to reference samples in neighboring blocks in the same image or temporal prediction with respect to reference samples in other reference images. Images can be called frames, and reference images can be called reference frames.

[005] Spatial or temporal prediction results in a predicted block for a block to be encoded. Residual data represents pixel differences between the original block to be encoded and the prediction block. An intercoded block is coded according to a motion vector that points to a block of reference samples that forms the prediction block, with the residual data indicating the difference between the coded block and the prediction block. An intercoded block is coded according to an intercoding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to a transform domain, resulting in residual coefficients that can then be quantized. On quantized transform coefficients, initially arranged in a two-dimensional matrix, a scan can be performed to produce a one-dimensional transform coefficient vector, and entropy coding can be applied to achieve further compression.

SUMMARY

[006] In general, the present disclosure describes techniques that can improve the design of the HEVC operating point descriptor in the MPEG-2 (Moving Picture Expert Groups) transport stream (TS) for driving. Transporting encoded video data may also be called carrying encoded video data. The techniques of the present disclosure can be used to transport video data encoded to an extension of a video coding standard, e.g., an extension of the High Efficiency Video Coding (HEVC) standard. Such extensions may include multi-view extensions (e.g., MV-HEVC), scalable extensions (e.g., SHVC), and three-dimensional extensions (e.g., 3D-HEVC). However, it should be understood that the techniques of the present disclosure can be used with other transport streams and/or other video compression techniques.

[007] In one aspect, the present disclosure describes a method for processing video data, the method comprising: receiving a transport stream that includes a first descriptor, a second descriptor and a plurality of elementary streams, the first descriptor is a descriptor for an operating point of the video data, the second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor; decode a first syntax element and a second syntax element in the first descriptor, where: an elementary stream list is a list of elementary streams of the transport stream that are part of the operating point, where the second descriptor has a value of hierarchy layer index equal to a value of the second syntax element, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, shall be added to the list of elementary flows, with the elementary flow indicated by the second syntax element being one of a plurality of elementary flows, and an elementary flow indicated by a layer index embedded in the second descriptor, when not present in the list of elementary flows , shall be added to the list of elementary streams, wherein the elementary stream indicated by the layer index embedded in the second descriptor is one of a plurality of elementary streams, and in response to determining that the first syntax element has a different second value of the first value, add the elementary flow indicated by the second syntax element, when it is not present in the list of elementary flows, to the list of elementary flows, however, do not add the elementary flow indicated by the layer index incorporated in the second descriptor to the list of elementary flows.

[008] In another aspect, the present disclosure describes a method for processing video data, the method comprising: generating a first descriptor, the first descriptor being a descriptor of an operating point of the video data; generating a second descriptor, wherein the second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor; wherein generating the first descriptor comprises: including a first syntax element and a second syntax element in the first descriptor, wherein: a list of elementary streams is a list of elementary streams that are part of the operating point, the second descriptor has a hierarchy layer index value equal to a value of the second syntax element, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, shall be added to the list of elementary streams, and an elementary stream indicated by a layer index embedded in the second descriptor, is not present in the list of elementary streams, shall be added to the list of elementary streams, and a second value of the first element of syntax specifies that the elementary flow indicated by the second syntax element, when present in the list of elementary flows, should be added to the list of elementary flows, but not the elementary flow indicated by the layer index embedded in the second descriptor; and including the first descriptor and the second descriptor in a transport stream.

[009] In another aspect, the present disclosure describes a device for processing video data, the device comprising: an input interface configured to receive information, the information including a transport stream that includes a first descriptor, a second descriptor and a plurality of elementary streams, wherein the first descriptor is a descriptor for an operating point of the video data, the second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor; and one or more processors configured to: decode a first syntax element and a second syntax element in the first descriptor, wherein: an elementary stream list is a list of elementary streams of the transport stream that are part of the operating point, the second descriptor has a hierarchy layer index value equal to a value of the second syntax element, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary flows, shall be added to the list of elementary flows, the elementary flow indicated by the second syntax element is one of a plurality of elementary flows, and an elementary flow indicated by a layer index embedded in the second descriptor, when not present in the list of elementary streams, shall be added to the list of elementary streams, wherein the elementary stream indicated by the layer index embedded in the second descriptor is one of a plurality of elementary streams, and in response to the determination that the first syntax element has a second value different from the first value, add the elementary flow indicated by the second syntax element, when not present in the list of elementary flows, to the list of elementary flows, however, do not add the elementary flow indicated by the layer index embedded in the second descriptor to the list of elementary flows.

[0010] In another aspect, the present disclosure describes a device for processing video data, the device comprising: one or more processors configured to: generate a first descriptor, the first descriptor being a descriptor of an operating point of video data; generating a second descriptor, wherein the second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor; wherein the one or more processors are configured such that, as part of generating the first descriptor, the one or more processors: includes a first syntax element and a second syntax element in the first descriptor, wherein: a list of streams elementary streams is a list of elementary streams that are part of the operating point, the second descriptor has a hierarchy layer index value equal to a value of the second syntax element, a first value of the first syntax element specifies that: a stream element indicated by the second syntax element, when not present in the list of elementary flows, must be added to the list of elementary flows, and an elementary flow indicated by a layer index embedded in the second descriptor, when not present in the list of flows elementary streams, shall be added to the list of elementary streams, and a second value of the first syntax element specifies that the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, shall be added to the list of elementary streams , however, not the elementary flow indicated by the layer index embedded in the second descriptor; and includes the first descriptor and the second descriptor in a transport stream; and an output interface configured to output an encoded version of the video data.

[0011] In another aspect, the present disclosure describes a device for processing video data, the device comprising means for receiving a transport stream that includes a first descriptor, a second descriptor and a plurality of elementary streams, the first descriptor is a descriptor for an operating point of the video data, the second descriptor being one of: a hierarchy descriptor or a hierarchy extension descriptor; means for decoding a first syntax element and a second syntax element in the first descriptor, wherein: an elementary stream list is a list of elementary streams of the transport stream that are part of the operating point, the second descriptor has a value of hierarchy layer index equal to a value of the second syntax element, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, shall be added to the list of elementary flows, with the elementary flow indicated by the second syntax element being one of a plurality of elementary flows, and an elementary flow indicated by a layer index embedded in the second descriptor, when not present in the list of elementary flows , shall be added to the list of elementary streams, the elementary stream indicated by the layer index embedded in the second descriptor being one of a plurality of elementary streams, and means for adding, in response to determining that the first syntax element has a second value different from the first value, the elementary flow indicated by the second syntax element, when not present in the list of elementary flows, to the list of elementary flows, however, do not add the elementary flow indicated by the layer index incorporated in the second descriptor to the list of elementary flows.

[0012] In another aspect, the present disclosure describes a device for processing video data, the device comprising means for generating a first descriptor, the first descriptor being a descriptor of an operating point of the video data; means for generating a second descriptor, the second descriptor being one of: a hierarchy descriptor or a hierarchy extension descriptor; wherein the means for generating the first descriptor comprises: means for including a first syntax element and a second syntax element in the first descriptor, wherein: a list of elementary streams is a list of elementary streams that are part of the operating point , where the second descriptor has a hierarchy layer index value equal to a value of the second syntax element, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary flows, shall be added to the list of elementary flows, and an elementary flow indicated by a layer index embedded in the second descriptor, when not present in the list of elementary flows, d shall be added to the list of elementary flows, and a second value of the first syntax element specifies that the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, but not the elementary stream indicated by the index of layer embedded in the second descriptor; and means for including the first descriptor and the second descriptor in a transport stream.

[0013] In yet another aspect, the present disclosure describes a computer-readable data storage medium comprising instructions stored thereon that, when executed, cause a device for processing video data to perform the methods of the above paragraphs and described in the present revelation.

[0014] Details of one or more examples are presented in the attached drawings and in the description below. Other features, objects and advantages will be apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram illustrating an exemplary video encoding and decoding system that may utilize techniques for carrying video data encoded in accordance with extensions of a video coding standard.

[0016] Figure 2 is a block diagram illustrating an example of a video encoder that can implement techniques for transporting video data encoded in accordance with extensions of the video coding standard.

[0017] Figure 3 is a block diagram illustrating an example of a video decoder that can implement techniques for transporting video data encoded in accordance with extensions of the video coding standard.

[0018] Figure 4 is a flowchart illustrating an exemplary operation of a device that generates a transport flow, in accordance with a technique of the present disclosure.

[0019] Figure 5 is a flowchart illustrating an exemplary operation of a device that receives a transport flow, in accordance with a technique of the present disclosure.

[0020] Figure 6 is a flowchart illustrating an exemplary operation of a device for generating a transport flow in accordance with an exemplary technique of the present disclosure.

[0021] Figure 7 is a flowchart illustrating an exemplary operation of a device receiving a transport flow in accordance with an exemplary technique of the present disclosure.

DETAILED DESCRIPTION

[0022] In general, the present disclosure describes techniques related to Moving Picture Expert Group (MPEG)-2 Systems level data for media data. The specification of MPEG-2 Systems should generally address how two or more data streams are multiplexed together to form a single data stream. The present disclosure describes techniques related to MPEG-2 Systems data for multi-layer video data. For example, the present disclosure describes changes that can potentially improve the design of the High Efficiency Video Coding (HEVC) operating point descriptor in MPEG-2 Transport Streams (TSs) to drive HEVC EXTENSIONS. However, the techniques of the present disclosure are not necessarily limited to MPEG-2 TS or HEVC.

[0023] For ease of description only, the techniques of the present disclosure are described generally for conducting (e.g., transporting) video data encoded in accordance with an extension to a video coding standard (e.g., a extension for HEVC, also called ITU-T H.265). Such extensions may include multi-view, three-dimensional and/or scalable extensions. Thus, the techniques of the present disclosure can be applied to multiview HEVC (MV-HEVC), three-dimensional HEVC (3D-HEVC) and scalable HEVC (SHVC).

[0024] Multi-layer video data, e.g., multi-view video data and/or scalable multi-layer video data, may include designated operating points. Generally speaking, an operating point describes a subset of layers (e.g. views) of a complete set of multi-layer video data layers. The operating point can also identify output target layers, i.e. layers to which data should be output. In some cases, data from a layer may be included at an operating point only for use as a reference layer, and therefore such a layer is not considered an output target layer. A reference layer may be a layer used for reference by another layer for encoding and decoding.

[0025] HEVC operating points are generally signaled in a HEVC operating point descriptor specifying references to elementary flows, as described in more detail below. However, some HEVC operating point descriptor designs allow a list of elementary streams for a target operating point to include a layer or an elementary stream twice. Additionally, in some cases, not all layers that need to be decoded are flagged in some HEVC operating point descriptor designs due to the fact that information for such layers is available elsewhere. In some examples, because a particular layer is not signaled, certain syntax elements associated with the layer may not be available to the video decoder.

[0026] The techniques described in the present disclosure can address one or more of the problems described above. However, it is not a necessity for the techniques to address the problems described above. For example, the disclosure describes examples that can ensure that an elementary flow/layer is not included multiple times in the operating point's list of elementary flows. The disclosure also describes that examples of including information in the bitstream that a video decoder receives that includes syntax elements associated with a layer even if that layer is not explicitly flagged as being in the operating point elementary stream list. .

[0027] Figure 1 is a block diagram illustrating an exemplary video encoding and decoding system 10 that may utilize techniques for carrying video data encoded in accordance with extensions of a video coding standard. As shown in Figure 1, system 10 includes a source device 12 that provides encoded video data to be later decoded by a destination device 14. In particular, the source device 12 provides the video data to the destination device 14 via a computer-readable medium 16. The source device 12 and the destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook-type (i.e., laptop-type) computers, tablet computers, set-top boxes, telephone sets such as so-called "smart" telephones, tablet-type devices, televisions, cameras, display devices, digital media players, electronic game consoles, streaming device video or similar. In some examples, the source device 12 and the destination device 14 may be equipped for wireless communication. Thus, in some examples, the source device 12 and the destination device 14 are wireless communication devices.

[0028] In the example of Figure 1, the source device 12 includes a video source 18, a video encoder 20, a multiplexer 21, and an output interface 22. In some examples, the output interface 22 includes a modulator/demodulator (modem) and/or a transmitter. The video source 18 may include a source such as a video capture device, e.g., a video camera, a video file containing previously captured video, a video feed interface for receiving video from a video content provider, and/or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. However, the techniques described in the present disclosure may be applicable to video encoding in general and may be applied to wired and/or wireless applications.

[0029] Video encoder 20 can encode captured, pre-captured, or computer-generated video data. The source device 12 may transmit the encoded video data directly to the destination device 14 via the output interface 22 of the source device 12. The encoded video data may be stored (or alternatively) on a storage device for access. later by the destination device 14 or other devices, to decode and/or play back. Then, the encoded video information may be output by the output interface 22 onto a computer-readable medium 16.

[0030] The destination device 14 includes an input interface 28, a demultiplexer 29, a video decoder 30, and a display device 32. In some examples, the input interface 28 may include a receiver and/or a modem. The input interface 28 of destination device 14 receives the encoded video data over a link 16. The encoded video data communicated over the link 16, or provided to the storage device 33, may include a variety of generated syntax elements. by video encoder 20 for use by a video decoder, such as a video decoder 30, in decoding the video data. Such syntax elements may be included with encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0031] The destination device 14 may receive the encoded video data to be decoded via a computer readable medium 16. The computer readable 16 may comprise any type of medium or devices capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may comprise a communication medium for enabling a source device 12 to transmit the encoded video data directly to a destination device 14 in time. real. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a destination device 14. The communication medium may comprise any wireless and wired communication medium, such as as, a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the source device 12 to the destination device 14.

[0032] In some examples, encoded data may be output from output interface 22 to a storage device. For example, the source device 12 may include a storage device configured to store the encoded video data. In some examples, the encoded data may be accessed from the storage device via input interface 28. The storage device may include any of a variety of locally accessed or distributed data storage media, such as a hard drive, Blu-ray disks, -ray, DVDs, CD-ROMs, flash memories, volatile or non-volatile memory, or any other digital storage media suitable for storing encoded video data. In a further example, the storage device may correspond to a file server or other intermediate storage device that may store the encoded video generated by the source device 12. The destination device 14 may access the video data stores of the device. storage through streaming or through download transfer. The file server can be any type of server capable of storing encoded video data and transmitting that encoded video data to the destination device 14. Exemplary file servers include web servers (e.g., for a website). web), FTP servers, network attached storage devices (NAS), or local disk drives or other servers configured to serve files. The target device 14 can access the encoded video data over any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a DSL modem, a cable modem), or a combination thereof that is suitable for accessing encoded video data stored in a file server. The transmission of encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0033] The computer-readable medium 16 may include transient media, such as a wireless broadcast or wired broadcast transmission, or storage media (i.e., non-transient storage media), such as a hard drive, flash drive, disk compact disc, digital video disc, Blu-ray disc or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from the source device 12 and may provide the encoded video data to the destination device 14, for example, via wireless transmission. Similarly, a computing device of a media production facility, such as a disc stamping facility, may receive encoded video data from the source device 12 and produce a disk containing the encoded video data. Therefore, computer-readable medium 16 may be understood to include one or more computer-readable media in various forms, in various examples.

[0034] The techniques of the present disclosure are not necessarily limited to wireless applications or configurations. The techniques can be applied to video coding to support any of a variety of multimedia applications, such as television broadcasts over radio waves, cable television broadcasts, satellite television broadcasts, streaming video broadcasts over the Internet, such as dynamic adaptive streaming over HTTP (DASH), digital video that is encoded on a data storage medium, decoding digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support unidirectional or bidirectional video transmission to support applications such as video streaming, video replay, video broadcasting, and/or video telephony.

[0035] In the example of Figure 1, in accordance with the present disclosure, the multiplexer 21 of the source device 12 may be configured to apply techniques for transporting video data encoded in accordance with extensions of a video coding standard, whereas demultiplexer 29 may receive such data for processing and may forward the processed video data to another device or component, e.g., video decoder 30. In other examples, a source device and a destination device may include other components or provisions. For example, source device 12 may receive video data from an external video source 18, such as an external camera. Likewise, the target device 14 may interface with an external display device, rather than including an integrated display device.

[0036] The illustrated system 10 of Figure 1 is just an example. Techniques for transporting video data encoded in accordance with extensions of a video coding standard can be performed by any digital video encoding and/or decoding device. Although, in general, the techniques of the present disclosure are performed by a video encoding device, the techniques can also be performed by a video encoder/decoder, typically referred to as a "CODEC". Source device 12 and destination device 14 are just examples of such encoding devices, wherein source device 12 generates encoded video data for transmission to destination device 14. In some examples, devices 12, 14 may operate in a substantially symmetric manner such that each of the devices 12, 14 includes video encoding and decoding components. Accordingly, the system 10 may support unidirectional or bidirectional video transmission between video devices 12, 14, for example, for video streaming, video playback, video broadcasting or video telephony.

[0037] The input interface 28 of the target device 14 receives information from the computer-readable medium 16. The information from the computer-readable medium 16 may include syntax information defined by the video encoder 20, which is also used by the video decoder 30, which includes syntax elements that describe characteristics and/or processing of blocks and other coded units.

[0038] The display device 32 may be integrated with or external to the target device 14. In some examples, the target device 14 may include an integrated display device and may also be configured to interface with an external display device. In other examples, the target device 14 may be a display device. The display device 32 displays the decoded video data to a user and may comprise any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display , an organic light-emitting diode (OLED) display, or other type of display device.

[0039] Video encoder 20, multiplexer 21, demultiplexer 29, and video decoder 30 may each be deployed as any of a variety of suitable encoder or decoder circuit sets, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combinations thereof. When the techniques are partially implemented in software, a device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, each of which may be integrated as part of a combined encoder/decoder (CODEC) in a respective device.

[0040] The present disclosure may generally refer to video encoder 20 that "signals" or "transmits" certain information to another device, such as a video decoder 30. The term "signaling" or "transmission" may refer generally to the communication of syntax elements and/or other data used to decode compressed video data. Such communication may occur in real or near real time. Alternatively, such communication may occur over an extension of time, such as may occur when storing syntax elements on a computer-readable storage medium in a bit stream encoded at the time of encoding, which may be retrieved, in then by a decoding device at any time after being stored on that medium. Thus, although the video decoder 30 may be referred to as a "receiver" of certain information, the receipt of information does not necessarily occur in real time or near real time and may be retrieved from a medium at some point after storage.

[0041] Furthermore, although not specifically illustrated in Figure 1, some of the exemplary techniques described in the present disclosure can be implemented by some external means, such as a media recognition network element (MANE). The MANE may receive video data from the video encoder 20 and signal a bit stream that the video decoder 30 receives. In some examples, the MANE may receive and process a transport stream. Accordingly, sometimes when the disclosure describes signaling, such signaling may be performed by the video encoder 20 or by some external means, such as MANE.

[0042] Video encoder 20 and video decoder 30 may operate in accordance with a video coding standard. For example, video encoder 20 and video decoder 30 may operate in accordance with other proprietary or industry standards, such as the ITU-T H.264 standard, alternatively referred to as MPEG-4, Part 10, Advanced Video Coding. (AVC) or pattern extensions. However, the techniques of the present disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. The ITU-T H.264/MPEG-4 (AVC) standard was formulated by the Video Coding Expert Group (VCEG) of ITU-T together with the Moving Picture Expert Group (MPEG) of ISO/IEC as the product of a collective partnership known as the Joint Video Team (JVT). In some aspects, the techniques described in this disclosure can be applied to devices that generally comply with the H.264 standard. The H.264 standard is described in the ITU-T Recommendation H.264, Advanced Video Coding for generic audiovisual services, by the ITU-T Study Group and dated March 2005, which can be referred to in this document as the H standard .264 or H.264 specification or H.264/AVC standard or specification. The Joint Video Team (JVT) continues to work on extensions for H.264/MPEG-4 AVC. More generally, video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO /IEC MPEG-4 Visual and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions.

[0043] In another example, video encoder 20 and video decoder 30 may encode and decode video data using another video coding standard, such as the High Efficiency Video Coding (HEVC) standard, also called the H.265 Video Coding Standard. HEVC was finalized by the Joint Video Coding Collaboration Team (JCT-VC) of the ITU-T Video Coding Expert Group (VCEG) and the ISO/IEC Moving Image Expert Group (MPEG). The most recent initial specification of HEVC is referred to as HEVC WD from now on.

[0044] Furthermore, the multiview extension to HEVC, namely MV-HEVC, is also being developed by JCT-3 V. A scalable extension to HEVC, called SHVC, is also being developed by JCT-VC. The latest document including the specification for HEVC, HEVC range extension, SHVC and MV-HEVC is available at http://phenix.int-evry.fr/jct/doc_end_user/documents/ 18_Sapporo/wgl l/JCTVC- R1013-v6.zip.

[0045] In HEVC WD10 and other video coding standards, a video sequence typically includes a series of images. Images can also be called "frames". In order to generate a coded representation of an image, the video encoder 20 may generate a set of coding tree units (CTUs). Each of the CTUs can be a encoding tree block of luma samples, two corresponding encoding tree blocks of chroma samples, and syntax structures used to encode the samples from the encoding tree blocks. A coding tree block can be an NxN block of samples. A CTU may also be called a "tree block" or a "largest coding unit" (LCU).

[0046] In order to generate a coded CTU, the video encoder 20 may divide the coding tree blocks of a CTU into coding blocks, hence the name "coding tree units". A CU may be a luma sample encoding block and two corresponding chroma sample encoding blocks of an image that has a luma sample matrix, a Cb sample matrix, and a Cr sample matrix, and syntax structures used to encode the samples of the coding blocks. Video encoder 20 may partition a coding block of a CU into one or more prediction blocks. A prediction block can be a rectangular (i.e., square or non-square) block of samples to which the same prediction is applied. A prediction unit (PU) of a CU can be a luma sample prediction block, two corresponding chroma sample prediction blocks of an image, and syntax structures used to predict the sample prediction block. The video encoder 20 may generate predictive luma, Cb, and Cr blocks for luma, Cb, and Cr prediction blocks of each PU of the CU. The video encoder 20 may use intraprediction or interprediction to generate the prediction blocks for a PU.

[0047] After video encoder 20 generates prediction blocks for one or more PUs of a CU, video encoder 20 may generate residual blocks for the CU. Each sample in a residual block of the CU may indicate a difference between a sample in a prediction block of a PU of the CU and a corresponding sample in a coding block of the CU. Furthermore, the video encoder 20 may decompose the residual blocks of a CU into one or more transform blocks. A transform unit (TU) of a CU may be a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax structures used to transform the transform block samples. The video encoder 20 may apply one or more transforms to a transform block to generate a coefficient block for a TU. After generating a coefficient block, the video encoder 20 may quantize the coefficient block. After the video encoder 20 quantizes a coefficient block, the video encoder 20 may entropy encode the syntax elements indicating the quantized transform coefficients. For example, the video encoder 20 may perform Context-Adaptive Binary Arithmetic Coding (CAB AC) on the syntax elements indicating the quantized transform coefficients. The video encoder 20 may output entropy-encoded syntax elements in a bit stream. The bitstream may also include syntax elements that are not entropy encoded.

[0048] The bit stream may include a sequence of bits that forms a representation of encoded images and associated data. The bit stream may comprise a sequence of network abstraction layer (NAL) units. Each of the NAL units includes a NAL unit header and encapsulates a raw byte sequence (RBSP) payload. An RBSP can be a syntax structure that contains an integer number of bytes that is encapsulated within an NAL unit. In some examples, an RBSP includes zero bits.

[0049] Different types of NAL units can encapsulate different types of RBSPs. For example, a first type of NAL unit may encapsulate an RBSP for a picture parameter set (PPS), a second type of NAL unit may encapsulate an RBSP for an encoded slice, a third type of NAL unit may encapsulate an RBSP for Supplementary Enhancement Information (SEI), and so on. The NAL units that encapsulate the RBSPs for video coding data (as opposed to the RBSPs for parameter sets and SEI messages) can be called video coding layer (VCL) NAL units.

[0050] Video decoder 30 may receive a bitstream generated by video encoder 20. Additionally, video decoder 30 may obtain syntax elements from the bitstream. For example, video decoder 30 may parse the bitstream to decode syntax elements of the bitstream. The video decoder 30 may reconstruct images of the video data based, at least partially, on the syntax elements obtained (e.g., decoded) from the bit stream. The process for reconstructing the video data may be generally reciprocal to the process performed by the video encoder 20. The video decoder 30 may reconstruct the coding blocks of the current CU by adding the samples from the prediction blocks to the PUs of the current CU to the corresponding samples of the transform blocks of the current CU TUs. By rebuilding the coding blocks for each CU of an image, the video decoder 30 can reconstruct the image.

[0051] In multi-view coding, there may be multiple views of the same scene from different points of view. In the context of multiview coding, the term "access unit" can be used to refer to the set of images that correspond to the same instant in time. In this way, video data can be conceptualized as a series of access units that occur over time. A "view component" can be an encoded representation of a view in a single access unit. In the present disclosure, a "view" may refer to a sequence of view components associated with the same view identifier. In some examples, a view component may be a texture view component (i.e., a texture image) or a depth view component (i.e., a depth image).

[0052] In MV-HEVC and SHVC, a video encoder can generate a bit stream comprising a series of NAL units. Different bitstream NAL units can be associated with different bitstream layers. A layer can be defined as a set of VLC NAL units and associated non-VCL NAL units that have the same layer identifier. A layer can be equivalent to a view in video coding in multiview code. In multi-layer video coding, a layer can contain all view components of the same layer with different time instants. Each view component can be an encoded image of the video scene that belongs to a specific view in a specific time period. In some examples of encoding video into multi-view or three-dimensional code, a layer may contain both all depth images encoded from a specific view and texture images encoded from a specific view. In other 3D video encoding examples, a layer may contain both the texture view components and the depth view components of a specific view. Similarly, in the context of scalable video coding, a layer typically corresponds to encoded images that have different video characteristics than images encoded in other layers. Such video characteristics typically include spatial resolution and quality level (e.g., Signal-to-Noise Ratio). In HEVC and its extensions, time scalability can be achieved within a layer by defining a group of images with a particular time level as a sublayer.

[0053] For each respective layer of the bit stream, data in a lower layer can be decoded without reference to data in any higher layer. In scalable video coding, for example, data in a base layer can be decoded without reference to data in an enhancement layer. In general, NAL units can only encapsulate data from a single layer. In this way, NAL units that encapsulate data from the highest remaining layer of the bitstream can be removed from the bitstream without affecting the ability to decode data in the remaining layers of the bitstream. In multiview coding, upper layers can include additional view components. In SHVC, the upper layers may include signal-to-noise ratio (SNR) enhancement data, spatial enhancement data, and/or temporal enhancement data. In MV-HEVC and SHVC, a layer may be called a "base layer" if a video decoder can decode images at the layer without reference to data from any other layer. The base layer may conform to the HEVC base specification (e.g. Rec. ITU-T H.265 | ISO/IEC 23008-2).

[0054] In scalable video coding, layers other than the base layer may be referred to as "enhancement layers" and may provide information that enhances the visual quality of video data decoded from the bitstream. Scalable video coding can improve spatial resolution, signal-to-noise ratio (i.e., quality), or temporal rate. In scalable video coding (e.g. SHVC), a "layer representation" can be an encoded representation of a spatial layer in a single access unit. For the sake of ease of explanation, the present disclosure may refer to view components and/or layer representations as "view components/layer representations" or simply "images".

[0055] Multiview coding supports prediction between views. Between-view prediction is similar to the interprediction used in HEVC and can use the same syntax elements. However, when a video encoder performs view-to-view prediction on a current video unit (such as a PU), the video encoder 20 may use, as a reference image, an image that is on the same access unit as the current video unit, however, in a different view. In contrast, conventional interprediction only uses images in different access units as reference images.

[0056] In multi-view coding, a view may be called a "base view" if a video decoder (e.g., video decoder 30) can decode images in the view without reference to images in any other view. When encoding an image into one of the non-base views, a video encoder (such as video encoder 20 or video decoder 30) may add an image to a list of reference images if the image is in a different view, however, within the same instant in time (i.e., access unit) as the image that the video encoder is currently encoding. Like other interprediction reference images, the video encoder can insert a prediction reference image between views at any position in a list of reference images.

[0057] For example, NAL units may include headers (i.e., NAL unit headers) and payloads (e.g., RBSPs). NAL unit headers can include nuh_reserved_zero_6bits syntax elements, which can also be called nuh_layer_id syntax elements. NAL units that have nuh_layer_id syntax elements that specify that different values belong to different "layers" of a bitstream. Therefore, in multiview, MV-HEVC, SVC, or SHVC coding, the NAL unit nuh_layer_id syntax element specifies a layer identifier (that is, a layer ID) of the NAL unit. The nuh_layer_id syntax element of an NAL unit is equal to 0 if the NAL unit refers to a base layer in multiview, MV-HEVC, or SHVC encoding. Data in one base layer of a bitstream can be decoded without reference to data in any other layer of the bitstream. If the NAL unit does not refer to a base layer in multiview, MV-HEVC or SHVC encoding, the nuh_layer_id syntax element can be a non-zero value. In multiview coding, different layers of a bitstream can correspond to different views. In SVC or SHVC, layers other than the base layer may be called "enhancement layers" and may provide information that enhances the visual quality of video data decoded from the bitstream.

[0058] Additionally, some images within a layer may be decoded without reference to other images within the same layer. In this way, NAL units that encapsulate data from certain images in a layer can be removed from the bitstream without affecting the ability to decode other images in the layer. Removing NAL units that encapsulate data from such images may reduce the frame rate of the bitstream. A subset of images within a layer that can be decoded without reference to other images within the layer may be referred to herein as a "sublayer" or a "temporal sublayer." Therefore, one type of scalable dimension is the temporal dimension.

[0059] For example, in temporal scalability, a video dataset may support various frame rates or playback rates, for example, 15 frames per second (FPS), 30 FPS, 60 FPS, and 120 FPS. A given temporal level may include all photos at that level or lower levels. For example, continuing the previous example, a temporal level of 0 may correspond to 15 FPS, a temporal level of 1 may include images of temporal level 0 as well as images at temporal level 1 to support 30 FPS, a temporal level of 2 may include images from temporal levels 0 and 1 as well as images at temporal level 2 to support 60 FPS and so on.

[0060] A temporal_identifier, or TemporalID, can be flagged as representing the temporal level to which a particular image belongs. For example, NAL units can include temporal_id syntax elements. The temporal_id syntax element of an NAL unit specifies a temporal_identifier of the NAL unit. The temporal_identifier of an NAL unit identifies a temporal subshell with which the NAL unit is associated. In this way, each temporal sublayer of a bit stream can be associated with a different temporal_identifier. If the temporal_identifier of a first NAL unit is smaller than the temporal_identifier of a second NAL unit, the data encapsulated by the first NAL unit can be decoded without reference to the data encapsulated by the second NAL unit.

[0061] A bit stream can be associated with a plurality of operating points. In some examples, each operating point of a bitstream may be associated with a set of layer identifiers (that is, a set of nuh_reserved_zero_6bits values or nuh_layer_id values) and a temporal_identifier. The set of layer identifiers can be denoted as OpLayerldSet and the temporal_identifier can be denoted as TemporalID. If a layer identifier of the NAL unit is at an operation layer identifier set point and the temporal_identifier of the NAL unit is less than or equal to the temporal_identifier of the operation point, the NAL unit is associated with the operation point. Thus, an operating point can be a bitstream created from another bitstream by the subbitstream extraction process operation with that other bitstream, a higher target TemporalID, and a target list. of layer identifiers as input to the bit substream extraction process. The operating point may include each NAL unit that is associated with the operating point. In some examples, the operating point does not include VCL NAL units that are not associated with the operating point.

[0062] As indicated above, an operation point describes a subset of layers (e.g., views) of a complete set of multi-layer video data layers. The operating point can also identify output target layers, i.e. layers to which data (e.g. decoded images) should be output. In some instances, data from a layer may be included in an operating point only for use as a reference layer (for example, only for prediction between layers or between views of a layer or a view that is to be displayed, where reference layer or view is not displayed) and therefore such a layer is not considered an output target layer.

[0063] HEVC and other video coding standards specify profiles, ranks, and levels. Profiles, ranks, and levels specify constraints on bitstreams and, therefore, limits on the capabilities required to decode the bitstreams. Profiles, ranks, and levels can also be used to indicate points of interoperability between individual decoder deployments. Each profile specifies a subset of features and algorithmic tools present in a video coding standard. Thus, a "profile" is a subset of an entire bitstream syntax that is specified by an applicable video coding standard. Video encoders are not required to use all features supported in a profile. Each level of a row can specify a set of limits on the values that syntax elements and variables can have. Thus, a "level" corresponds to decoder resource consumption limitations, such as, for example, memory and decoder consumption, which are related to image resolution, bit rates and block processing rate. The same set of row and level definitions can be used with all profiles, however, individual deployments can support a different row and within a row a different level for each supported profile. For any given profile, a row level can generally correspond to a decoder's ability to decode payload and memory. Video decoder capabilities can be specified in terms of the ability to decode video streams in accordance with the constraints of particular profiles, ranks, and levels. For each profile, the supported row and level for the profile can be expressed as well. Some video decoders may not be able to decode particular profiles, rows, or levels.

[0064] The present disclosure describes techniques that can improve the HEVC operating point descriptor in the MPEG-2 transport stream (TS) for driving HEVC extension bitstreams. For example, in accordance with a technique of the present disclosure, the multiplexer 21 and/or the demultiplexer 29 may be configured to carry video data (i.e., send or receive video data) that is encoded in accordance with a coding standard. video coding standard, such as HEVC, an extension of a video coding standard (e.g., extensions of the HEVC standard, such as SHVC or MV-HEVC), or other video coding standards thus far developed. In general, the multiplexer 21 can encapsulate encoded video data to form a data stream, e.g., substantially in accordance with MPEG-e Systems and the techniques of the present disclosure, while the demultiplexer 29 can receive and decapsulate data. encapsulated, for example, video data encoded according to an extension of a video coding standard.

[0065] The most recent MPEG-2 TS specification is the ITU-T Recommendation H.222.0, June 2012 version, in which support for the AVC and AVC extensions is provided. An amendment from MPEG-2 TS to HEVC has also been developed. The most recent document, "Text of ISO/IEC 13818-1:2013 / Final Draft Amendment 3 - Transport of HEVC video over MPEG-2 Systems", is available in MPEG document wl3656, July 2013. Recently, an MPEG amendment - 2 TS for conducting layered HEVC, called MPEG-2 TS for L-HEVC, has been started. The most recent document is "Text of ISO/IEC 13818-1:2013 / DAM 3 - Carriage of Layered HEVC", hereinafter "DAM3". The term "layered HEVC" refers to extensions of the HEVC standard that use multiple layers, such as SHVC, MV-HEVC, and 3D-HEVC.

[0066] The MPEG-2 systems specification describes how compressed multimedia data streams (Video and Audio) can be multiplexed together with other data to form a single data stream suitable for digital transmission or storage. The MPEG-2 Systems specification describes an elementary stream, which is a single component digitally converted to code (possibly compressed in MPEG) of a program (also sometimes called programmed writing). For example, the encoded video and audio part of the program may be an elementary stream. An elementary stream is first converted into a packaged elementary stream (PES) before being multiplexed into a program stream or a transport stream. Within the same schedule, a stream_id syntax element is used to distinguish between PES packets that belong to an elementary stream. Each PES package may contain one or more NAL units, or one NAL unit may be divided among multiple PES packages.

[0067] In the MPEG-2 Systems specification, program streams and transport streams are two alternative multiplexings that target different applications. Program streams are influenced for the storage and display of a single schedule from a digital storage service, and a program stream is intended for use in an error-free environment due to the fact that it is susceptible to error.

[0068] A program flow includes the elementary flows that belong to it and generally contains packages with variable length packages. In a program flow, PES packages that are derived from elementary contribution flows are organized into "packages". A packet includes a packet reader, an optional system header, and any number of PES packets taken from any of the elementary contribution streams in any order. The system header contains a summary of the characteristics of the program flow, such as: maximum data rate; the number of contributing elementary video and audio streams; and additional weather information. The video decoder 30 may use information contained in a system header to determine whether or not the video decoder 30 is capable of decoding the program stream.

[0069] Transport streams are intended for simultaneous delivery of multiple programs to potentially error-prone channels. A transport stream is a multiplex designed for multi-program applications, such as streaming, so that a single transport stream can accommodate many independent programs. A transport stream includes a succession of transport packets, and each of the transport packets is 188 bytes long. The use of short fixed-length packets means that the transport stream is not as susceptible to errors as the program stream. Additionally, each 188 byte long transport packet is easily provided with additional error protection by processing it through a standard error protection process such as Reed-Solomon encoding. The improved error resilience of the transport stream means it has a better chance of surviving error channels encountered in a broadcast environment, for example. It can be seen that the transport stream is clearly the better of the two multiplexes with its increased error resilience and ability to drive many simultaneous programs. However, the transport stream is a more sophisticated multiplex than the program stream and is consequently more difficult to create and demultiplex.

[0070] The first byte of a transport packet is a synchronization byte which, in some occurrences, is 0x47. A single transport stream can post many different programs, each of which comprises many packaged elementary streams. A Packet Identifier (PID) field is used to distinguish transport packets that contain data from one elementary stream from those that carry data from other elementary streams. In some instances, the PID is 13 bits. It may be the responsibility of the multiplexer 21 to ensure that each elementary stream receives a unique PID value.

[0071] Although it is clear based on a PID value which elementary stream a transport packet belongs to, the video decoder 30 may need to know which elementary streams belong to which program. Consequently, a transport stream comprises program specific information (PSI) to explicitly specify the relationships between programs and component elementary streams. In other words, the transport stream may include transport packets that contain PSI.

[0072] The PSI may include a program association table (PAT). The program accession table includes a complete list of all programs available in a transport stream. In some examples, the PAT always has a PID value of 0. Each program is striped next to the PID value of the transport packets that contain the program's program map table.

[0073] Additionally, the PSI may include one or more Program Map Tables (PMTs). Each program ported into a transport stream has an associated PMT. The PAT can specify the value of PID transport packets that contain the PMT for a program. The PMT for a program provides details about the program and the elementary flows that comprise the program. For example, the PMT for a program with program number 3 might specify that the program contains transport packets with PID values 33, 57, and 60. In this example, transport packets with PID values of 33 might include an elementary stream that contains encoded video data, transport packets with PID values of 57 can include English audio data, and transport packets with PID values of 60 can be Chinese audio data. A PMT may include details in relation to more than one program.

[0074] The basic PMT for a program may include some of the many descriptors within the MPEG-2 systems specification. Such descriptors convey additional information about a program or its elementary component flows. Descriptors may include video encoding parameter, audio encoding parameter, language identification, full screen information, condition access details, copyright information, and so on. A broadcaster or other user can define additional private descriptors if required.

[0075] The descriptors are separate from the encoded video data. Thus, a device, such as a media-aware network element (MANE) or video decoder, may have the ability to use a descriptor to perform various functions on transport streams and program streams without decoding or otherwise ,analyze encoded video data. For example, if video data is encoded using HEVC, the device does not need to be configured to decode HEVC-encoded video data in order to use the descriptor to perform particular functions in transport or program streams. For example, the device may have the ability to use the descriptors as part of a process to determine whether to forward particular program elements to a destination device or decode particular elementary streams.

[0076] The descriptors for a program that has video-related component elementary streams may include one or more hierarchy descriptors. A hierarchy and design descriptor to signal the hierarchy of bitstream substreams of different elementary streams. The hierarchy descriptor provides information that identifies program elements that contain hierarchically encoded video, audio, and private stream components. Private streams can include metadata, such as a stream of program-specific information. Generally speaking, a program element is one of the elementary data or streams included in a program (that is, a program component elementary stream). In MPEG-2 transport streams, program elements are typically packaged. In MPEG-2 program streams, program elements are not packaged.

[0077] In some instances, each respective temporal sublayer of each respective layer of a program may correspond to a different program component (e.g., elementary flow) of the program. In other instances, each respective layer of a program corresponds to a different elementary flow. Furthermore, in some instances, two or more elementary streams may correspond to different temporal sublayers of the same layer of a program whereas a single elementary stream may correspond to all temporal sublayers of another layer of the same program. As indicated above, descriptors may include hierarchy descriptors. Each respective hierarchy descriptor provides information regarding a corresponding program component (i.e., elementary flow). For example, a hierarchy descriptor corresponding to a particular elementary stream may include a syntax element that specifies another elementary stream necessary to decode video data from the particular elementary stream. In an example, a hierarchy descriptor corresponding to an elementary stream for a particular temporal sublayer may include a syntax element that specifies an embedded temporal sublayer necessary to decode the particular temporal sublayer.

[0078] The embedded temporal sublayer may be "embedded" in the sense that decoding the embedded temporal sublayer is required for successful decoding of the particular temporal sublayer. Additionally, the hierarchy descriptor may include syntax elements that specify whether a temporal sublayer corresponding to the hierarchy descriptor provides temporal scalability (e.g., increases the frame rate) relative to the embedded temporal sublayer provides spatial scalability (e.g., increases the frame rate). (e.g., increases image resolution) relative to the embedded temporal sublayer, provides quality scalability (e.g., enhances quality or signal-to-noise fidelity) relative to the embedded temporal sublayer, and so on.

[0079] Table 2-49, below, specifies the syntax of a hierarchy descriptor as defined and DAM3. TABLE 2-49 - HIERARCHY DESCRIPTOR

[0080] In Table 2-49, the hierarchy_layer_index is a 6-bit field that defines a unique index of the associated program element in a layer hierarchy coding table. The indices must be unique within a single program definition. For video substreams of HEVC video streams conforming to one or more profiles defined in Annex F of Rec. ITU-T H.265 | ISO/IEC 23008-2, this is the program element index, which is assigned in such a way that the bitstream order is corrected if the associated dependency layers of the video substreams of the same HEVC access unit are reassembled in ascending order of the hierarchy_layer_index. In other words, the hierarchy_layer_index of a hierarchy descriptor identifies the program element (i.e., elementary stream) that corresponds to the hierarchy descriptor.

[0081] Additionally, in Table 2-49, the hierarchy_embedded_layer_index is a 6-bit field that defines the hierarchy_layer_index of the program element that needs to be associated and present in the decoding order before decoding the elementary stream associated with this hierarchy_descriptor . The hierarchy_embedded_layer_index is undefined if the hierarchy_type value is 15. In other words, the hierarchy_embedded_layer_index of a hierarchy descriptor identifies an elementary stream that the elementary stream corresponding to the hierarchy descriptor depends on.

[0082] In addition to the one or more hierarchy descriptors, the descriptors signaled in an MPEG-2 transport or program stream may include one or more hierarchy extension descriptors. Each respective hierarchy extension descriptor in the MPEG-2 transport or program stream may correspond to a respective elementary stream. Each hierarchy extension descriptor can provide additional information regarding a corresponding elementary flow. As indicated above, an elementary flow may correspond to one or more temporal sublayers or may correspond to a layer as a whole. In other words, an elementary stream may include encoded video data from one or more temporal sublayers or may correspond to a layer as a whole. Therefore, a hierarchy extension descriptor may correspond to an elementary flow corresponding to one or more temporal sublayers or may correspond to a layer as a whole.

[0083] When a hierarchy extension descriptor is present, the hierarchy extension descriptor is used to specify the elementary flow dependencies corresponding to the elementary flow. For example, a hierarchy extension descriptor corresponding to a particular elementary stream may indicate multiple elementary streams whose decoding is required in order to successfully decode the elementary stream corresponding to the hierarchy extension descriptor. In contrast, a hierarchy descriptor has the capacity only to indicate an elementary stream required for successful decoding of an elementary stream corresponding to the hierarchy descriptor. The 2-103deciens table, below, indicates a syntax for a hierarchy extension descriptor as specified in DAM3. TABLE 2-103DECIENS - HEVC HIERARCHY EXTENSION DESCRIPTOR

[0084] In Table 2-103deciens, hierarchy_layer_index is a 6-bit field that defines a unique index of the associated program element in a layer hierarchies encoding table. Indexes must be unique within a single program definition. For video bit substreams of HEVC video streams conforming to one or more profiles defined in Annex G or H of ITU-T Rec. H.265 | ISO/IEC 23008-2, this is the program element index, which is assigned so that the bitstream order is correct if the associated dependency layers of video bitstreams from the same access unit are reassembled in ascending order of hierarchy_layer_index. In this way, the hierarchy_layer_index of a hierarchy extension descriptor identifies the elementary flow corresponding to the hierarchy extension descriptor.

[0085] The nuh_layer_id is a 6-bit field that specifies the nuh_layer_id of the NAL units in the elementary stream associated with this HEVC_hierarchy_extension_descriptor().

[0086] The temporal_id is a 3-bit field that specifies the largest TemporalID of the NAL units in the elementary stream associated with this HEVC_hierarchy_extension_descriptor().

[0087] The num_embedded_layers is a 6-bit field that specifies the number of directly dependent program elements that need to be accessed and present in the decoding order before encoding the elementary stream associated with this HEVC_hierarchy_extension_descriptor().

[0088] The hierarchy_ext_embedded_layer_index is a 6-bit field that defines the hierarchy_layer_index of the program element that needs to be accessed and present in the decoding order before decoding the elementary stream associated with this hierarchy_extension_descriptor. This field is undefined if the hierarchy_type value is 15.

[0089] In addition to the hierarchy descriptor and extension hierarchy descriptors, a transport stream may include a HEVC operating point descriptor. HEVC operating points are signaled in a HEVC operating point descriptor by specifying references to the elementary streams corresponding to the layers at the operating points. As described in DAM3, the HEVC operating point descriptor provides a method for indicating a profile and level for one or more HEVC operating points. The target device 14 may use the operating point descriptors included in a bitstream to select one of the operating points to be decoded and thereby presented (e.g., displayed) to a user. Instead of passing the data for all views or layers to the video decoder 30 upon reception, the destination device 14 may send only the views from a selected operating point to the video decoder 30. For example, the destination 14 may discard data for views that will not be decoded. Additionally or alternatively, an intermediate network device (e.g., a Media-Aware Network Element (MANE)) may discard data for views or layers that do not match a requested operating point (e.g., to better utilize image width). band). The destination device 14 may select an operating point based on the highest quality operating point supported among the operating points for a bit stream and/or based on the available amount of network bandwidth.

[0090] Table 2-103noniens below indicates the syntax of the HEVC operating point descriptor, as defined in D AM3. TABLE 2-103NONIENS - HEVC OPERATING POINT DESCRIPTOR

[0091] There may be certain issues in the design in DAM3 for the HEVC operating point descriptor (i.e., the syntax structure of the HEVC operating point descriptor). The following paragraphs describe two of the issues in the current design of the HEVC operating point descriptor defined in DAM3.

[0092] First, as shown in Table 2-103noniens, the HEVC operating point descriptor includes prepend_dependencies[i][j] syntax elements. However, in DAM3, the semantics of prepend_dependencies[i][j] equal to 0 are missing. Furthermore, in the design of DAM3, it is possible that an elementary flow (e.g., an elementary flow corresponding to a layer) is included twice in the list of elementary flows for the target point of operation. The problem of more than one inclusion can occur in any of the following situations: 1. When a layer is explicitly signaled as part of an operating point by signaling the ES_reference[i][j] value of the same at the same time is also a reference layer of another layer that has the value of prepend_dependencies[i][j] equal to 1. 2. When a layer is a reference layer for two other layers at the same operating point that both have prepend_dependencies[ i][j] equals 1.

[0093] A second issue in the design in DAM3 for the HEVC operating point descriptor may be that for an operating point, not all layers whose decoding is required by the video decoder 30 for the HEVC operating point need be explicitly flagged due to the fact that some layers (elementary flows) may depend on others, and such information is present elsewhere (for example, in the hierarchy descriptor or hierarchy extension descriptor). When the value of the prepend_dependencies[i][j] syntax element is equal to 1, other layers on which the jth layer depends can be derived and therefore not explicitly signaled to the ith HEVC operating point. However, at the HEVC operating point, a derived layer is not present in the explicit layer cycle (as indicated by the text between the “<EMPHASIS>“ and </EMPHASIS>“ tags in Table 2-103noniens above). Therefore, there is currently no way to know the values of the following syntax elements for a derived layer: output_layer_lag[i][j] and ptl_ref_idx[i][j], although the elementary flow of the derived layer can be identified by the hierarchy descriptor and/or by the hierarchy extension descriptor. In other words, when the signaling layers of an operating point with references to elementary flows and multiple layers are not explicitly signaled, but rather derived (when the prepend_dependencies[i][j] syntax element value is 1), the Information, such as whether a layer is an output layer, and the profile, row, and level for those derived layers, is missing for those derived layers.

[0094] An output operating point in layered HEVC is defined based on a set of output layers. However, knowing whether a layer is an output layer may be required for defining compliance in layered HEVC, and each required layer (i.e., the layer that is both an output layer and a layer that is indicated directly or indirectly by an output layer within an operating point or both) must be associated with a set of profile, row and level (PTL) information in accordance with the MV-HEVC/SHVC specification. Therefore, it may be necessary to know whether or not a layer is an output target layer. For example, you may need to know the value of the output_layer_flag[i][j] syntax element. However, the output_layer_flag[i][j] syntax element may also not be present in the bitstream for derived layers due to the fact that the paraloop, within the “<EMPHASIS>“ and “</EMPHASIS> tags “ in table 2-103noniens above, may only be applicable for explicitly flagged layers.

[0095] Additionally, layered HEVC, by design, requires that profile, row, and level information be present for each required layer (i.e., a layer that is either an output target layer or required to decode the output target layer). The DAM3 design lacks such information for the derived layers. Furthermore, for a non-required layer (that is, a layer that is not a required layer), the PTL information is not signaled according to the coding specification, and therefore, in the transport flow conduction design, the Continuous PTL signaling to non-required layers is problematic since there is no PTL information for these layers to signal.

[0096] Below, exemplary techniques that can address the above issues are described. However, this should not be interpreted as a requirement that the techniques described below address the above issues. Exemplary techniques may be applied by video encoder 20, multiplexer 21, demultiplexer 29, an intermediate device (e.g., MANE), or other device as part of producing the bit stream that video decoder 30, for example, Ultimately, it receives or can be applied by the video decoder 30 or other device as part of decoding the video data to produce the images to be displayed. Some of the techniques can be applied independently and some of them can be applied in combination. Furthermore, although the disclosure describes HEVC operating point descriptors and otherwise refers to HEVC, the techniques of the present disclosure may be applicable to video coding standards other than HEVC.

[0097] In accordance with a first technique of the present disclosure, a flag is signaled for each elementary stream (ES)/layer at the HEVC operating point to indicate whether the ES/layer is a required layer or not. In the present disclosure, references to ES/layer or layer/ES refer to an elementary flow corresponding to a layer or a temporal sublayer of the layer. Therefore, data that identifies an elementary flow can be used to identify the corresponding layer vice versa. For example, video encoder 20, multiplexer 21, or an intermediate device may emit (e.g., signal) a flag (e.g., necessary_layer_flag[i][k], described above) in the bit stream for each elementary stream or layer at the point of operation that indicates whether the flow or elementary layer is required or not. The video decoder 30 may receive such a flag and use the flag to decode from the bit stream. In other words, the video decoder 30 can decode the operating point based on the received flag.

[0098] In accordance with a second technique of the present disclosure, the semantics of prepend_dependencies[i][j] are updated to clarify that the inclusion of an ES/layer in the list of elementary flows for an operating point will be applied only when the ES is not present on the list yet. For example, the semantics of prepend_dependencies[i][j] equal to 0 is added and clarifies that an elementary layer/flow is included in the list of elementary flows for target point of operation only if the elementary layer/flow is not present in the list.

[0099] Thus, in some examples, a device, such as video encoder 20, generates a first descriptor, such as an operating point descriptor. Additionally, the device may generate a second descriptor, such as a hierarchy descriptor or a hierarchy extension descriptor. As part of generating the first descriptor, the device may include a first syntax element and a second syntax element in the first descriptor. In at least some examples, a list of elementary streams is a list of elementary streams that are part of the operating point and the second descriptor has a hierarchy layer index value equal to a value of the second syntax element. A first value of the first syntax element specifies that an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, and an elementary stream indicated by an embedded layer index in the second descriptor, when it is not present in the list of elementary flows, it must be added to the list of elementary flows. A second value of the first syntax element specifies that the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, but not the elementary stream indicated by the layer index. incorporated in the second descriptor. The device may include the first descriptor, and the second descriptor in a transport stream.

[00100] In a similar example, a device, such as video decoder 30, receives a transport stream that includes a first descriptor, a second descriptor, and a plurality of elementary streams. The first descriptor may be a descriptor for an operating point of the video data. The second descriptor can be one of: a hierarchy descriptor or a hierarchy extension descriptor. The device may decode a first syntax element and a second syntax element in the first descriptor. In at least some examples, a list of elementary streams is a list of elementary streams of the transport stream that are part of the operating point, and the second descriptor has a hierarchy layer index value equal to a value of the second element of syntax. Furthermore, in such examples, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, shall be added to the list of elementary streams, the The elementary stream indicated by the second syntax element is one of a plurality of elementary streams, and an elementary stream indicated by a layer index embedded in the second descriptor, when not present in the list of elementary streams, must be added to the list of elementary streams. , with the elementary flow indicated by the layer index incorporated in the second descriptor being one of a plurality of elementary flows. In response to determining that the first syntax element has a second value different from the first value, the device may add the elementary stream indicated by the second syntax element, when it is not present in the list of elementary streams, to the list of elementary streams, however, it may not add the elementary flow indicated by the layer index embedded in the second descriptor to the list of elementary flows.

[00101] In accordance with a third technique of the present disclosure, a restriction is added to the value of ES_reference[i][j] so that the same elementary flow is not explicitly signaled more than once for any particular operating point. In other words, it is suggested to additionally impose a restriction that the same elementary flow should not be explicitly signaled more than once for any particular operating point. For example, the video encoder 20 or the intermediate device may be configured so that the same elementary stream is not explicitly signaled more than once.

[00102] Furthermore, the present disclosure describes several exemplary techniques that solve the second issue described above (i.e., the problem of being able to determine values of certain syntax elements for a derived layer, such as the output layer flag problem missing and the PTL information problem). In some of these techniques to solve the second issue, it is proposed to maintain the functionality of the derivation layers for an operating point, signal the missing information to the derived layers, however, not signal PTL to unnecessary layers.

[00103] In a first exemplary technique to resolve the second issue, the prepend_dependencies[i][j] syntax elements are removed and all layers are always explicitly signaled.

[00104] In a second exemplary technique for solving the second question, two loops for the layers of an operating point in the HEVC operating point descriptor are introduced. A loop contains the ES_reference[i][j] syntax elements and the prepend_dependencies[i][j] syntax elements of layers that are explicitly associated with the reference to an elementary stream (ES_reference [i][j]). Another loop contains the output_layer_flag[i][k] syntax elements and the ptl_ref_idx[i][j] syntax elements of all layers. Video encoder 20, multiplexer 21, demultiplexer 29, video decoder 30, an intermediate device, and/or other device may deploy these exemplary loops to encode or decode the HEVC operating point descriptor.

[00105] In one instance of the second exemplary technique for solving the second question, a device may be configured to cause a first loop to generate or decode multiple instances (e.g., sets) of a first syntax element (e.g., ES_reference [i][j]) in a HEVC operating point descriptor. The first syntax element indicates the hierarchy layer index value present in the hierarchy descriptor or HEVC hierarchy extension descriptor that identifies an elementary flow from an operating point. In the first loop, the device also generates or decodes multiple occurrences of a second syntax element (for example, prepend_dependencies[i][j]). The second syntax element indicates from which structures elementary flows should be included in a list of elementary flows for the operating point, as part of realizing the first loop. Furthermore, in this example, the device may cause a second loop to generate or decode multiple occurrences of a third syntax element (e.g., output_layer_flag[i][k]) that indicates which elementary stream of the operating point is a layer. about to leave. Additionally, in the second loop, the device may generate or decode multiple occurrences of a fourth syntax element (e.g., ptl_ref_idx[i][k]) that indicates an index to determine a profile, row, or level of an elementary stream from the point of operation. In some instances, the device may generate a transport stream that includes the HEVC operating point descriptor or decodes the operating point based on one or more of the first, second, third, and fourth syntax elements.

[00106] In a third exemplary technique for solving the second question, the two loops in the above second exemplary technique for solving the second question may be joined together, with a flag for each layer indicating whether the layer is explicitly or implicitly flagged. For this example, it is assumed that the layers as derived by the hierarchy descriptor or hierarchy extension descriptor indicated by ES_reference[i][j] form a certain order, such that it is possible for the layers as ordered in the second loop to map each of the explicitly flagged layers to the layers as flagged or derived by the information in the first loop.

[00107] A fourth exemplary technique to solve the second question still allows the functionality of the prepend_dependencies[i][j] syntax element, however, when it is applied, the possibility of a derived layer being an output layer is inferred by a default value. For example, when prepend_dependencies[i][j] equals 1, a device can infer that the value of an output_layer_lag[i][j] syntax element is equal to a default value. Such a default value, for example, 0 or 1, can be signaled to the current operating point. The profile, row, and level of a derived layer are set so that they are the same as the same layer that was previously explicitly flagged at a different operating point.

[00108] In order not to signal PTLs for unnecessary layers, the present disclosure proposes to signal a flag for each ES/layer at the HEVC operating point to indicate whether the ES/layer is a necessary layer or not. Signaling uses a bit that would otherwise be reserved, consequently no additional overhead is introduced, and signaling avoids a complicated derivation process that might otherwise need to be specified. For example, a device (e.g., video encoder 20, multiplexer 21, demultiplexer 29, video decoder 30, an intermediate device, or other device) may be configured to determine elementary streams or layers from an operating point , to determine a flag (e.g., necessary_layer_flag[i][k], described below) for each elementary flow or layer at the point of operation, wherein the flag indicates whether a respective elementary flow or layer is a necessary flow or layer, and produces a bit stream that includes the respective elementary streams and flags.

[00109] Figure 2 is a block diagram illustrating an example video encoder 20 that can implement techniques for transporting video data encoded in accordance with extensions of a video coding standard. The video data may include multiple (e.g., two or more) enhancement layers to a base layer, where the enhancement layers may correspond to different scalability dimensions.

[00110] In the example of Figure 2, the video encoder 20 includes the video data memory 49, the mode selection unit 40, the decoded picture buffer (DPB) 64, the adder 50, the processing unit transform unit 52, the quantization unit 54, and the entropy coding unit 56. The mode selection unit 40, in turn, includes the mode compensation unit 44, the motion estimation unit 42, the intraprediction 46 and the partitioning unit 48. For the video reconstruction block, the video encoder 20 also includes the inverse quantization unit 58, the inverse transform unit 60 and the adder 62.

[00111] Video data memory 49 can store video data to be encoded by video encoder components 20. Video data stored in video data memory 49 can be obtained, for example, from the video data source video 18. DPB 64 may be a reference image memory that stores ES_reference video data for use in encoding video data by video encoder 20, for example, in intra- or inter-coding modes. Video data memory 49 and DPB 64 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), RAM resistive (RRAM) or other types of memory devices. The video data memory 49 and the DPB 64 may be provided by the same memory device or by separate memory devices. In various examples, video data memory 49 may be on-chip with other video encoder components 20 or off-chip with respect to those components.

[00112] During the encoding process, video encoder 20 receives a video frame or slice to be encoded. The frame or slice can be divided into multiple video blocks. The motion estimation unit 42 and the motion compensation unit 44 perform interprediction coding of the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. The intra-prediction unit 46 may alternatively perform intra-prediction coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be encoded to provide spatial prediction. The video encoder 20 may perform multiple pass encoding steps, for example, to select an appropriate encoding mode for each block of video data.

[00113] Furthermore, the partitioning unit 48 may partition blocks of video data into sub-blocks, e.g., based on evaluation of previous partitioning schemes in previous encoding passes. For example, the partitioning unit 48 may initially partition a frame or slice into LCUs, and partition each of the LCUs into sub-CUs based on rate distortion analysis (e.g., rate distortion optimization). The mode selection unit 40 may further produce a quadratic tree data structure indicating the partitioning of an LCU into sub-CUs. The leaf node CUs of the quadratic tree may include one or more PUs and one or more TUs.

[00114] The mode selection unit 40 may select one of the coding modes, intra or inter, for example, based on error results, and may provide the resulting intra or interpredicted block to the adder 50 to generate block data residuals and to adder 62 to reconstruct the encoded block for use in a reference frame. The mode selection unit 40 also provides syntax elements, such as motion vectors, intramode indicators, partitioning information, and other such syntax information, to the entropy coding unit 56.

[00115] Motion estimation, performed by motion estimation unit 42, is the process of generating motion vectors, which estimates motion for video blocks. The motion estimation unit 42 may calculate a motion vector for a PU of a video block in an intercoded slice by comparing the position of the PU with the position of a prediction block of a reference image. The reference image may be selected from a first reference image list (RefPicListO) or a second reference image list (RefPicListl), each of which identifies one or more reference images stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to entropy coding unit 56 and motion compensation unit 44. Motion compensation, performed by motion compensation unit 44, may involve fetching or generating the prediction block based on the motion vector determined by motion estimation unit 42. The adder 50 forms a residual video block by subtracting pixel values of the prediction block from the pixel values of the current video block that is encoded , which forms pixel difference values, as discussed below. The mode selection unit 40 may also generate syntax elements associated with the video blocks and the video slice for use by video decoder 30 in decoding the video blocks of the video slice.

[00116] The intraprediction unit 46 may intrapredict a current block, as an alternative to the interprediction performed by the motion estimation unit 42 and the motion compensation unit 44, as described above. After selecting an intraprediction mode for a block, the intraprediction unit 46 may provide information indicative of the selected intraprediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 may encode information indicating the selected intraprediction mode.

[00117] The video encoder 20 may form a residual video block by subtracting the prediction data of the mode selection unit 40 from the original video block that is encoded. Adder 50 represents the component or components that perform this subtraction operation. The transform processing unit 52 may apply a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, which produces a video block comprising residual transform coefficient values. The quantization unit 54 may quantize the transform coefficients to further reduce bit rate. After quantization, the entropy coding unit 56 may entropy code syntax elements indicating the quantized transform coefficients. For example, the entropy coding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CAB AC), syntax-based context adaptive binary arithmetic coding (SBAC), entropy coding of probability interval partitioning (PIPE) or other entropy coding technique. After entropy coding by entropy coding unit 56, the encoded bit stream may be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[00118] The inverse quantization unit 58 and the inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, for later use as a reference block. The motion compensation unit 44 may calculate a reference block by adding the residual block to a prediction block of one of the DPB frames 64. The adder 62 may add the reconstructed residual block to the motion-compensated prediction block produced by the motion compensation unit 44 to produce a reconstructed video block for storage in DPB 64. The reconstructed video block can be used by the motion estimation unit 42 and the motion compensation unit 44 as a reference block to interconvert into code a block in a subsequent video frame.

[00119] In some examples, the video encoder 20 of Figure 2 generates a first descriptor, such as a descriptor operating point. Additionally, video encoder 20 may generate a second descriptor, such as a hierarchy descriptor or a hierarchy extension descriptor. As part of generating the first descriptor, the video encoder 20 may include a first syntax element and a second syntax element in the first descriptor. In at least some such examples, a list of elementary streams is a list of elementary streams that are part of the operating point and the second descriptor has a hierarchy layer index value equal to a value of the second syntax element. A first value of the first syntax element specifies that an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, is to be added to the list of elementary streams, and an elementary stream indicated by a layer index embedded in the second descriptor, when not present in the list of elementary flows, must be added to the list of elementary flows. A second value of the first syntax element specifies that the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, but not the elementary stream indicated by the layer index embedded in the second descriptor. The video encoder 20 may include the first descriptor and the second descriptor in a transport stream.

[00120] Figure 3 is a block diagram illustrating an example of a video decoder 30 that can implement techniques for transporting video data encoded according to extensions of a video coding standard. In the example of Figure 3, the video decoder 30 includes a video data memory 69, an entropy decoding unit 70, motion compensation unit 72, intraprediction unit 74, inverse quantization unit 76, inverse transformation unit 78, decoded picture temporary storage (DPB) 82, and adder 80.

[00121] Video data memory 69 may store encoded video data, as an encoded video bit stream, to be decoded by video decoder components 30. The encoded video data stored in video data memory 69 may be obtained, for example, from computer-readable medium 16, for example, from a local video source such as a camera, through wireless or wired video data network communication, or by accessing it. if physical data storage media. The video data memory 69 may form a coded recording buffer (CPB) that stores coded video data from a coded video bit stream. The DPB 82 may be a reference picture memory that stores reference video data for use in decoding video data by video decoder 30, for example, in intra- or inter-coding modes. Video data memory 69 and DPB 82 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), which includes synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), RAM resistive (RRAM), or other types of memory devices. The video data memory 69 and the DPB 82 may be provided by the same memory device or by separate memory devices. In various examples, video data memory 69 may be on-chip with other video decoder components 30, or off-chip with respect to those components.

[00122] Video data memory 69 can receive and store encoded video data (e.g., NAL units) from a bit stream. The entropy decoding unit 70 may receive encoded video data (e.g., NAL units) from the video data memory 69 and may parse the NAL units to decode the syntax elements. During the decoding process, the video decoder 30 receives an encrypted video bit stream representing video blocks of an encoded video slice and associated syntax elements from the video encoder 20. The entropy decoding unit 70 of video decoder 30 entropy decodes the bit stream to generate quantized coefficients, intraprediction mode indicators, and other syntax elements.

[00123] The intraprediction unit 74 may generate prediction data for a video block of the current video slice based on a signaled intraprediction mode and data from previously decoded blocks of the current frame or image. The motion compensation unit 72 produces prediction blocks for a video block of the current video slice based on the motion vectors and other syntax elements received from the entropy decoding unit 70. The prediction blocks can be produced from of one of the reference images within one of the reference image lists. The motion compensation unit 72 can determine prediction information for a video block using prediction information, and can produce the prediction blocks for the current video block that is decoded.

[00124] The inverse quantization unit 76 may inversely quantize the quantized transform coefficients provided in the bit stream and decoded by entropy decoding unit 70. The inverse transform unit 78 may apply an inverse transform, e.g. Inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients to produce residual blocks in the pixel domain.

[00125] After the motion compensation unit 72 generates the prediction block for the current video block based on the motion vectors and other syntax elements, the video decoder 30 may form a decoded video block by summing the residual blocks of the inverse transform unit 78 with the corresponding prediction blocks generated by motion compensation unit 72. The adder 80 represents the component or components that perform this summation operation. The video blocks decoded into a given frame or image are then stored in DPB 82, which stores reference images used for subsequent motion compensation. The DPB 82 also stores decoded video for later presentation on a display device, such as display device 32 of Figure 1.

[00126] In some examples, the video decoder 30 receives a transport stream that includes a first descriptor, a second descriptor, and a plurality of elementary streams. The first descriptor may be a descriptor for an operating point of the video data. The second descriptor may be one of: a hierarchy descriptor or a hierarchy extension descriptor. The video decoder 30 may decode a first syntax element and a second syntax element in the first descriptor. In at least some such examples, a list of elementary streams is a list of elementary streams of the transport stream that are part of the operating point, and the second descriptor has a hierarchy layer index value equal to a value of the second syntax element. Furthermore, in such examples, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, is to be added to the list of elementary streams, wherein the elementary stream indicated by the second syntax element is one of the plurality of elementary streams, and an elementary stream indicated by a layer index embodied in the second descriptor, when not present in the list of elementary streams, is to be added to the list of elementary streams, wherein the elementary stream indicated by the layer index embodied in the second descriptor is one of the plurality of elementary streams. In response to determining that the first syntax element has a second value different from the first value, the video decoder 30 may add the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, to the list of elementary streams, but not add the elementary stream indicated by the layer index embodied in the second descriptor to the list of elementary streams.

[00127] Figure 4 is a flowchart illustrating an exemplary operation of a device that generates a transport flow, in accordance with a technique of this disclosure. The flowcharts of this disclosure are provided as examples of operations. Other operations in accordance with techniques of this disclosure may include more, less, or different actions, or may perform such actions in different orders or in parallel. In the example of Figure 4, the device may be source device 12 or another type of device for processing video data. The operation in Figure 4 can implement a solution to the second issue described above (i.e., the problem of not being able to determine values of certain syntax elements for a derived layer).

[00128] In the example of Figure 4, the device generates a first descriptor (for example, a HEVC operating point descriptor) (150). The first descriptor is a descriptor of an operating point. An elementary flow list (for example, OperationPointESList[i]) is a list of elementary flows of the transport stream that are part of the operation point. A device receiving or processing a stream can generate the elementary stream list by adding elementary streams to the elementary stream list. Furthermore, the device generates a second descriptor (152). The second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor. The device may include the first descriptor and the second descriptor in a transport stream (154). For example, the device may output a series of bits representing the first descriptor and the second descriptor in the transport stream.

[00129] Furthermore, in the example of Figure 4, as part of generating the first descriptor, the device includes a first syntax element (e.g., prepend_dependencies[i][j]) in the first descriptor (156). Additionally, as part of generating the first descriptor, the device includes a second syntax element (e.g., ES_reference[i][j]) in the second descriptor (158). The second descriptor has a hierarchy layer index value (for example, hierarchy_layer_index) equal to a value of the second syntax element. When the device includes a syntax element in a descriptor, the device may store a value of the syntax element in a memory in a manner that indicates that the value is part of the descriptor, e.g., as part of a data structure.

[00130] In the example of Figure 4, a first value (e.g., 1) of the first syntax element specifies that an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary flows. Additionally, the first value of the first syntax element specifies an elementary stream indicated by an embedded layer index (e.g., hierarchy_embedded_layer_index or hierarchy_ext_embedded_layer_index) in the second descriptor, when not present in the list of elementary streams, must be added to the list of elementary streams . A second value of the first syntax element specifies that the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, but not the elementary stream indicated by the layer index embedded in the second descriptor.

[00131] Figure 5 is a flowchart illustrating an exemplary operation of a device that receives a transport flow, in accordance with the technique of this disclosure. In the example of Figure 5, the device may be target device 14, a MANE, or another type of device for processing video data. The operation in Figure 5 can implement a solution to the second issue described above (i.e., the problem of not being able to determine values of certain syntax elements for a derived layer).

[00132] In the example of Figure 5, the device receives a transport stream that includes a first descriptor (e.g., a HEVC operating point descriptor), a second descriptor and a plurality of elementary streams (200). The first descriptor is a descriptor for an operating point. The second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor. In some examples, input interface 28 receives the transport stream.

[00133] Furthermore, the device may decode a first syntax element (e.g., prepend_dependencies[i][j]) in the first descriptor (202). In other words, the device can determine a value from the first syntax element. Exemplary ways of decoding a syntax element may include parsing the syntax element to a series of bits, applying an entropy decoding algorithm, or otherwise deriving the value of the syntax element. Additionally, the device may decode a second element of different syntax (e.g., ES_reference[i][j]) in the first descriptor (204). In other words, the device may determine a value of the second syntax element, for example, by analyzing the second bit syntax element of the first descriptor. An elementary flow list (for example, OperationPomtESListf[i]) is a list of elementary flows of the transport stream that are part of the operation point. The second descriptor has a hierarchy layer index value (for example, hierarchy_layer_index) equal to a value of the second syntax element.

[00134] In the example of Figure 5, a first value (e.g., 1) of the first syntax element specifies that an elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary flows. The elementary stream indicated by the second syntax element may be one of a plurality of elementary streams received by the device. Additionally, the first value of the first syntax element specifies an elementary stream indicated by an embedded layer index (e.g., hierarchy_embedded_layer_index or hierarchy_ext_embedded_layer_index) in the second descriptor, when not present in the list of elementary streams, must be added to the list of elementary streams . The elementary stream indicated by the layer index embedded in the second descriptor may be one of a plurality of elementary streams received by the device. A second value (e.g., 0) of the first syntax element specifies that the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, but not the elementary stream indicated by the layer index embedded in the second descriptor.

[00135] Thus, in some examples, such as the example of Figure 5, the device may determine the first syntax element has the second value (206). In response to determining that the first syntax element has the second value, the device may add the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, to the list of elementary streams, but does not add the elementary stream indicated by the layer index embedded in the second descriptor to the list of elementary streams (208). For example, a device may add an elementary stream to a list by storing a reference to the elementary stream in a location in memory in a manner that indicates that the elementary stream is in the list, such as by adding the reference to the elementary stream to an array or linked list.

[00136] Although not illustrated in the example of Figure 5, the device may discard data for layers that do not correspond to the operating point. For example, the device may discard transport packets from elementary streams that correspond to layers that do not correspond to the operating point. The device may forward the remaining data (e.g., elementary stream transport packets corresponding to the operating point layers) to another device or a video decoder, such as video decoder 30.

[00137] The following text describes some exemplary ways to implement the aspects described above. Deployment details are described in terms of changes to DAM3. In the text changes of these deployments, text included by the “<ins>“ and “</ins>“ tags is added or modified (for example, <ins>inserted or modified text</ins>), while deletions are included in the “<dlt>“ and “<dlt>“ tags (e.g. <dlt>deleted text</dlt>).

[00138] According to a first exemplary implementation technique, Table 1 and its syntax elements describe detailed modifications to DAM3 for the first, second and third examples to resolve the first issue described above and the second example to resolve the second question described above. TABLE 1

[00139] The following text indicates modifications to the semantics according to the first exemplary implementation technique. num_ptl - This <dlt>8-bit</dlt><ins>6-bit</ins> field specifies the number of profile, row, and level structures flagged in this descriptor. <ins>Let OperationPointESList[i] be the list of ESs that are part of the i-th HEVC operation point. </ins> prepend_dependencies[i][j] - This flag, if set to 1 <dlt>indicates</dlt><ins> specifies that the ES indicated by ES_reference[i][j], when not already present in OperationPointESList[i, should be added to OperationPointESList[i], and </ins><dlt>that</dlt> the ES <ins>indicated</ins> by the hierarchy_embedded_layer_index syntax element in the hierarchy descriptor, or all ESs <ins>indicated</ins> by the hierarchy_ext_embedded_layer_index syntax element in the HEVC hierarchy extension descriptor, with the hierarchy layer index value specified by the following syntax element ES_reference[i][j]<ms>, when not already present in OperationPointESList[i],</ms> should be added <dlt> to the elementary streams for the target operation point </dlt> <ins> in OperationPointLayerList[i] immediately </ins> before the ES flagged by ES_reference[i][j] <ins> in ascending order of their associated hierarchy_embedded_layer_index or hierarchy_ext_embedded_layer_index values. When the value of prepend_dependencies[i][j] is equal to 0, only the ES flagged by ES_reference[i][j], when not already present in OperationPointESList[i], should be added to OperationPointESList[i]. The ES flagged by ES_reference[i][m] should be placed earlier (i.e., with a lower index) in OperationPointESList[i] than the ES flagged with ES_reference[i][n] when m is less than n</ins>. <ins>CHECK: The order of ES in OperationPointESList[i] must be in ascending order of their hierarchy_layer_index values. </ins> ES_reference[i][j] - This 6-bit field indicates the hierarchy layer index value present in the HEVC hierarchy descriptor or hierarchy extension descriptor that identifies an elementary stream. <ins>The value of ES_reference[i][m] and ES_reference[i][n] for m is not equal to n and must not be the same.</ins> <ins>Let NumESinOP[i] be the number of ESs in OperationPointESList[i] after all ESs that are part of the i-th HEVC operation point have been included in OperationPointESList[i] (i.e., after parsing ES_reference[i][ES_count[i]-1]).</ins> <ins>necessary_layer_flag[i][k] - This flag, when set to T indicates that the k-th ES in OperationPointESList[i] is a necessary layer, as defined in 23008-2, of the i-th operation point. This flag equal to "0" indicates that the k-th ES in OperationPointESList[i] is not a required layer, as defined in 23008-2, of the i-th operation point.</ins> output_layer_flag[i]<ins>[k]</ins> - This flag when set to T indicates that the <ins>k</ins>-th <ins>ES in OperationPointESList[i]</ins> <dlt>layer of the i-th HEVC operation point defined in this descriptor</dlt> is an output layer. Otherwise, when set to "0", it indicates that the <ins>k</ins>-th <ins>ES in OperationPointESList[i]</ins> <dlt>layer of the i-th HEVC operation point defined in this descriptor</dlt> is not an output layer. <ins>When the value of necessary_layer_flag[i][k] is equal to 0, the value of output_layer_flag[i][k] should be ignored.</ins> ptl_ref_idx[i][k] - <dlt> An 8-bit field</dlt> <ins>A 6-bit</ins> indicating the index x into the element of profile_tier_level_info[x] of the profile_tier_level_array that applies to the <ins>k</ins>-th <ins>ES in OperationPointESList[i]</ins><dlt>layer of the i-th HEVC operation point defined in this descriptor</dlt>. <ins>When the value of necessary_layer_flag[i][k] is equal to 0, the value of ptl_ref_idx[i][k] should be ignored.</ins>

[00140] Figure 6 is a flowchart illustrating an exemplary operation of a device for generating a transport flow in accordance with an exemplary technique of this disclosure. For example, the operation of Figure 6 may be in accordance with the first exemplary implementation technique of this disclosure discussed above (i.e., the technique described above in relation to Table 1). The operation of Figure 6 can be performed by source device 12 or another device. The operation in Figure 6 can be an expanded version of the operation in Figure 4.

[00141] In the example of Figure 6, the device can generate a set of second descriptors (250). Each respective second descriptor among the set of second descriptors may be one of: a hierarchy descriptor or a hierarchy extension descriptor. The set of second descriptors may include the “second descriptor” described with respect to Figure 4.

[00142] Additionally, the device can generate a first descriptor (for example, a HEVC operating point descriptor) (252). As part of generating the first descriptor, the device may include an elementary stream count syntax element (e.g., ES_count) in the first descriptor (254). The elementary stream count syntax element can indicate a number of elementary streams.

[00143] Furthermore, as part of generating the first descriptor, the device may include a set of first syntax elements (e.g., prepend_dependencies[i][j]) in the first descriptor (256). Additionally, as part of generating the first descriptor, the device may include a set of second syntax elements (e.g., ES_reference[i][j]) in the first descriptor (258). The number of first syntax elements in the set of first syntax elements and the number of second syntax elements in the set of second syntax elements is equal to an elementary stream count syntax element value. The set of first syntax elements may include the "first syntax element" referred to in relation to Figure 4. The set of second syntax elements may include the "second syntax element" referred to in relation to Figure 4.

[00144] In some examples, the set of second syntax elements is restricted such that no two second syntax elements within the set of second syntax elements have the same value. In other words, as indicated in the changes to DAM3 above, the value of ES_reference[i][m] and ES_reference[i][n] for m is not equal to n and must not be the same.

[00145] For each respective first syntax element among the set of first syntax elements, a respective second syntax element among the set of second syntax elements corresponds to the respective first syntax element. For example, a first syntax element and a second syntax element with the same i and j values correspond to each other. A respective second descriptor among the set of second descriptors has a hierarchy layer index value (for example, hierarchy_layer_id) equal to a value of the respective second syntax element.

[00146] In the example of Figure 6, a first value (e.g., 1) of the respective first syntax element specifies that an elementary stream indicated by the respective second syntax element, when not present in the list of elementary streams, should be added to the list of elementary flows, and an elementary flow indicated by a layer index embedded in the respective second descriptor, when not present in the list of elementary flows of the operating point, shall be added to the list of elementary flows. A second value (e.g., 0) of the respective first syntax element specifies that the respective elementary stream indicated by the respective second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, but not the respective elementary flow indicated by the layer index embedded in the respective second descriptor.

[00147] Furthermore, in the example of Figure 6, the device may include a set of necessary layer flag syntax elements (e.g., necessary_layer_flag[i][k]) in the first descriptor (260). Each respective syntax element in the set of required layer flag syntax elements can correspond to a respective elementary stream in the list of elementary streams. Each respective syntax element among the set of required layer flag syntax elements indicates whether the elementary stream that corresponds to the respective syntax element among the set of required layer flag syntax elements is a required layer. A required layer can be defined as a layer at an output operating point associated with a set of output layers, wherein the layer is an output layer of the set of output layers, or a reference layer of a set of output layers. output from the output layer set.

[00148] The device may also include a set of output layer syntax elements (e.g., output_layer_flag[i][k]) in the first descriptor (262). Each respective syntax element in the set of output syntax elements can correspond to a respective elementary stream in the list of elementary streams. Each respective syntax element among the set of output layer flag syntax elements may indicate whether the elementary stream corresponding to the respective syntax element among the set of output layer flag syntax elements is an output layer.

[00149] Furthermore, the device may include a set of PTL reference index syntax elements (e.g., ptl_ref_idx[i][k]) in the first descriptor (264). Each respective syntax element in the PTL reference index syntax element set can correspond to a respective elementary stream in the list of elementary streams. Each respective syntax element among the set of PTL reference index syntax elements may indicate an index to determine a profile, a row, or a level of the elementary stream that corresponds to the respective syntax element among the set of PTL syntax elements. PTL benchmark.

[00150] In the example of Figure 6, the device may include the first descriptor and the second descriptor in a transport stream (266). For example, the device may generate a set of transport packets that contain the first descriptor and the second descriptor. The transport stream may additionally include one or more elementary streams. In other examples, one or more actions from Figure 6 may be omitted.

[00151] Figure 7 is a flow chart illustrating an exemplary operation of a device that receives a transport flow in accordance with an exemplary technique of this disclosure. For example, the operation of Figure 7 may be consistent with the first exemplary deployment technique discussed above. The operation in Figure 7 can be an expanded version of the operation in Figure 5.

[00152] In the example of Figure 7, the device receives a transport stream that includes a first descriptor (e.g., a HEVC operating point descriptor), a set of second descriptors and a plurality of elementary streams (300). The set of second descriptors may include the "second descriptor" mentioned above in relation to Figure 5. Each respective second descriptor among the set of second descriptors may be one of: a hierarchy descriptor or a hierarchy extension descriptor.

[00153] Additionally, in the example of Figure 7, the device may decode an elementary stream count syntax element (e.g., ES_count, which may be called in the claims a “third syntax element”) in the first descriptor. The elementary stream count syntax element indicates a number of elementary streams.

[00154] Furthermore, in the example of Figure 7, the device may decode a set of first syntax elements (e.g., prepend_dependencies[i][j]) in the first descriptor (302). Furthermore, in the example of Figure 7, the device may decode a set of second syntax elements (e.g., ES_reference[i][j]) in the first descriptor (304). The number of first syntax elements in the set of first syntax elements and the number of second syntax elements in the set of second syntax elements is equal to a value of the third syntax element. The set of first syntax elements may include the "first syntax element" mentioned in relation to Figure 5. The set of second syntax elements may include the "second syntax element" mentioned in relation to Figure 5.

[00155] In some examples, the set of second syntax elements is restricted so that no two second syntax elements among the set of second syntax elements have the same value. In other words, as stated in the changes to DAM3 above, the value of ES_reference[i][m] and ES_reference[i][n] for m is not equal to n and must not be the same.

[00156] For each respective first syntax element among the set of first syntax elements, a respective second syntax element among the set of second syntax elements corresponds to the respective first syntax element. For example, a first syntax element and a second syntax element with the same values of i and j correspond to each other. A respective second descriptor among the set of second descriptors has a hierarchy layer index value (e.g., hierarchy_layer_id) equal to a value of the respective second syntax element.

[00157] In the example of Figure 7, a first value (e.g., 1) of the respective first syntax element specifies that an elementary stream indicated by the respective second syntax element, when not present in the list of elementary streams, should be added to the list of elementary flows, and an elementary flow indicated by a layer index embedded in the respective second descriptor, when not present in the list of elementary flows of the operating point, shall be added to the list of elementary flows. The elementary stream indicated by the respective second syntax element may be one of a plurality of elementary streams. A second value (e.g., 0) of the respective first syntax element specifies that the respective elementary stream indicated by the respective second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, but not the respective elementary flow indicated by the layer index embedded in the respective second descriptor. The elementary flow indicated by the layer index embedded in the respective second descriptor may be one of a plurality of elementary flows.

[00158] For each respective first syntax element among the set of first syntax elements, a set of one or more embedded layer indices is included in the respective second descriptor. Furthermore, in the example of Figure 7, based on the respective first syntax element having the first value, the device may add, in ascending order of value from the set of embedded layer indices, elementary flows indicated by the set of embedded layer indices. in the list of elementary streams immediately before the elementary stream indicated by the respective second syntax element (308). In other words, in response to determining that the respective first syntax element has the second value, the device may add the respective elementary stream indicated by the respective second syntax element, when not present in the list of elementary streams, to the list of streams elementary flows, but does not add the respective elementary flow indicated by the layer index embedded in the respective second descriptor to the list of elementary flows.

[00159] In the example of Figure 7, the device may decode a set of necessary layer flag syntax elements (e.g., necessary_layer_flag[i][k]) in the first descriptor (310). Each respective syntax element in the set of required layer flag syntax elements can correspond to a respective elementary stream in the list of elementary streams. Each respective syntax element among the set of required layer flag syntax elements may indicate whether the elementary stream corresponding to the respective syntax element among the set of required layer flag syntax elements is a required layer. As described above, a required layer is defined as a layer at an output operating point associated with a set of output layers, wherein the layer is an output layer of the set of output layers, or a reference layer of an output layer from the set of output layers.

[00160] Furthermore, in the example of Figure 7, the device may decode a set of output layer flag syntax elements (e.g., output_layer_flag[i][k]) in the first descriptor (312). Each respective syntax element in the set of output layer signaling syntax elements can correspond to a respective elementary stream in the list of elementary streams. Each respective syntax element among the set of output layer flag syntax elements indicates whether the elementary stream corresponding to the respective syntax element among the set of output layer flag syntax elements is an output layer.

[00161] Additionally, the device may decode a set of PTL reference index syntax elements in the first descriptor (314). Each respective syntax element in the PTL reference index syntax element set can correspond to a respective elementary stream in the list of elementary streams. Each respective syntax element among the set of PTL reference index syntax elements may indicate an index to determine a profile, a row, or a level of the elementary stream that corresponds to the respective syntax element among the set of PTL syntax elements. PTL benchmark.

[00162] In the example of Figure 7, the device may discard data for layers that are not part of the operating point (316). In other words, the device may discard data for layers that do not correspond to the operating point. In other examples, one or more actions from Figure 7 may be omitted.

[00163] According to a second exemplary implementation technique, Table 2 and its syntax elements describe detailed modifications to DAM3 for the third exemplary technique to resolve the second issue described above. TABLE 2

[00164] The following text indicates modifications to the semantics according to the second example deployment technique. <ins>es_present_flag[i][j] specifies whether the ES_reference[i][j] and prepend_dependencies[i][j] are present for the j-th layer of the i-th operating point. When es_present_flag[i][j] is equal to 0, the value of ES_reference[i][j] is derived based on information in the hierarchy descriptor or hierarchy extension descriptor of the layer previously flagged at the i-th operation point with es_present_flag values equal to 1 and prepend_dependencies values equal to 1. When es_present_flag[i][j] is equal to 0, the value of prepend_dependencies[i][j] is derived to be 0. The value of es_present_flag[i][0] must not be equal to 0.</ins> prepend_dependencies[i][j] - This flag, if set to 1, indicates that the ES flagged by the hierarchy_embedded_layer_index syntax element in the hierarchy descriptor, or all ESs flagged by the hierarchy_ext_embedded_layer_index syntax element in the hierarchy extension descriptor, are not flagged. HEVC, with the hierarchy layer index value specified by the following syntax element ES_reference[i][j] must be added to the list of elementary streams for the operation target point <d\t>before</d\t> <ins>after</ins> at the ES flagged by ES_reference[i][j]. <ins>VERIFY: The list of elementary streams after the operation target point is populated, it must be re-sorted in ascending order of its hierarchy_layer_index value.</ins> Note that the solutions to the required layer issues and the possibility of duplicating ES inclusion in the list of ESs for an HEVC operation can also be applied to this alternative.

[00165] According to a third exemplary implementation technique, Table 3 and its syntax elements describe detailed modifications to DAM3 for the fourth exemplary technique to resolve the second issue described above. TABLE 3

[00166] The following text indicates modifications to the semantics according to the third exemplary implementation technique. prepend_dependencies[i][j] - This flag, if set to 1, indicates that the ES flagged by the hierarchy_embedded_layer_index syntax element in the hierarchy descriptor, or all ESs flagged by the hierarchy_ext_embedded_layer_index syntax element in the HEVC hierarchy extension descriptor, with the hierarchy layer index value specified by the following syntax element ES_reference[i][j] should be added to the list of elementary streams for the target point of operation before the ES flagged by ES_reference[i][j]. <ins>When prepend_dependencies[i][j] is equal to 1, for each elementary stream esA that is flagged by the hierarchy_embedded_layer_index syntax element in the hierarchy descriptor, or all elementary streams flagged by the hierarchy_ext_embedded_layer_index syntax element in the HEVC hierarchy extension descriptor, with the hierarchy layer index value specified by the following syntax element ES_reference[i][j], the following applies: • If default_output_ref_layer[i] is equal to 0, the layer associated with esA is not an output layer of the i-th HEVC operation point; otherwise, the layer associated with esA is an output layer of the i-th HEVC operation point. • There must be at least one HEVC operation point opX with index between 0 to i to 1 that also contained esA. If there is more than one candidate for opX, the one with the highest index must be used. The index of the element of profile_tier_level_info[x] of the profile_tier_level_array that applies to esA at the i-th HEVC operating point is set equal to the index of the element of profile_tier_level_info[x] of the profile_tier_level_array that applies to esA at opX.</ins>

[00167] Furthermore, to avoid unnecessary complication, added dependencies may be flagged at the operation point level rather than at the layer level. The solutions to the required layer issues and the possibility of duplicating ES inclusion in the ES list for an HEVC operation may also be applied to this alternative. prepend_dependencies[i] - This flag, if set to 1, indicates that the ES flagged by the hierarchy_embedded_layer_index syntax element in the hierarchy descriptor, or all ESs flagged by the hierarchy_ext_embedded_layer_index syntax element in the HEVC hierarchy extension descriptor, with the hierarchy layer index value specified by the following syntax element ES_reference[i][j] should be added to the list of elementary streams for the target point of operation before the ES flagged by ES_reference[i][j] <ins>for each elementary stream indicated by ES_reference[i][j] of the i-th point of operation.</ins> <ins>When prepend_dependencies[i] is equal to 1, for each elementary stream esA that is flagged by the hierarchy_embedded_layer_index syntax element in the hierarchy descriptor, or all elementary streams flagged by the syntax element hierarchy_ext_embedded_layer_index in the HEVC hierarchy extension descriptor, with the hierarchy layer index value specified by the following syntax element ES_reference[i][j], the following applies: • If default_output_ref_layer[i] is equal to 0, the layer associated with esA is not an output layer of the i-th HEVC operation point; otherwise, the layer associated with esA is an output layer of the i-th HEVC operation point. • There must be at least one HEVC operation point opX with index between 0 and i to 1 that also contained esA. If there is more than one candidate for opX, the one with the highest index must be used. The index of the element of profile_tier_level_info[x] of the profile_tier_level_array that applies to esA at the i-th HEVC operating point is set equal to the index of the element of profile_tier_level_info[fx] of the profile_tier_level that applies to esA at opX.</ins>

[00168] In addition to the issue in DAM3 described above, when a layered HEVC bitstream contains one or more auxiliary layers, there is no mechanism for describing auxiliary enhancement in either the hierarchy descriptor or the hierarchy extension descriptor. An auxiliary image is an image that has no normative effect on the primary image decoding process, and with a value of nuh_alyer_id such that Auxld[nuh_layer_id] is greater than 0. An auxiliary layer is a layer that contains auxiliary images. To solve this problem, it is suggested to include description for auxiliary hierarchy type in Table 2-50 and auxiliary enhancement in Table 2-103.

[00169] For processing auxiliary layers, HEVC layering supports the presence of auxiliary layers in the bitstream. An auxiliary layer may contain, for example, an alpha channel or depth representation. The current text of ISO/IEC 13818-1:201x/DAM 3 Carriage of Layered HEVC does not have the mechanism for describing auxiliary layers. Transporting SHVC/MV-HEVC auxiliary layers in an elementary stream with a specific stream type (e.g. OxlE, i.e. Auxiliary video stream, as defined in ISO/IEC 23002-3) is not a sufficient solution due to the fact that there is still no way to describe the hierarchy of this elementary flow and other elementary flows, especially, the flow with which the auxiliary layer is associated with a program. Additionally, the current L-HEVC temporary storage model does not process stream types other than 0x27 ~ 0x2A, thus this solution (using a specific stream type) may require additional modification to the storage model. temporary.

[00170] To overcome the above problem, the disclosure describes the inclusion of auxiliary hierarchy type in Table 2-50 and the inclusion of auxiliary enhancement in Table 2-103 undeciens, below. With these modifications, there is no additional need to designate a specific stream type for auxiliary layers, no additional need to update the buffering mode, and an auxiliary layer can be carried in an elementary stream with a stream type in the range 0x27 ~ 0x2A.

[00171] The proposed text change for Tables 2 to 50 is as follows: Replace in Table 2-50 the description for values 8<ins>, 10</ins> and 15, as follows: TABLE 2-50 - HIERARCHY TYPE FIELD VALUES The proposed text change for Table 2-103 undeciens: TABLE 2-103UNDECIENS - EXTENSION DIMENSION BITS SEMANTICS

[00172] The following paragraphs provide additional examples of this disclosure.

[00173] In some examples, video decoder 30 may be configured to receive a first syntax element (e.g., prepend_dependencies[i][j]) for an operating point, wherein a first value of the syntax element specifies that an elementary flow indicated by a second syntax element (e.g., ES_reference[i][j]), when not already present in an operation point list (e.g., OperationPointESList[i]), must be added to the list of operation point, and wherein a second value of the syntax element specifies that only the elementary stream indicated by the second syntax element, when not already present in the operation list, should be added to the operation point list, and decode the operating point based on the first syntax element received.

[00174] In some examples, video decoder 30 may be configured to perform a first loop to decode multiple occurrences of a first syntax element (e.g., ES_reference[i][j]) that indicates the layer index value hierarchical descriptor present in the HEVC hierarchy descriptor or hierarchy extension descriptor that identifies an elementary flow from an operating point, and/or decodes multiple occurrences of a second syntax element (e.g., prepend_dependencies[i][j]) that indicates from which structures the elementary streams should be included in a list of elementary streams for the operating point, as part of the performance of the first loop, perform a second loop to decode multiple occurrences of a third syntax element (e.g. output_layer_flag[i][k]) which indicates which operating point elementary stream is an output layer and/or decode multiple occurrences of a fourth syntax element (e.g. ptl_ref_idx[i][k]) which indicates an index to determine a profile, a row, or a level of an elementary flow from the operating point, and decode the operating point based on one of the first, second, third, and fourth syntax elements.

[00175] In some examples, video encoder 20 or an intermediate device (e.g., a MANE) may be configured to implement the exemplary techniques described in this disclosure. For example, the video encoder 20 or the intermediate device (or a combination of the two) may be configured to determine elementary streams or layers from an operating point, determine a flag (e.g., necessary_layer_flag[i][k], described below) for each elementary stream or layer at the point of operation, wherein the flag indicates whether a respective elementary stream or layer is a required stream or layer, and produces a bit stream that includes the elementary streams and respective flags.

[00176] In one example, the video encoder 20 or the intermediate device (or a combination of the two) may be configured to determine a first syntax element (e.g., prepend_dependencies[i][j]) for an operating point , wherein a first syntax element value specifies that an elementary flow is indicated by a second syntax element (e.g., ES_reference[i][j]), when not already present in an operating point list (e.g., OperationPointESList[i]), shall be added to the operation point list, and where a second syntax element value specifies that only the elementary flow indicated by the second syntax element, when not already present in the operation list, shall be added to the operation point list, and produces a bitstream that includes operation point information that includes the first given syntax element.

[00177] In one example, the video encoder 20 or the intermediate device (or a combination of the two) may be configured to perform a first loop to determine multiple occurrences of a first syntax element (e.g., ES_reference[i][ j]) that indicates the hierarchical layer index value present in the HEVC hierarchy descriptor or hierarchy extension descriptor that identifies an elementary flow of an operating point, and/or determining multiple occurrences of a second syntax element (e.g. example, prepend_dependencies[i][j]) which indicates from which structures elementary flows should be included in a list of elementary flows for the operating point, as part of the performance of the first loop, perform a second loop to determine multiple occurrences of a third syntax element (e.g., output_layer_flag[i][k]) that indicates which operating point elementary stream is an output layer, and/or determining multiple occurrences of a fourth syntax element (e.g., ptl_ref_idx [i][k]) which indicates an index to determine a profile, a row, or a level of an elementary stream from the operating point, and produce a bit stream that includes one or more of the first, second, third and the fourth syntax elements for the operating point.

[00178] It should be recognized that depending on the example, certain events or actuations of any of the techniques described herein may be performed in a different sequence, may be added, merged, or left all together (e.g., not all of the events or actuations described are necessary to practice the techniques). Furthermore, in certain examples, events or actuations may be performed concurrently, e.g., through multi-threaded processing, switch processing, or multiple processors, rather than in a sequential manner.

[00179] In one or more examples, the described functions can be implemented in hardware, software, firmware or any combination thereof. If implemented in software, the functions may be stored in, or transmitted over, as one or more instructions or code in a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media, which include any media that facilitate the transfer of a computer program from one place to another, for example, according to a communication protocol. In this way, computer-readable media, in general, can correspond to (1) tangible computer-readable storage media that is non-transitory or (2) a communication medium, such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code structures and/or data for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[00180] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of data structures or instructions and that can be accessed by a computer. Furthermore, any connection is properly called a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transient media, but are instead directed to tangible, non-transient storage media. Magnetic disc and optical disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where discs, in general, reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above shall also fall within the scope of computer-readable media.

[00181] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other set of integrated or discrete equivalent logic circuits. Accordingly, the term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured to encode and decode or incorporated into a combined codec. Furthermore, the techniques can be completely implemented in one or more circuits or logic elements.

[00182] The techniques of this disclosure can be implemented in a wide variety of devices or apparatus, including a wireless headset, an integrated circuit (IC), or a set of ICs (e.g., a chip set). Various components, modules or units are described in this disclosure to emphasize the functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require performance by different hardware units. Instead, as described above, several units may be combined into a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors, as described above, together with suitable software and/or firmware. .

[00183] Several examples have been described. These and other examples are within the scope of the following claims.

Claims (15)

1. Method for processing video data, characterized by the fact that it comprises: receiving (200, 300) a transport stream that includes a first descriptor, a second descriptor and a plurality of elementary streams, wherein the first descriptor is a descriptor for an operating point of the video data, the second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor; decode (202, 204, 304, 306) a first syntax element and a second syntax element in the first descriptor, where: an elementary stream list is a list of elementary streams of the transport stream that are part of the operating point , the second descriptor has a hierarchical layer index value equal to a value of the second syntax element, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of streams elementary streams, shall be added to the list of elementary streams, wherein the elementary stream indicated by the second syntax element is one of a plurality of elementary streams, and an elementary stream indicated by a layer index embedded in the second descriptor, when not present in the list of elementary flows, shall be added to the list of elementary flows, wherein the elementary flow indicated by the layer index embedded in the second descriptor is one of the plurality of elementary flows, and in response to the determination (206) that the first element of syntax has a second value different from the first value, add (208, 308) the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, to the list of elementary streams, but do not add the elementary stream indicated by layer index embedded in the second descriptor to the list of elementary flows. 2. The method of claim 1, further comprising: receiving (300) a set of second descriptors in the transport stream, wherein the set of second descriptors includes the second descriptor, wherein each respective second descriptor descriptor among the set of second descriptors is one of: a hierarchy descriptor or a hierarchy extension descriptor; decoding (302) a third syntax element in the first descriptor, wherein the third syntax element indicates a number of elementary streams; decoding (304) a set of first syntax elements in the first descriptor; and decoding (306) a set of second syntax elements in the first descriptor, wherein: the number of first syntax elements in the set of first syntax elements and the number of second syntax elements in the set of second syntax elements are equal to a value of the third syntax element, the set of first syntax elements includes the first syntax element, the set of second syntax elements includes the second syntax element, for each respective first syntax element of the set of first syntax elements syntax: a respective second syntax element of the set of second syntax elements corresponds to the respective first syntax element, a respective second descriptor of the set of second descriptors has a hierarchy layer index value equal to a value of the respective second syntax element syntax, the first value of the respective first syntax element specifies that: an elementary flow indicated by the respective second syntax element, when not present in the list of elementary flows, shall be added to the list of elementary flows, where the elementary flow indicated by respective second syntax element is one of a plurality of elementary streams, and an elementary stream indicated by a layer index embedded in the respective second descriptor, when not present in the operating point's list of elementary streams, shall be added to the list of streams elementary streams, wherein the elementary stream indicated by the layer index embedded in the respective second descriptor is one of a plurality of elementary streams, and in response to determining that the respective first syntax element has the second value, adding (308) the respective elementary flow indicated by the respective second syntax element, when not present in the list of elementary flows, to the list of elementary flows, but do not add the respective elementary flow indicated by the layer index embedded in the respective second descriptor to the list of elementary flows. 3. Method according to claim 2, characterized by the fact that it further comprises: for each respective first syntax element among the set of first syntax elements: including, a set of one or more layer indices incorporated, in the respective second descriptor, and add, based on the respective first syntax element having the first value, in ascending order of value from among the set of embedded layer indices, elementary streams indicated by the set of embedded layer indices to the list of immediately elementary streams before the elementary flow indicated by the respective second syntax element. 4. The method of claim 1, further comprising: decoding (310) a set of syntax elements in the first descriptor, wherein each respective syntax element in the set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element among the set of syntax elements indicates whether the elementary stream corresponds to the respective syntax element among the set of syntax elements is a required layer, wherein a required layer is defined as a layer at an output operating point associated with a set of output layers, wherein the layer is an output layer among the set of output layers, or a reference layer of an output layer among the set of output layers; or decoding (312) a first set of syntax elements in the first descriptor, wherein each respective syntax element in the first set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element of the first set of syntax elements indicates whether the elementary stream corresponding to the respective syntax element of the first set of syntax elements is an output layer; and decoding (314) a second set of syntax elements in the first descriptor, wherein each respective syntax element in the second set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element of the second set of syntax elements indicates an index for determining a profile, a row or a level of the elementary flow that corresponds to the respective syntax element of the second set of syntax elements; or receiving (300) a set of second descriptors in the transport stream, wherein the set of second descriptors includes the second descriptor, wherein each respective second descriptor among the set of second descriptors is one of: a hierarchy descriptor or a hierarchy extension; and decoding (306) a set of second syntax elements in the first descriptor, wherein: the set of second syntax elements includes the second syntax element, each respective second syntax element among the set of second syntax elements indicates a value of hierarchy layer index present in a descriptor among the set of second descriptors, and the set of second syntax elements is restricted such that no two second syntax elements among the set of second syntax elements have the same value; or discard (316) data for layers that are not part of the operating point. 5. Device (14) for processing video data, characterized in that the device comprises: an input interface (28) configured to receive information including a transport stream that includes a first descriptor, a second descriptor, and a plurality of elementary streams, wherein the first descriptor is a descriptor for an operating point of the video data, wherein the second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor; and one or more processors (29, 30) configured to: decode a first syntax element and a second syntax element in the first descriptor, wherein: an elementary stream list is a list of elementary streams of the transport stream that are part from the point of operation, the second descriptor has a hierarchical layer index value equal to a value of the second syntax element, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary flows, shall be added to the list of elementary flows, wherein the elementary flow indicated by the second syntax element is one of the plurality of elementary flows, and an elementary flow indicated by a layer index embedded in the second descriptor, when not present in the list of elementary flows, shall be added to the list of elementary flows, wherein the elementary flow indicated by the layer index embedded in the second descriptor is one of the plurality of elementary flows, and in response to the determination that the first syntax element has a second value different from the first value, add the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, to the list of elementary streams, but do not add the elementary stream indicated by the embedded layer index in the second descriptor to the list of elementary flows. 6. The device (14) of claim 5, wherein the one or more processors (29, 30) are configured to: receive a set of second descriptors in the transport stream, wherein the set of second descriptors includes the second descriptor, wherein each respective second descriptor among the set of second descriptors is one of: a hierarchy descriptor or a hierarchy extension descriptor; decode a third syntax element in the first descriptor, wherein the third syntax element indicates a number of elementary streams; decode a set of first syntax elements in the first descriptor; and decode a set of second syntax elements into the first descriptor, wherein: the number of first syntax elements in the set of first syntax elements and the number of second syntax elements in the set of second syntax elements are equal to a value of the third syntax element, the set of first syntax elements includes the first syntax element, the set of second syntax elements includes the second syntax element, for each respective first syntax element among the set of first syntax elements: a respective second syntax element among the set of second syntax elements corresponds to the respective first syntax element, a respective second descriptor among the set of second descriptors has a hierarchy layer index value equal to a value of the respective second syntax element, the first value of the respective first syntax element specifies that: an elementary stream indicated by the respective second syntax element, when not present in the list of elementary streams, is to be added to the list of elementary streams, wherein the elementary stream indicated by the respective second syntax element is one of the plurality of elementary streams, and an elementary stream indicated by a layer index embodied in the respective second descriptor, when not present in the list of elementary streams of the operating point, must be added to the list of elementary streams, wherein the elementary stream indicated by the layer index embodied in the respective second descriptor is one of the plurality of elementary streams, and in response to determining that the respective first syntax element has the second value, the one or more processors are configured to add the respective elementary stream indicated by the respective second syntax element, when not present in the list of elementary streams, to the list of elementary streams, but not add the respective elementary stream indicated by the layer index embodied in the respective second descriptor to the list of elementary streams. 7. Device (14), according to claim 6, characterized by the fact that, for each respective first syntax element among the set of first syntax elements, a set of one or more embedded layer indices is included in the respective second descriptor, and the one or more processors (29, 30) are configured such that, based on the respective first syntax element having the first value, the one or more processors add, in ascending order of value, from the set of embedded layer indices, elementary streams indicated by the set of layer indices embedded in the list of elementary streams immediately preceding the elementary stream indicated by the respective second syntax element. 8. Device (14), according to claim 5, characterized by the fact that the one or more processors (29, 30) are configured to: decode a set of syntax elements in the first descriptor, wherein each respective syntax element syntax in the set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element among the set of syntax elements indicates whether the elementary stream corresponds to the respective syntax element among the set of syntax elements. syntax is a required layer, wherein a required layer is defined as a layer at an output operating point associated with a set of output layers, wherein the layer is an output layer among the set of output layers, or a reference layer of an output layer among the set of output layers; or decoding a first set of syntax elements in the first descriptor, wherein each respective syntax element in the first set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element of the first set of syntax elements indicates whether the elementary stream corresponding to the respective syntax element of the first set of syntax elements is an output layer; and decoding a second set of syntax elements in the first descriptor, wherein each respective syntax element in the second set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element of the second set of syntax elements indicates an index to determine a profile, a row, or a level of the elementary stream that corresponds to the respective syntax element of the second set of syntax elements; or receiving a set of second descriptors in the transport stream, wherein the set of second descriptors includes the second descriptor, wherein each respective second descriptor among the set of second descriptors is one of: a hierarchy descriptor or a file extension descriptor hierarchy; and decoding a set of second syntax elements in the first descriptor, wherein: the set of second syntax elements includes the second syntax element, wherein each respective second syntax element among the set of second syntax elements indicates a value of hierarchy layer index present in a descriptor among the set of second descriptors, and the set of second syntax elements is restricted such that no two second syntax elements among the set of second syntax elements have the same value; or discard data for layers that are not part of the operating point. 9. Device (14), according to any one of claims 5 to 8, characterized in that it additionally comprises a display device (32) for displaying decoded video data. 10. Device (14), according to any one of claims 5 to 9, characterized by the fact that the device (14) is a wireless communication device. 11. Device (12) for processing video data, characterized in that it comprises: one or more processors (20, 21) configured to: generate a first descriptor, wherein the first descriptor is a descriptor of an operating point of video data; generating a second descriptor, wherein the second descriptor is one of: a hierarchy descriptor or a hierarchy extension descriptor; wherein the one or more processors are configured such that, as part of generating the first descriptor, the one or more processors: include a first syntax element and a second syntax element in the first descriptor, wherein: a list of streams elementary stream is a list of elementary streams that are part of the operating point, the second descriptor has a hierarchical layer index value equal to a value of the second syntax element, a first value of the first syntax element specifies that: an elementary stream indicated by the second syntax element, when not present in the list of elementary flows, must be added to the list of elementary flows, and an elementary flow indicated by a layer index embedded in the second descriptor, when not present in the list of elementary flows, must be added to the list of elementary streams, and a second value of the first syntax element specifies that the elementary stream indicated by the second syntax element, when not present in the list of elementary streams, should be added to the list of elementary streams, but not the elementary flow indicated by the layer index embedded in the second descriptor; and including the first descriptor and the second descriptor in a transport stream; and an output interface (22) configured to output an encoded version of the video data. 12. Device (12), according to claim 11, characterized by the fact that: the one or more processors (20, 21) are configured to generate a set of second descriptors, wherein the set of second descriptors includes the second descriptor, wherein each respective second descriptor among the set of second descriptors is one of: a hierarchy descriptor or a hierarchy extension descriptor; the one or more processors are configured such that, as part of generating the first descriptor, the one or more processors: include a third syntax element in the first descriptor, wherein the third syntax element indicates a number of elementary streams; include a set of first syntax elements in the first descriptor; and include a set of second syntax elements in the first descriptor, wherein: the number of first syntax elements in the set of first syntax elements and the number of second syntax elements in the set of second syntax elements are equal to a value of the third syntax element, the set of first syntax elements includes the first syntax element, the set of second syntax elements includes the second syntax element, for each respective first syntax element among the set of first syntax elements: a respective second syntax element among the set of second syntax elements corresponds to the respective first syntax element, a respective second descriptor among the set of second descriptors has a hierarchy layer index value equal to a value of the respective second syntax element. syntax, the first value of the respective first syntax element specifies that: an elementary stream indicated by the respective second syntax element, when not present in the list of elementary streams, shall be added to the list of elementary streams, and an elementary stream indicated by a layer index embedded in the respective second descriptor, when not present in the operating point's list of elementary streams, shall be added to the list of elementary streams, and the second value of the respective first syntax element specifies that the respective elementary stream indicated by the respective second syntax element, when not present in the list of elementary flows, must be added to the list of elementary flows, but not the respective elementary flow indicated by the layer index embedded in the respective second descriptor. 13. Device (12), according to claim 12, characterized by the fact that the one or more processors (20, 21) are configured so that, as part of the generation of the first descriptor, the one or more processors: include a set of syntax elements in the first descriptor, wherein each respective syntax element in the set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element among the set of syntax elements indicates if the elementary flow corresponding to the respective syntax element among the set of syntax elements is a required layer, wherein a required layer is defined as a layer at an output operating point associated with a set of output layers, then that the layer is an output layer among the set of output layers, or a reference layer of an output layer among the set of output layers. 14. Device (12), according to claim 11, characterized by the fact that: the one or more processors (20, 21) are configured so that, as part of the generation of the first descriptor, the one or more processors: include a first set of syntax elements in the first descriptor, wherein each respective syntax element in the first set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element of the first set of elements syntax layer indicates whether the elementary stream corresponding to the respective syntax element of the first set of syntax elements is an output layer; and include a second set of syntax elements in the first descriptor, wherein each respective syntax element in the second set of syntax elements corresponds to a respective elementary stream in the list of elementary streams, wherein each respective syntax element of the second set of syntax elements indicates an index to determine a profile, a row, or a level of the elementary stream that corresponds to the respective syntax element of the second set of syntax elements; or the one or more processors (20, 21) are further configured to include a set of second descriptors in the transport stream, wherein the set of second descriptors includes the second descriptor, wherein each respective second descriptor among the set of second descriptors is one of: a hierarchical descriptor or a hierarchical extension descriptor, and the one or more processors are configured so that, as part of generating the first descriptor, the one or more processors: include a set of second syntax elements in the first descriptor, where: the set of second syntax elements includes the second syntax element, each respective second syntax element among the set of second syntax elements indicates a hierarchy layer index value present in a descriptor among the set of second descriptors, and the set of second syntax elements is restricted such that no two second syntax elements among the set of second syntax elements have the same value. 15. Device according to any one of claims 11 to 14, characterized in that the device (14) is a wireless communication device.