JP7530751B2 - Multiplex signal conversion device, program thereof, and receiver - Google Patents

Description

The present invention relates to a multiplex signal conversion device, a program for the device, and a receiver.

MMT (MPEG Media Transport) has been established as an international standard for a new IP (Internet Protocol)-based media transport method to replace MPEG-2 TS (Transport Stream) used in conventional digital broadcasting (Non-Patent Document 1). In addition, the method of using MMT in digital broadcasting services in Japan has been standardized (Non-Patent Document 2), and new 4K8K satellite broadcasting that uses MMT was launched in December 2018.

The application layer packet format (packet header data structure) defined by MMT is called MMTP (MMT Protocol). MMTP packets are transmitted unidirectionally over broadcast or communication transmission paths as the payload of UDP (User Diagram Protocol)/IP packets, as shown in Figure 8 (a).

In MMT, the processing unit of video and audio codec is called MPU (Media Processing Unit). The first data of an MPU needs to be a random access point that can be processed without relying on previously transmitted data. In Non-Patent Document 2, which specifies the use of MMT in broadcasting, a GOP (Group Of Picture) starting with an intra frame (a frame that performs intra-frame compression) of video coding is treated as an MPU. Note that the term GOP is not used in the HEVC standard, which is used as an example of a video coding method, but following conventional methods such as MPEG-2 Video, a set of frames starting with an intra frame is sometimes called a GOP for convenience.

As shown in FIG. 8(b), in broadcasting services, GOPs are configured with a period of about 0.5 seconds to ensure random access when changing the receiving channel. As a specific example, in HEVC, 32 frames may be used as a GOP. In AAC (Advanced Audio Coding), which is used as an example of an audio coding method, for example, for audio samples in which sound pressure is sampled at a sampling frequency of 48 kHz, a data block in which encoding processing is performed independently for every 1024 samples is treated as an AU (Access Unit). Generally, the beginning of each AU is a random access point, but since there may be multiple random access points in an MPU, a set of one or more audio AUs can be treated as an MPU. An MPU transmitted by MMTP can be uniquely identified by an MPU sequence number. The MPU sequence number is written in the MMTP payload header of an MMTP packet that stores the corresponding MPU as a payload.

According to Non-Patent Document 1, MPU is originally defined based on the ISOBMFF (ISO Base Media File Format). Non-Patent Document 1 also defines a method of transmitting the metadata portion of ISOBMFF as MPU metadata and movie fragment metadata by MMTP. However, in the broadcasting applications defined in Non-Patent Document 2, the generation and transmission of movie fragment metadata is omitted in order to reduce processing latency, and the NAL (Network Abstraction Layer) units generated by the HEVC encoder are multiplexed as media fragment units directly into MMTP/UDP/IP packets and transmitted.

The basic data structure of ISOBMFF is specified in Non-Patent Document 3, and since it is part of the MPEG-4 standard, it is commonly called MP4 (.mp4). The Box format, which is a metadata description method defined in ISOBMFF, is extensible, and new metadata data structures can be added and more detailed operation methods can be specified according to the requirements of the application. For example, Non-Patent Document 4 specifies detailed operation methods and data structures of ISOBMFF metadata in MPEG-DASH (Dynamic Adaptive Streaming over HTTP), a video streaming distribution method using HTTP (Hypertext Transfer Protocol)/TCP (Transmission Control Protocol) communication. In MPEG-DASH, video segments lasting several to several tens of seconds are made public on a WEB server, and the playback terminal continuously downloads the segments according to the manifest file to play the video.

HLS (HTTP Live Streaming), defined in Non-Patent Document 5, is known as a video streaming delivery method that uses HTTP/TCP like MPEG-DASH. Currently, HLS is widely used alongside MPEG-DASH. HLS initially used MPEG-2 TS format segments, but was revised to allow the use of ISOBMFF format segments, the same as MPEG-DASH. As a result, MPEG-DASH and HLS use the same segments, even though they use different manifest files during playback, making it possible to streamline video delivery using CDNs (Contents Delivery Networks), etc.

An ISOBMFF-based segment format that can be used in common with MPEG-DASH and HLS is specified as CMAF in Non-Patent Document 6. Note that CMAF is a newer standard than Non-Patent Documents 1 and 2.

In addition to standardizing segment formats, CMAF defines a chunk structure that further subdivides the segment structure as a technology for reducing delays in video streaming delivery. While a segment is generally several to tens of seconds long, a chunk is video data that is shorter than the duration of a few frames. In general file-based transfers using HTTP, a file is transferred only after an entire segment is completed, so in principle, a video delay of the duration of the segment (several to tens of seconds) is unavoidable. In reality, the receiver also stores several segments in a buffer to stabilize playback, so the total video delay can be several tens of seconds to several minutes. On the other hand, when the chunk structure (several frames) of CMAF is transmitted to the receiver using Chunked Transfer, an extension technology of HTTP, video streaming delivery with a delay of a few seconds is possible by transmitting in chunks of several frames without waiting for the entire segment to be completed. When the ISOBMFF standard was first created, the generation of metadata in units of a few frames was not taken into consideration, but the functionality has been expanded to meet the demands of new applications that have arisen with the development of communication technology.

Even in video transmission by MMT, if metadata is generated after the entire MPU corresponding to a segment is encoded and transmitted as MPU metadata and movie fragment metadata, delays similar to those when converting segments into files using MPEG-DASH or the like are unavoidable in principle. Therefore, in Non-Patent Document 2, in order to reduce delays in video transmission in broadcasting, the generation of metadata in ISOBMFF format and the transmission as MPU metadata and movie fragment metadata are omitted. However, if the transmission of movie fragment metadata is omitted, there is a problem in that the DTS-PTS differential information (dts_pts_offset) cannot be transmitted to the receiver. This DTS-PTS differential information is information indicating the difference value between the DTS (Decoding Timestamp) that indicates the decoding timing of a video frame and the PTS (Presentation Timestamp) that indicates the presentation timing of a video frame, which is associated with the inter-frame reference structure of video encoding. Therefore, in Non-Patent Document 2, an "extended MPU timestamp descriptor" is defined that describes this DTS-PTS difference information separately, and it is specified that it is transmitted as a descriptor in the MP table (MMT Package Table), which is a control message. Here, a descriptor is a general name for a data structure for extending a control message in order to multiplex and transmit various auxiliary information in the control message. For example, in MMT, the structure of the MP table, which is a control message, is specified in Non-Patent Document 1, while various descriptors that extend it are specified in Non-Patent Document 2, and standardization organizations and service providers in each country can independently define additional descriptors. The "extended MPU timestamp descriptor" is a structure that lists the DTS-PTS difference information of all frames that make up the MPU, and is transmitted separately from the "MPU timestamp descriptor" that indicates the PTS of the first frame of the MPU. The "MPU timestamp descriptor" is specified separately in Non-Patent Document 1.

Furthermore, Non-Patent Document 6 specifies a more detailed method of operating MMT for an actual service (advanced wideband digital satellite broadcasting). Non-Patent Document 6 requires that the "extended MPU timestamp descriptor" be transmitted at an earlier timing than the corresponding MPU. In this case, as shown in FIG. 8(b), the PTS written in the "MPU timestamp descriptor" is determined at the beginning of the MPU (GOP) (symbol α). On the other hand, the "extended MPU timestamp descriptor" can only be generated after video coding of the entire MPU has been completed (symbol β). This has created a need to delay the transmission of the MPU's video code data by more than the MPU length.

In order to avoid such video delay, it is possible to use ISOBMFF metadata with a chunk structure newly defined in CMAF in MMT as shown in FIG. 9(a). Since the CMAF chunk has a structure based on ISOBMFF, multiplexing in MMTP is possible according to the provisions of Non-Patent Document 1. In the ISOBMFF metadata of the CMAF chunk, metadata in a Box format that indicates the difference value between the presentation timing and the decoding timing of each frame in the chunk is defined, so that metadata can be generated with a chunk length delay. When the Box format metadata is applied to MMT, it is possible to generate metadata with a chunk length delay and transmit an MPU as shown in FIG. 9(b). This makes it possible to avoid the MPU length delay associated with the generation of metadata in video transmission using MMT. In FIG. 9(b), as an example, a set of 8 frames obtained by dividing a 32-frame GOP into 4 is configured as a chunk. In addition, by describing DTS-PTS differential information in a box-format metadata structure defined in ISOBMFF and CMAF and transmitting it as MMTP movie fragment metadata, it is possible to transmit the necessary metadata to the decoder without using the "extended MPU timestamp descriptor." Specifically, in CMAF, DTS-PTS differential information can be described for each frame in a chunk using "sample_composition_time_offset" in "TrackFragmentRunBox ('trun')."

Below, the video coding parameter set is explained. ISOBMFF uses FourCC (Four Character Code), which is defined by four alphabetical letters, to identify codecs. For example, HEVC specifies two types, "hev1" and "hvc1". Here, "hev1" indicates a format in which the video coding parameter sets specified in HEVC, VPS (Video Parameter Set), SPS (Sequence Parameter Set), and PPS (Picture Parameter Set), are included in the media fragment unit. Also, "hvc1" indicates a format in which the parameter sets are included in the MPU metadata.

In other words, an MMTP packet in which video coding data, etc. are multiplexed by applying CMAF includes a parameter set in the MPU metadata (in the case of "hvc1") or in the media fragment unit (in the case of "hev1"), as shown in FIG. 9(b). Hereinafter, an MMTP packet in which video coding data, etc. are multiplexed by applying CMAF may be abbreviated as "CMAF-applied MMT." In other words, CMAF-applied MMT may be "hvc1" or "hev1." Note that, although it is a special case, it is technically possible to transmit each parameter set in both the media fragment unit and the MPU metadata.

On the other hand, in an MMT in which video coding data, etc. are multiplexed without applying CMAF, as shown in FIG. 8(b), MPU metadata is not transmitted, and the parameter set is transmitted in a format including the media fragment unit, so it only supports "hev1". Hereinafter, an MMTP packet in which video coding data, etc. are multiplexed without applying CMAF may be abbreviated as "MMT without applying CMAF".

Note that MPU metadata is generally called movie metadata because it includes the "MovieBox ('moov')" defined in ISOBMFF. Movie fragment metadata is also generally called movie fragment metadata because it includes the "MovieFragmentBox ('moof')" defined in ISOBMFF. Media fragment units are generally called media data because they include the "MediaDataBox ('mdat')" defined in ISOBMFF. In CMAF, a processing unit that starts with a random access point such as an MPU is called a fragment.

“High efficiency coding and media delivery in heterogeneous environments: MPEG media transport”, ISO/IEC 23008-1 "Media Transport Method Using MMT in Digital Broadcasting", ARIB STD-B60 “ISO/IEC base media file format”, ISO/IEC 14496-12 “Dynamic adaptive streaming over HTTP (DASH)-Part 1:Media presentation description and segment formats”, ISO/IEC 23009-1 “Common media application format (CMAF) for segmented media”, ISO/IEC 23000-19 "Advanced Wideband Satellite Digital Broadcasting Operational Standards (Volume 3)", ARIB TR-B39

If the above-mentioned CMAF-applied MMT is input to a receiver that does not support CMAF, video and audio may not be played back properly, or the program may terminate abnormally due to a processing error. Similarly, if a CMAF-unapplied MMT is input to a receiver that supports CMAF, video and audio may not be played back properly, or the program may terminate abnormally due to a processing error.

The present invention aims to provide a multiplex signal conversion device and associated program, as well as a receiver, that allows the receiver to play video and audio normally, regardless of whether CMAF is appropriate.

To solve the above problems, the multiplexed signal conversion device of the present invention is a multiplexed signal conversion device that converts a CMAF-applied multiplexed signal, which is a multiplexed signal to which CMAF is applied, into a CMAF-non-applied multiplexed signal, which is a multiplexed signal to which CMAF is not applied, and is configured to include a separation unit, a descriptor conversion unit, a descriptor addition unit, an output unit, and a mixing unit.

According to this configuration, the separation unit separates the movie metadata, the movie fragment metadata, the control message, and the media data from the CMAF applied multiplexed signal.
The descriptor conversion unit converts the DTS-PTS difference information of the movie fragment metadata into a descriptor.
The descriptor adder adds the descriptor to the control message.
The output unit outputs the media data in fragment units in accordance with the output timing of the control message.
The mixer mixes the control message from the descriptor adding unit with the media data from the output unit, and outputs the result as a CMAF non-applied multiplex signal.

In this way, the multiplex signal conversion device can convert a CMAF-applied multiplex signal into a CMAF-non-applied multiplex signal, allowing the receiver to play video and audio normally regardless of whether CMAF is appropriate.

In order to solve the above problem, the multiplexed signal conversion device according to the present invention is a multiplexed signal conversion device that converts a CMAF non-applied multiplexed signal, which is a multiplexed signal to which CMAF is not applied, into a CMAF-applied multiplexed signal, which is a multiplexed signal to which CMAF is applied, and is configured to include a separation unit, a descriptor extraction/deletion unit, a conversion unit, an output unit, and a mixing unit.

According to this configuration, the demultiplexer demultiplexes the control message and the media data from the CMAF non-applied multiplex signal.
The descriptor extraction and deletion unit extracts a descriptor including DTS-PTS difference information from the control message, and deletes the descriptor from the control message.
The conversion unit converts the DTS-PTS difference information of the descriptor into movie fragment metadata.
The output unit outputs the media data in chunk units in accordance with the output timing of the movie fragment metadata.
The mixer mixes the control message from the descriptor extractor/deleter, the movie fragment metadata from the converter, and the media data from the output unit, and outputs the result as a CMAF-applied multiplexed signal.

In this way, the multiplex signal conversion device can convert a CMAF-non-applied multiplex signal into a CMAF-applied multiplex signal, allowing the receiver to play video and audio normally regardless of whether CMAF is appropriate.

The present invention can also be realized by a program for causing a computer to function as the multiplex signal conversion device described above.
The present invention can also be realized in a receiver including the above-mentioned multiple signal conversion device.

According to the present invention, the receiver can play video and audio normally regardless of whether CMAF is appropriate.

FIG. 1 is a schematic configuration diagram of a broadcasting system according to each embodiment. FIG. 2 is a block diagram showing the configuration of an MMT conversion device according to the first embodiment. 4 is a flowchart showing the operation of the MMT conversion device according to the first embodiment. FIG. 13 is a block diagram showing the configuration of an MMT conversion device according to a first modified example. FIG. 11 is a block diagram showing the configuration of an MMT conversion device according to a second embodiment. 13 is a flowchart showing the operation of an MMT conversion device according to the second embodiment. FIG. 11 is a block diagram showing the configuration of an MMT conversion device according to a second modified example. As a conventional technique, (a) is an explanatory diagram explaining multiplexing of MMT not applying CMAF, and (b) is an explanatory diagram explaining the data structure of MMT not applying CMAF. As a conventional technique, (a) is an explanatory diagram explaining multiplexing of CMAF-applied MMT, and (b) is an explanatory diagram explaining the data structure of CMAF-applied MMT.

Each embodiment of the present invention will be described below with reference to the drawings. However, the embodiments described below are intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, in each embodiment, the same means are given the same reference numerals, and the description may be omitted.

First Embodiment
[Broadcasting system overview]
With reference to FIG. 1, an overview of a broadcasting system 100 according to a first embodiment will be described.
1, the broadcasting system 100 performs digital broadcasting and includes an encoding device 2, a transmitting device 3, and a receiver 4. The receiver 4 also includes an MMT conversion device (multiplexed signal conversion device) 1, which will be described later.

The encoding device 2 encodes the video of a broadcast program using a predetermined video encoding method and outputs the encoded video to the transmission device 3. In this embodiment, the video encoding method is assumed to be HEVC.

The transmission device 3 multiplexes the video and audio of a broadcast program using a predetermined multiplexing method and transmits it to the receiver 4. In this embodiment, the multiplexing method is MMT. In other words, the transmission device 3 multiplexes the video and audio input from the encoding device 2 and transmits it to the receiver 4 as an MMTP packet sequence.

The receiver 4 receives and demultiplexes the MMTP packet sequence sent by the sending device 3, and decodes and plays back the video and audio of the broadcast program. For example, the receiver 4 may be a general television, a smartphone, or a tablet. Note that in FIG. 1, only one receiver 4 is shown to make the drawing easier to understand, but typically there are multiple receivers 4.

Here, a sending device 3 compatible with CMAF may send a CMAF-adapted MMT (CMAF-applied multiplex signal) to a receiver 4 that does not support CMAF. In this case, the receiver 4 converts the CMAF-adapted MMT into a CMAF-non-adapted MMT (CMAF-non-applied multiplex signal) using a built-in MMT conversion device 1.
In this embodiment, the CMAF-adaptive MMT that transmits HEVC video signals as assets corresponds to "hvc1", and the parameter set is included in the MPU metadata.

[Configuration of MMT conversion device]
The configuration of the MMT conversion device 1 will be described with reference to FIG.
The MMT conversion device 1 converts the CMAF-adapted MMT (hvc1) in Fig. 9(b) into the CMAF-non-adapted MMT in Fig. 8(b). As shown in Fig. 2, the MMT conversion device 1 includes a packet filter (separation unit) 10, a message buffer 11, a descriptor conversion unit 12, a descriptor addition unit 13, a parameter set extraction unit 14, a parameter set addition unit 15, an MPU buffer (output unit) 16, and a packet mixer (mixer) 17.

The packet filter 10 separates movie fragment metadata, MPU metadata (movie metadata), control messages (MP tables), media fragment units (media data), and other packets from the CMAF-applied MMT.

As shown in FIG. 9B, the packet filter 10 refers to the PID and fragment type (fragment_type) of the CMAF-applied MMT (MMTP packet) and separates movie fragment metadata, etc. Specifically, the packet filter 10 separates MMTP packets with PID=0 as control messages (MP tables), MMTP packets with PID=X and fragment type=0 as MPU metadata, MMTP packets with PID=X and fragment type=1 as movie fragment metadata, and MMTP packets with PID=X and fragment type=2 as media fragment units from the CMAF-applied MMT. The packet filter 10 also separates data other than the movie fragment metadata, etc. (e.g., control messages other than the MP table) from the CMAF-applied MMT as other packets.

Here, the packet filter 10 can identify the PID (=X) that transmits the asset to be converted by referencing the asset location information in the MP table. Note that the control message with PID=0, which is the entry point, includes a package list table, and the MP table may be transmitted by another PID referenced from the package list table. In this case, the packet filter 10 can identify the PID that transmits the control message (MP table) by referencing the package list table, and separate the control message (MP table).

Then, the packet filter 10 outputs the control message (MP table) to the message buffer 11, outputs the movie fragment metadata to the descriptor conversion unit 12, and outputs the MPU metadata to the parameter set extraction unit 14. Furthermore, the packet filter 10 outputs the media fragment unit to the parameter set addition unit 15, and outputs the other packets to the packet mixing unit 17.

The message buffer 11 is a buffer that accumulates the control message (MP table) input from the packet filter 10. In addition, the message buffer 11 outputs the control message (MP table) to the descriptor addition unit 13 according to an output instruction from the descriptor conversion unit 12.

The descriptor conversion unit 12 converts the DTS-PTS differential information of the movie fragment metadata input from the packet filter 10 into a descriptor for multiplexing and transmitting the DTS-PTS differential information in a control message. In this embodiment, the descriptor is an extended MPU timestamp descriptor. Specifically, the descriptor conversion unit 12 analyzes the movie fragment metadata and converts the DTS-PTS differential information into the format of an extended MPU timestamp descriptor. Then, when the descriptor conversion unit 12 completes the conversion of the DTS-PTS differential information for one MPU, it instructs the message buffer 11 to output and outputs the extended MPU timestamp descriptor to the descriptor addition unit 13. The output instruction from the descriptor conversion unit 12 to the message buffer 11, for example, specifies an MPU sequence number and causes the control message (MP table) including a descriptor (for example, an MPU timestamp descriptor) corresponding to that MPU to be output.

The descriptor adding unit 13 adds the extended MPU timestamp descriptor input from the descriptor conversion unit 12 to the control message (MP table) input from the message buffer 11. In other words, this control message has an MP table to which the extended MPU timestamp descriptor has been added. The descriptor adding unit 13 then outputs this control message (MP table) to the packet mixing unit 17. Furthermore, after outputting the control message (MP table) to the packet mixing unit 17, the descriptor adding unit 13 instructs the MPU buffer 16 to output the MPU corresponding to the extended MPU timestamp descriptor.

The parameter set extraction unit 14 extracts a video encoding parameter set from the MPU metadata input from the packet filter 10. Specifically, the parameter set extraction unit 14 extracts an HEVC parameter set from the Box-format metadata of the MPU metadata, and outputs the extracted parameter set to the parameter set addition unit 15.

The parameter set addition unit 15 adds the parameter set input from the parameter set extraction unit 14 to the media fragment unit input from the packet filter 10. Specifically, the parameter set addition unit 15 adds the HEVC parameter set to the beginning of the media fragment unit of the first frame of the MPU. Then, the parameter set addition unit 15 outputs this media fragment unit to the MPU buffer 16.

The MPU buffer 16 is a buffer that accumulates the media fragment units input from the parameter set adding unit 15. The MPU buffer 16 also outputs the media fragment units to which the parameter sets have been added in MPU (fragment) units according to the output timing of the control message (MP table). In other words, the MPU buffer 16 outputs the media fragment units of the MPU specified by the output instruction from the descriptor adding unit 13 to the packet mixing unit 17. The output instruction from the descriptor adding unit 13 to the MPU buffer 16 is, for example, to specify an MPU by its MPU sequence number and output it.

The packet mixing unit 17 mixes the control message (MP table) input from the descriptor adding unit 13, the media fragment unit input from the MPU buffer 16, and other packets input from the packet filter 10, and outputs the result as a CMAF-non-applied MMT.

Here, in the processing of the message buffer 11, the descriptor conversion unit 12, the descriptor addition unit 13, the parameter set extraction unit 14, the parameter set addition unit 15, and the MPU buffer 16, the format of the MMTP packet may be maintained, or the processing target data, which is the payload of the MMTP packet, may be processed in a format in which it is extracted once. In the former case, the packet mixing unit 17 inputs a plurality of MMTP packet strings and outputs them as a single MMTP packet string in which they are mixed. In the latter case, the packet mixing unit 17 generates MMTP packets containing a control message (MP table) and a media fragment unit as a payload, mixes them with the MMTP packet string input as other packets, and outputs them as a single MMTP packet string. Furthermore, the packet mixing unit 17 may rewrite the header part, such as correcting the continuity of the packet sequence number for the MMTP packet string to be output, as necessary.

[Operation of MMT conversion device]
The operation of the MMT conversion device 1 will be described with reference to FIG.
3, in step S1, the packet filter 10 separates MPU metadata, movie fragment metadata, control messages (MP tables), media fragment units, and other packets from the CMAF-applied MMT. The message buffer 11 stores the control messages (MP tables) separated by the packet filter 10.

In step S2, the parameter set extraction unit 14 extracts a parameter set from the MPU metadata.
In step S3, the parameter set adding unit 15 adds the parameter set to the media fragment unit. The media fragment unit is buffered in the MPU buffer 16 until an output instruction is received from the descriptor adding unit 13.

In step S4, the descriptor conversion unit 12 converts the DTS-PTS differential information of the movie fragment metadata into an extended MPU timestamp descriptor, and when conversion of the DTS-PTS differential information for one MPU is complete, it issues an output instruction to the message buffer 11. Then, in accordance with the output instruction from the descriptor conversion unit 12, the message buffer 11 outputs a control message (MP table) to the descriptor addition unit 13.

In step S5, the descriptor adding unit 13 adds the extended MPU timestamp descriptor to the control message (MP table) and outputs the control message (MP table). Furthermore, the descriptor adding unit 13 instructs the MPU buffer 16 to output the MPU corresponding to the extended MPU timestamp descriptor.

In step S 6 , the MPU buffer 16 outputs the media fragment unit of the MPU specified by the output instruction from the descriptor adding unit 13 to the packet mixing unit 17 .
In step S7, the packet mixer 17 mixes the control message (MP table), the media fragment unit, and other packets, and outputs the result as a CMAF-non-applied MMT.

The processing of steps S1 to S7 does not have to be performed sequentially in the order shown in FIG. 3. Depending on the type and order of input packets, the processing order of each step may be changed, or each step may be performed simultaneously in parallel.

[Action and Effects]
As described above, the MMT conversion device 1 converts a CMAF-compliant MMT into a CMAF-non-compliant MMT, so that the receiver 4 can normally play back video and audio regardless of the suitability of CMAF. In this way, even if the receiver 4 does not support ISOBMFF metadata with a chunk structure defined by CMAF, the MMT conversion device 1 can normally play back video and audio.

(Variation 1)
With reference to FIG. 4, an MMT conversion device 1B according to the first modification will be described with respect to differences from the first embodiment.
In the first modification, the CMAF-adaptive MMT corresponds to "hev1", and the parameter set is included in the media fragment unit. In other words, since the parameter set is originally included in the input media fragment unit and the MMT conversion device 1B can output the parameter set as is, it is not necessary to extract the parameter set from the MPU metadata and add it to the media fragment unit.

The MMT conversion device 1B converts the CMAF-adaptive MMT (hev1) of FIG. 9(b) into the CMAF-non-adaptive MMT of FIG. 8(b). As shown in FIG. 4, the MMT conversion device 1B includes a packet filter (separation unit) 10B, a message buffer 11, a descriptor conversion unit 12, a descriptor addition unit 13, an MPU buffer (output unit) 16B, and a packet mixer (mixer) 17.

The packet filter 10B outputs the movie fragment metadata separated from the CMAF-applied MMT to the descriptor conversion unit 12. The packet filter 10B also outputs the media fragment unit separated from the CMAF-applied MMT to the MPU buffer 16B. Note that when MPU metadata is input, the packet filter 10B discards the MPU metadata and does not output it. Other than this, the packet filter 10B is the same as in the first embodiment, so a description thereof will be omitted.

The MPU buffer 16B outputs the media fragment units input from the packet filter 10B in MPU units according to the output timing of the control message (MP table). Other than this, the MPU buffer 16B is the same as in the first embodiment, so a description is omitted.

[Action and Effects]
As described above, even when the CMAF-adaptive MMT corresponds to "hev1", the MMT conversion device 1B converts the CMAF-adaptive MMT to a CMAF-non-adaptive MMT as in the first embodiment, so that the receiver 4 can play video and audio normally regardless of the suitability of the CMAF.

Second Embodiment
[Broadcasting system overview]
The outline of a broadcasting system 100C according to the second embodiment will be described with reference to FIG.
As shown in FIG. 1, a broadcasting system 100C performs digital broadcasting, and includes an encoding device 2, a transmitting device 3C, and a receiver 4C.

In this embodiment, a sending device 3C that does not support CMAF sends a CMAF-non-compliant MMT to a receiver 4C that supports CMAF. Therefore, the receiver 4C converts the CMAF-non-compliant MMT to a CMAF-compliant MMT using an internal MMT conversion device (multiplexed signal conversion device) 5.

[Configuration of MMT conversion device]
The configuration of the MMT conversion device 5 will be described with reference to FIG.
The MMT conversion device 5 converts the CMAF non-adaptive MMT of Fig. 8(b) into the CMAF adapted MMT (hvc1) of Fig. 9(b). As shown in Fig. 5, the MMT conversion device 5 includes a packet filter (separation unit) 50, a descriptor extraction/deletion unit 51, a parameter set extraction/deletion unit 52, a metadata conversion unit (conversion unit) 53, an MPU buffer (output unit) 54, and a packet mixer (mixer) 55.

The packet filter 50 separates control messages (MP tables), media fragment units (media data), and other packets from the CMAF-non-applied MMT.

As shown in FIG. 8(b), the packet filter 50 refers to the PID of the CMAF-non-applied MMT (MMTP packet) and separates media fragment units, etc. Specifically, the packet filter 50 separates MMTP packets with PID=0 as control messages (MP tables) and MMTP packets with PID=X as media fragment units from the CMAF-non-applied MMT. The packet filter 50 also separates data other than the control messages (MP tables) and media fragment units as other packets from the CMAF-non-applied MMT.

Here, the packet filter 50 can identify the PID (=X) that transmits the asset to be converted by referencing the asset location information in the MP table. Note that the control message with PID=0, which is the entry point, includes a package list table, and the MP table may be transmitted by another PID referenced from the package list table. In this case, the packet filter 50 can identify the PID that transmits the control message (MP table) by referencing the package list table, and separate the control message (MP table).

The packet filter 50 also outputs the control message (MP table) to the descriptor extraction and deletion unit 51, outputs the media fragment unit to the parameter set extraction and deletion unit 52, and outputs other packets to the packet mixing unit 55.

The descriptor extraction/deletion unit 51 extracts the extended MPU timestamp descriptor from the control message (MP table) input from the packet filter 50, and deletes the extended MPU timestamp descriptor from the control message (MP table). The descriptor extraction/deletion unit 51 then outputs the extended MPU timestamp descriptor extracted from the control message (MP table) to the metadata conversion unit 53. Furthermore, the descriptor extraction/deletion unit 51 outputs the control message (MP table) from which the extended MPU timestamp descriptor has been deleted to the packet mixing unit 55.

The parameter set extraction/deletion unit 52 extracts a video encoding parameter set from the media fragment unit input by the packet filter 50, and deletes the parameter set from the media fragment unit. Specifically, the parameter set extraction/deletion unit 52 extracts a HEVC parameter set from the media fragment unit of the first frame of the MPU, and outputs the extracted parameter set to the metadata conversion unit 53. Furthermore, the parameter set extraction/deletion unit 52 outputs the media fragment unit from which the parameter set has been deleted to the MPU buffer 54.

The metadata conversion unit 53 converts the DTS-PTS difference information of the extended MPU timestamp descriptor input from the descriptor extraction and deletion unit 51 into movie fragment metadata. Specifically, the metadata conversion unit 53 analyzes the extended MPU timestamp descriptor and converts the DTS-PTS difference information into metadata in the Box format defined by ISOBMFF and CMAF.

The metadata conversion unit 53 also generates MPU metadata (movie metadata) including the parameter set input from the parameter set extraction/deletion unit 52. Specifically, the metadata conversion unit 53 generates metadata in a Box format defined by ISOBMFF and CMAF from the HEVC parameter set, and outputs it to the packet mixing unit 55 as MPU metadata. Note that the MPU metadata is output only once at the beginning of the MPU.

Then, when the metadata conversion unit 53 has completed the conversion of the DTS-PTS difference information for one chunk, it outputs the movie fragment metadata to the packet mixing unit 55 and issues an output instruction to the MPU buffer 54. This output instruction specifies the chunk that the MPU buffer 54 should output, synchronized with the output timing of the movie fragment metadata.

The MPU buffer 54 is a buffer that accumulates the media fragment units input from the parameter set extraction/deletion unit 52. The MPU buffer 54 also outputs the media fragment units in chunk units according to the output timing of the movie fragment metadata. In other words, the MPU buffer 54 outputs the media fragment units corresponding to the chunks specified in the output instruction from the metadata conversion unit 53 to the packet mixing unit 55. The output instruction from the metadata conversion unit 53 to the MPU buffer 54 specifies and outputs a chunk, for example, based on the MPU sequence number and the number of the chunk within that MPU.

The packet mixing unit 55 mixes the control message (MP table) input from the descriptor extraction/deletion unit 51, the MPU metadata and movie fragment metadata input from the metadata conversion unit 53, the media fragment unit input from the MPU buffer 54, and other packets input from the packet filter 50, and outputs the result as a CMAF-applied MMT.

Here, in the processes of the descriptor extraction/deletion unit 51, the parameter set extraction/deletion unit 52, and the MPU buffer 54, the format of the MMTP packet may be maintained and the MMTP packet may be generated by the metadata conversion unit 53, or the data to be processed, which is the payload of the MMTP packet, may be extracted and processed in a format. In the former case, the packet mixing unit 55 inputs a plurality of MMTP packet strings and outputs them as a single MMTP packet string by mixing them. In the latter case, the packet mixing unit 55 generates MMTP packets including a control message (MP table), MPU metadata, movie fragment metadata, and media fragment units as payloads, mixes them with the MMTP packet string input as other packets, and outputs them as a single MMTP packet string. Furthermore, the packet mixing unit 55 may rewrite the header part, for example, by correcting the continuity of the packet sequence number for the MMTP packet string to be output, as necessary.

[Operation of MMT conversion device]
The operation of the MMT conversion device 5 will be described with reference to FIG.
As shown in FIG. 6, in step S10, the packet filter 50 separates control messages (MP tables), media fragment units, and other packets from the CMAF-non-applied MMT.

In step S11, the parameter set extraction/deletion unit 52 extracts a video encoding parameter set from the media fragment unit and deletes the parameter set from the media fragment unit.

In step S12, the descriptor extraction/deletion unit 51 extracts the extended MPU timestamp descriptor from the control message (MP table) and deletes the extended MPU timestamp descriptor from the control message (MP table).

In step S13, the metadata conversion unit 53 generates MPU metadata including the extracted parameter set.
In step S14, the metadata conversion unit 53 generates movie fragment metadata by converting the DTS-PTS difference information of the extended MPU time stamp descriptor.

In step S15, the MPU buffer 54 outputs the media fragment unit in chunk units in accordance with the output timing of the movie fragment metadata.
In step S16, the packet mixing unit 55 mixes the control message, MPU metadata, movie fragment metadata, media fragment unit, and other packets, and outputs the result as a CMAF-applied MMT.

The processing of steps S10 to S16 does not have to be performed sequentially in the order shown in FIG. 6. Depending on the type and order of the input packets, the processing order of each step may be changed, or each step may be performed simultaneously in parallel.

[Action and Effects]
As described above, the MMT conversion device 5 converts a CMAF non-compliant MMT into a CMAF compliant MMT, so the receiver 4C can play back video and audio normally regardless of whether the CMAF is appropriate. In this way, even if the receiver 4C only supports ISOBMFF metadata with a chunk structure defined by CMAF, the MMT conversion device 5 can play back video and audio normally.

(Variation 2)
With reference to FIG. 7, an MMT conversion device 5B according to the second modification will be described with respect to differences from the second embodiment.
In the second modification, the CMAF-adapted MMT corresponds to "hev1", and the parameter set is included in the media fragment unit. In other words, the MMT conversion device 5B does not need to extract and delete the parameter set from the media fragment unit because the parameter set is originally included in the input media fragment unit and can simply output the input media fragment unit as is.

The MMT conversion device 5B converts the CMAF non-adaptive MMT of FIG. 8(b) into the CMAF adapted MMT (hev1) of FIG. 9(b). As shown in FIG. 7, the MMT conversion device 5B includes a packet filter (separation unit) 50B, a descriptor extraction/deletion unit 51, a metadata conversion unit (conversion unit) 53B, an MPU buffer (output unit) 54B, and a packet mixing unit (mixing unit) 55.

The packet filter 50B outputs the control message (MP table) separated from the CMAF-applied MMT to the descriptor extraction and deletion unit 51. Other than this, the packet filter 50B is the same as in the second embodiment, so a description thereof will be omitted.

The metadata conversion unit 53B is similar to the second embodiment except that it does not generate MPU metadata that includes a parameter set, so a description thereof will be omitted. Note that the metadata conversion unit 53B may generate and output MPU metadata (not shown) that does not include a parameter set.

The MPU buffer 54B outputs the media fragment units input from the packet filter 50B in chunk units according to the output timing of the control message (MP table). Other than this, the MPU buffer 54B is the same as in the second embodiment, so a description is omitted.

[Action and Effects]
As described above, even when the CMAF-adaptive MMT corresponds to "hev1", the MMT conversion device 5B converts the CMAF-non-adaptive MMT to a CMAF-adaptive MMT as in the second embodiment, so that the receiver 4C can play video and audio normally regardless of the suitability of the CMAF.

Although each embodiment of the present invention has been described in detail above, the present invention is not limited to these, and includes design modifications and the like within the scope of the gist of the present invention.
In the above-described embodiments, the multiplexing method is described as MMT, but is not limited thereto. For example, the multiplexing method may be DASH/ROUTE in at least one of the input to the MMT conversion device and the output from the MMT conversion device.

In the above-described embodiments, the video encoding method is described as HEVC, but this is not limited to this. For example, the video encoding method may be AVC (Advanced Video Coding) or VVC (Versatile Video Coding). Furthermore, the encoding method of the present invention is not limited to the video encoding method, and can also be applied to audio encoding methods such as AAC and 3DA (3D Audio).

In each of the above-described embodiments, the MMT conversion device is described as being built into the receiver, but this is not limited to the above. For example, the MMT conversion device may be implemented as independent hardware. Also, the MMT conversion device may be built into the encoding device or transmission device on the broadcasting station side.

It can also be realized by a program that causes the hardware resources of a computer, such as a CPU, memory, and hard disk, to operate as the MMT conversion device described above. These programs may be distributed via a communication line, or written to a recording medium such as a CD-ROM or flash memory and distributed.

1,1B MMT conversion device (multiplex signal conversion device)
10, 10B Packet filter (separation unit)
11 Message buffer 12 Descriptor conversion unit 13 Descriptor addition unit 14 Parameter set extraction unit 15 Parameter set addition unit 16, 16B MPU buffer (output unit)
17 Packet Mixing Unit (Mixing Unit)
2 Encoding device 3, 3C Transmission device 4, 4C Receiver 5, 5B MMT conversion device (multiplex signal conversion device)
50, 50B Packet filter (separation section)
51 Descriptor extraction/deletion unit 52 Parameter set extraction/deletion unit 53, 53B Metadata conversion unit (conversion unit)
54, 54B MPU buffer (output section)
55 Packet Mixing Unit (Mixing Unit)
100, 100C Broadcasting System

Claims (9)

A multiple signal conversion device that converts a CMAF-applied multiple signal, which is a multiple signal to which CMAF is applied, into a CMAF non-applied multiple signal, which is a multiple signal to which CMAF is not applied,
a separation unit for separating movie metadata, movie fragment metadata, control messages, and media data from the CMAF applied multiplexed signal;
a descriptor conversion unit that converts the DTS-PTS difference information of the movie fragment metadata into a descriptor;
a descriptor adding unit for adding the descriptor to the control message;
an output unit that outputs the media data in units of fragments in accordance with an output timing of the control message;
a mixer that mixes the control message from the descriptor adding unit and the media data from the output unit and outputs the mixed signal as the CMAF non-applied multiplex signal;
A multiplex signal conversion device comprising: a parameter set extraction unit for extracting an encoding parameter set from the movie metadata;
a parameter set adding unit for adding the parameter set to the media data,
2. The multiplexed signal conversion device according to claim 1, wherein the output unit outputs the media data to which the parameter set has been added in units of the fragments in accordance with an output timing of the control message. A multiplex signal conversion device that converts a CMAF non-applied multiplex signal, which is a multiplex signal to which CMAF is not applied, into a CMAF-applied multiplex signal, which is a multiplex signal to which CMAF is applied,
a separation unit that separates a control message and media data from the CMAF non-applied multiplex signal;
a descriptor extraction and deletion unit that extracts a descriptor including DTS-PTS differential information from the control message and deletes the descriptor from the control message;
A conversion unit that converts the DTS-PTS difference information of the descriptor into movie fragment metadata;
an output unit that outputs the media data in chunk units according to an output timing of the movie fragment metadata;
a mixer that mixes the control message from the descriptor extractor/deleter, the movie fragment metadata from the converter, and the media data from the output unit, and outputs the mixed signal as the CMAF applied multiplexed signal;
A multiplex signal conversion device comprising: a parameter set extraction/deletion unit that extracts an encoding parameter set from the media data and deletes the parameter set from the media data,
The multiple signal conversion device according to claim 3 , wherein the conversion unit generates movie metadata including the parameter set. The multiplexed signal conversion device according to claim 1, 2 or 4, characterized in that the multiplexed signal is MMT, the movie metadata is MPU metadata, the media data is media fragment units, and the descriptor is an extended MPU timestamp descriptor. The multiplexed signal conversion device according to claim 3, characterized in that the multiplexed signal is an MMT, the media data is a media fragment unit, and the descriptor is an extended MPU timestamp descriptor. The multiplex signal conversion device according to claim 1 or 2, characterized in that the fragment is an MPU. A program for causing a computer to function as a multiplex signal conversion device according to any one of claims 1 to 7. A receiver equipped with a multiplex signal conversion device according to any one of claims 1 to 7.

Images