JP7591878B2 - Encoding device, decoding device, and program - Google Patents

Description

The present invention relates to an encoding device, a decoding device, and a program.

There is hope for services that use large-screen, high-definition video, but high-definition video requires a huge amount of data, so services that involve transmission require broadband transmission paths. When using radio waves for transmission, the frequency band is limited, so technology is used to compress and encode video data according to the transmission band.

However, compression encoding causes encoding degradation, and the higher the compression rate, the greater the degradation. For this reason, there is a limit to the compression rate required to obtain quality suitable for a service. For example, if you try to broadcast 8K over terrestrial waves (6 MHz transmission bandwidth), you will not be able to obtain sufficient image quality, and the service cannot be provided as it is.

When sufficient transmission bandwidth cannot be secured, encoding degradation can be suppressed by lowering the video resolution, but this will result in a decrease in the quality of the service itself. For example, if 8K video is down-converted to 4K and broadcast, the service will naturally have 4K quality.

Therefore, it is considered that high quality video can be obtained by separately transmitting complementary information using additional transmission means and using the data sent over these transmission paths together. In other words, 4K service can be provided over terrestrial broadcasting, which has a limited transmission bandwidth, and additional information can be transmitted over a separate transmission path, which can then be combined with the 4K data to obtain 8K video. One possible method for achieving this type of service is to use hierarchical coding (see, for example, Patent Document 1). Note that hierarchical coding is also sometimes called scalable coding.

In hierarchical coding, high-quality video can be reproduced by combining the base layer and the enhancement layer, and video can be reproduced using the base layer alone, although the quality will be reduced. Several techniques have been proposed to achieve hierarchical coding. One method for scalably encoding video with different numbers of pixels (resolutions), such as the aforementioned 4K and 8K, is called spatial hierarchical coding.

There is also a known technology that hierarchically encodes video signals from coarse information to fine information (see, for example, Patent Document 2).

JP 2017-135522 A JP 2011-244403 A

In spatial hierarchical coding, for example, low-resolution 4K video is upconverted to generate 8K video, and additional information for generating 8K video from the 4K video is transmitted as an extension layer, resulting in poor prediction accuracy and low coding efficiency. This tendency is particularly noticeable in cases where a high compression rate is required, such as terrestrial broadcasting.

For this reason, in order to obtain 8K video of sufficient quality using spatial hierarchical coding within a limited transmission bandwidth, the amount of additional information required to obtain 8K will increase compared to 4K. Therefore, it is expected that there will be almost no change in the transmission path capacity required to obtain the same quality compared to transmitting 4K and 8K video as is without using hierarchical coding, and the benefits of adopting hierarchical coding are considered to be small.

Thus, although conventional hierarchical coding is useful, there is the problem that the coding efficiency is poor under certain conditions.

The present invention aims to provide an encoding device, a decoding device, and a program that improve the coding efficiency in hierarchical coding.

The encoding device according to the first aspect is a device that cuts out an input video into a plurality of videos and hierarchically encodes the plurality of cut out videos. The encoding device includes a cut-out position designation unit that outputs cut-out control information that designates the position and size of a main video area included in the input video, a region cut-out unit that cuts out the main video area included in the input video based on the cut-out control information, a region division unit that divides a video area other than the main video area that constitutes the input video into a plurality of sub-pictures, a base hierarchical encoding unit that encodes the main video area output by the region cut-out unit and outputs a base hierarchical encoding stream of only the encoded main video, and an extension hierarchical encoding unit that encodes each of the plurality of sub-pictures output by the region division unit and outputs an extension hierarchical encoding stream that can be used in combination with the base hierarchical encoding stream.

The decoding device according to the second aspect is a device that decodes an encoded stream obtained by hierarchical encoding. The decoding device includes a base hierarchical decoding unit that decodes a base hierarchical encoded stream and outputs a main video area included in an input video, an extension hierarchical decoding unit that decodes an extension hierarchical encoded stream that can be used in combination with the base hierarchical encoded stream and outputs a plurality of sub-pictures that constitute a video area other than the main video area of the input video, a region synthesis unit that synthesizes the plurality of sub-pictures output by the extension hierarchical decoding unit and outputs a video area other than the main video area, and a hierarchical synthesis unit that synthesizes the main video area output by the base hierarchical decoding unit and the video area other than the main video area output by the region synthesis unit to restore the input video.

The program according to the third aspect causes a computer to function as the encoding device according to the first aspect.

The program according to the fourth aspect causes a computer to function as a decryption device according to the second aspect.

The present invention provides an encoding device, a decoding device, and a program that improve the coding efficiency in hierarchical coding.

1 is a diagram showing a configuration of a video transmission system according to an embodiment. FIG. 1 is a diagram illustrating a configuration of an encoding device according to an embodiment. 11A to 11C are diagrams illustrating a method for specifying a cut-out position according to the embodiment. FIG. 4 is a diagram illustrating a first division example in an area dividing section according to the embodiment. FIG. 13 is a diagram illustrating a second division example in the region dividing unit according to the embodiment. FIG. 13 is a diagram illustrating a third example of division in the region dividing unit according to the embodiment. FIG. 2 is a diagram illustrating a configuration of an extension hierarchical encoding unit according to the embodiment. FIG. 2 is a diagram illustrating a configuration of a decoding device according to an embodiment. FIG. 13 is a diagram illustrating a configuration of an enhancement layer decoding unit according to the embodiment.

The encoding device and the decoding device according to the embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

(Video Transmission System)
First, a video transmission system according to an embodiment will be described. Fig. 1 is a diagram showing a configuration of a video transmission system 100 according to an embodiment.

As shown in FIG. 1, the video transmission system 100 has a transmitting device 10, a receiving device 20A, and a receiving device 20B. The transmitting device 10 transmits video to the receiving devices 20A and 20B via a transmission path. The transmitting device 10 has an encoding device 1 that performs hierarchical encoding on input video. In the following, an example is given of the case where the video to be transmitted is an 8K video, but it is not necessarily limited to 8K.

The encoding device 1 outputs a base hierarchically encoded stream and an extended hierarchically encoded stream by hierarchical encoding. In the example shown in FIG. 1, the transmitting device 10 transmits the base hierarchically encoded stream to the receiving devices 20A and 20B via the broadcast transmission path 30. For example, the broadcast transmission path 30 is a transmission path that uses terrestrial waves. The transmitting device 10 provides a 4K video transmission service using terrestrial waves, which have a limited transmission band.

The transmitting device 10 also transmits the enhanced hierarchical coding stream to the receiving device 20B via the communication transmission path 40. For example, the communication transmission path 40 is a wired communication line such as the Internet. The transmitting device 10 separately transmits the enhanced hierarchical coding stream as complementary information using an additional transmission means.

The receiving device 20A receives the base hierarchical encoded stream via the broadcast transmission path 30. The receiving device 20A has a decoding device 2A that decodes 4K video from the base hierarchical encoded stream. The receiving device 20A is able to play video from the base hierarchical encoded stream, although the quality is reduced.

The receiving device 20B receives the base layer encoded stream via the broadcast transmission path 30, and receives the extension layer encoded stream via the communication transmission path 40. The receiving device 20B has a decoding device 2B that combines the base layer encoded stream and the extension layer encoded stream to decode 8K video. The receiving device 20B can play back high-quality video by combining the base layer and the extension layer.

In this way, the video transmission system 100 transmits 4K and 8K video using hierarchical coding. Here, in the case of a large-screen, high-definition video service such as 8K, it is considered that viewers do not keep their eyes evenly fixed on the entire screen, but rather typically concentrate on a certain part of the screen while watching.

With this in mind, when generating 4K video from 8K, there is no need to reduce the entire screen; by cropping the 4K video (without reducing it) to include the focal part where the important information is concentrated, it is possible to convey sufficient information.

The encoding device 1 of the transmission device 10 according to the embodiment generates 4K video by extracting the main video area (hereinafter referred to as the "base layer area") from the 8K video, uses the extracted 4K base layer area as the base layer in the hierarchical encoding, and transmits the remaining area (hereinafter referred to as the "base layer area") as an enhancement layer. This makes it possible to provide 8K video services without reducing encoding efficiency while providing 4K video services.

The position where the 4K basic layer area is cut out may be fixed or may be changed for each frame. Also, an example of a basic layer area where 4K video is cut out from 8K video is shown, but the size of the basic layer area may be changed to suit the performance of the device that displays the basic layer.

Decoding device 2A of receiving device 20A decodes the base layer decoded stream to obtain 4K base layer video. Meanwhile, decoding device 2B of receiving device 20B decodes the base layer region from the base layer stream and decodes the enhancement layer region from the enhancement layer stream, and combines the base layer region and the enhancement layer region to obtain 8K video. In this way, 4K video and 8K video can be obtained.

Therefore, according to the video transmission system 100 according to the embodiment, it is possible to realize diverse and high-quality services by achieving both 4K (base layer) and 8K (base layer + enhanced layer) video without reducing the encoding efficiency. Using this, it is possible to realize hierarchical services of 4K and 8K by broadcasting 4K on narrow-band terrestrial waves and transmitting additional information via wired communication such as the Internet.

(Encoding device)
Next, a coding device according to an embodiment will be described. Fig. 2 is a diagram showing a configuration of a coding device 1 according to an embodiment.

As shown in FIG. 2, the encoding device 1 has a cutout position designation unit 11, a region cutout unit 12, a base hierarchical encoding unit 13, a region division unit 14, and an extended hierarchical encoding unit 15.

The cut-out position designation unit 11 outputs cut-out control information that designates the position and size of the base hierarchical layer region to the region cut-out unit 12, the region division unit 14, and the extended hierarchical coding unit 15. When determining the cut-out position, it is desirable to take care during program production so that important information is included in the base hierarchical layer region. Alternatively, it is also possible to designate the cut-out position using a separate means for determining the region to be cut out as important information from the 8K video.

Figure 3 is a diagram showing a method for specifying a cut-out position according to an embodiment. As shown in Figure 3, the cut-out position specification unit 11 specifies the position and size of a 4K basic layer area to be cut out from an 8K input video as cut-out control information. For example, the cut-out position specification unit 11 outputs the coordinate values (x, y) of the top left corner of the basic layer area and the number of horizontal and vertical pixels (w, h) as the cut-out control information.

Here, the cut-out position may be fixed or may be changed according to the content of the video content. The coordinate values may be specified by inputting values determined at the time of content production, or a method of automatically detecting and specifying from the input video may be provided. For example, the cut-out position specification unit 11 may specify a basic hierarchical region to be cut out from the 8K video as important information by face recognition processing on the input video. The size of the basic hierarchical region to be cut out may also be changed according to the content of the video content.

In addition, an example is shown in which the base layer area is positioned so that the long side of the 4K base layer area is parallel to the long side of the 8K input video, but this does not necessarily have to be parallel, and the base layer area may be tilted relative to the horizontal direction. However, if they are not parallel, the division method and encoding process of the enhancement layer, which will be described later, will become more complicated.

The area cutout unit 12 cuts out a base layer area from the input video based on the cutout control information output by the cutout position designation unit 11, i.e., the coordinate values (x, y) and the area size (w, h), and outputs the cut out base layer area to the base layer coding unit 13 and the area division unit 14.

The base layer coding unit 13 codes the base layer region output by the region extraction unit 12 and outputs a base layer coded stream that can be used independently. The coding method used by the base layer coding unit 13 may be any method.

Based on the cut-out control information output by the cut-out position designation unit 11, the region division unit 14 divides the extended hierarchical region other than the base hierarchical region of the input video into regions of shapes that can be encoded by the extended hierarchical encoding unit 15. Each of these divided regions is called a "sub-picture." Considering that a typical encoding method involves encoding a rectangular region as input, the shape of each sub-picture is assumed to be rectangular. The division method may be determined in advance, or may be changed depending on the content, etc., and the division method may be communicated to the decoding side by a method such as multiplexing the stream each time.

Figure 4 shows a first example of division in the region division unit 14 according to the embodiment.

As shown in Figure 4, in the first division example, the extended hierarchical area other than the base hierarchical area is divided into four sub-pictures by extending the vertical sides of the base hierarchical area. The divided rectangular areas, i.e., the sub-pictures, are A, B, C, and D. The shape (horizontal and vertical pixel count) of each of the sub-pictures A, B, C, and D is uniquely determined once the values (x, y, w, h) specified as the crop control information are determined. In other words, if the division method of "dividing by extending the vertical sides" is specified, it is possible to restore the original image using only the order of the sub-pictures A, B, C, and D and the (x, y, w, h) information.

Figure 5 shows a second example of division in the region division unit 14 according to the embodiment.

As shown in Figure 5, in the second division example, the top edge of the base layer area is extended to the left, the left edge of the base layer area is extended downward, the bottom edge of the base layer area is extended to the right, and the right edge of the base layer area is extended upward, thereby dividing the area other than the base layer area into four sub-pictures. In this case, as in the first division example, if the division method is specified in advance, restoration is possible using only the values (x, y, w, h) specified as the cut-out control information.

Figure 6 shows a third example of division in the region division unit 14 according to the embodiment.

As shown in Figure 6, in the third division example, the enhancement hierarchical region is divided into smaller regions to obtain multiple sub-pictures. For example, if the enhancement hierarchical region can be divided into small regions of the same shape (same size), there is an advantage in that the sub-encoders in the enhancement hierarchical encoding process described below can all be the same. However, it is necessary to limit (x, y, w, h) to values that allow the enhancement hierarchical region to be divided into an integer number of identical small regions both horizontally and vertically, without any excess or deficiency.

The extension hierarchical coding unit 15 codes each of the multiple sub-pictures output by the region division unit 14, and outputs an extension hierarchical coding stream. FIG. 7 is a diagram showing the configuration of the extension hierarchical coding unit 15 according to an embodiment. As shown in FIG. 7, the extension hierarchical coding unit 15 has multiple sub-encoders 151a, 151b, 151c, ... and a stream synthesis unit 152.

The multiple sub-encoders 151a, 151b, 151c, etc. each encode the multiple sub-pictures output by the region division unit 14, and output the multiple encoded streams to the stream synthesis unit 152. In the example of Figures 4 and 5, there are four sub-pictures, A, B, C, and D, so it is sufficient to use four sub-encoders.

The stream synthesis unit 152 synthesizes the multiple encoded streams output by the multiple sub-encoders 151a, 151b, 151c, etc. with the cut-out control information output by the cut-out position specification unit 11, and outputs an extended hierarchical encoded stream. Any synthesis method may be used, but it is necessary to be able to determine the cut-out control information and which sub-picture the stream corresponds to, and separate them.

In this way, the encoding device 1 according to the embodiment is a device that performs hierarchical encoding on input video, and includes an area extraction unit 12 that extracts a main video area (basic hierarchical area) included in the input video, an area division unit 14 that divides the remaining video area (extended hierarchical area) of the input video other than the main video area into multiple sub-pictures, a base hierarchical encoding unit 13 that encodes the main video area output by the area extraction unit 12 and outputs a base hierarchical encoding stream that can be used alone, and an extension hierarchical encoding unit 15 that encodes each of the multiple sub-pictures output by the area division unit 14 and outputs an extension hierarchical encoding stream that can be used in combination with the base hierarchical encoding stream. This makes it possible to realize a variety of high-quality services by achieving both 4K (basic hierarchical layer) and 8K (basic hierarchical layer + extended hierarchical layer) video without reducing encoding efficiency.

(Decoding device)
Next, a decoding device according to an embodiment will be described. Fig. 8 is a diagram showing the configuration of a decoding device 2 according to an embodiment. Here, as the decoding device 2, the configuration of the decoding device 2B shown in Fig. 1 is shown.

As shown in FIG. 8, the decoding device 2 has a base layer decoding unit 21, an enhancement layer decoding unit 22, a region synthesis unit 23, and a layer synthesis unit 24.

The base layer decoding unit 21 decodes the base layer encoded stream and outputs the base layer area. Note that when the decoding device 2 is the decoding device 2A shown in FIG. 1, only the base layer area, which is a 4K video, is reproduced.

The enhancement layer decoding unit 22 decodes the enhancement layer encoded stream and outputs a plurality of sub-pictures constituting the enhancement layer area and cutout control information. FIG. 9 is a diagram showing the configuration of the enhancement layer decoding unit 22 according to an embodiment. As shown in FIG. 9, the enhancement layer decoding unit 22 has a stream separation unit 221 and a plurality of sub-decoders 222a, 222b, 222c, etc.

The stream separation unit 221 separates the enhanced hierarchical encoding stream into multiple encoding streams corresponding to multiple sub-pictures and cut-out control information, outputs the multiple encoding streams to multiple sub-decoders 222a, 222b, 222c, etc., and outputs the cut-out control information to the region synthesis unit 23 and the hierarchical synthesis unit 24. Here, the method of separating the streams and outputting them to the corresponding sub-decoders corresponds to the synthesis method used by the stream synthesis unit 152 of the encoding device 1.

The multiple sub-decoders 222a, 222b, 222c, etc. decode multiple sub-pictures from the multiple encoded streams output by the stream separation unit 221, and output the decoded sub-pictures to the region synthesis unit 23.

The region synthesis unit 23 synthesizes multiple sub-pictures output by the extension hierarchical decoding unit 22 (multiple sub-decoders 222a, 222b, 222c, ...) and outputs an extension hierarchical region. The synthesis method corresponds to the division method used by the region division unit 14 of the encoding device 1. As described above, the division method used by the encoding device 1 may be determined in advance, or may be changed depending on the content, etc., and communicated each time by multiplexing the method into the stream, for example. The synthesis method is determined by the sub-picture division method and the cut-out control information, and the region synthesis unit 23 synthesizes multiple sub-pictures accordingly to output the extension hierarchical region.

The layer synthesis unit 24 synthesizes the base layer region output by the base layer decoding unit 21 and the extended layer region output by the region synthesis unit 23 based on the cut-out control information output by the extended layer decoding unit 22, and restores and outputs the 8K video.

In this way, the decoding device 2 according to the embodiment is a device that decodes an encoded stream obtained by hierarchical encoding, and has a base layer decoding unit 21 that decodes a base layer encoded stream that can be used alone and outputs a main video area (base layer area) included in the input video, an extension layer decoding unit 22 that decodes an extension layer encoded stream that can be used in combination with the base layer encoded stream and outputs a plurality of sub-pictures that constitute the remaining video area (extension layer area) other than the main video area of the input video, a region synthesis unit 23 that synthesizes the plurality of sub-pictures output by the extension layer decoding unit 22 and outputs the remaining video area, and a layer synthesis unit 24 that synthesizes the main video area output by the base layer decoding unit 21 and the remaining video area output by the region synthesis unit 23 to restore the input video. This makes it possible to realize a variety of high-quality services by achieving both 4K (base layer) and 8K (base layer + extension layer) video without reducing encoding efficiency.

Other Embodiments
In the embodiment, an example of configuring hierarchical coding as two layers using a base layer and an extension layer has been described, but it is also possible to use the method according to the embodiment recursively to configure three or more layers, or to combine it with other hierarchical coding methods.

By using the communication transmission path 40, information on the area that the viewer pays attention to may be fed back from the receiving device 20 (decoding device 2) to the transmitting device 10 (encoding device 1). In this case, the cut-out position designation unit 11 may determine the base layer area to be cut out as important information based on the fed-back information.

In the embodiment, an example has been described in which one base layer area is extracted from the input video, but the area extraction unit 12 may extract two or more base layer areas from the input video. The receiving device 20 may select one base layer area to decode from the two or more base layer areas, and decode only the selected base layer area.

A program may be provided that causes a computer to execute each process performed by the encoding device 1. Also, a program may be provided that causes a computer to execute each process performed by the decoding device 2. The program may be recorded on a computer-readable medium. Using a computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

The circuits that execute the processes performed by the encoding device 1 may be integrated, and the encoding device 1 may be configured as a semiconductor integrated circuit (chip set, SoC). The circuits that execute the processes performed by the decoding device 2 may be integrated, and the decoding device 2 may be configured as a semiconductor integrated circuit (chip set, SoC).

The above describes the embodiment in detail with reference to the drawings, but the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the invention.

1: Encoding device 2: Decoding device 2A: Decoding device 2B: Decoding device 10: Transmitting device 11: Cut-out position designation unit 12: Area cut-out unit 13: Base layer encoding unit 14: Area division unit 15: Enhancement layer encoding unit 20: Receiving device 20A: Receiving device 20B: Receiving device 21: Base layer decoding unit 22: Enhancement layer decoding unit 23: Area synthesis unit 24: Layer synthesis unit 30: Broadcast transmission path 40: Communication transmission path 100: Video transmission system 151a: Sub-encoder 151b: Sub-encoder 151c: Sub-encoder 152: Stream synthesis unit 221: Stream separation unit 222a: Sub-decoder 222b: Sub-decoder 222c: Sub-decoder

Claims (7)

1. An encoding device that cuts out an input video into a plurality of videos and hierarchically encodes the plurality of cut out videos, comprising:
a cut-out position designation unit that outputs cut-out control information that designates a position and a size of a main image area included in the input image;
an area cutout unit that cuts out the main image area included in the input image based on the cutout control information;
an area dividing unit that divides an image area other than the main image area constituting the input image into a plurality of sub-pictures;
a base hierarchical encoding unit that encodes the main video area output by the area extraction unit and outputs a base hierarchical encoded stream including only the encoded main video;
an extension hierarchical coding unit that codes each of the plurality of sub-pictures output by the region division unit and outputs an extension hierarchical coding stream that can be used in combination with the base hierarchical coding stream. The enhancement hierarchical coding unit includes:
a plurality of sub-encoders each encoding the plurality of sub-pictures;
2. The encoding device according to claim 1, further comprising: a stream synthesis unit that synthesizes a plurality of encoded streams output by the plurality of sub-encoders and the cut-out control information output by the cut-out position designation unit, and outputs the extended hierarchical encoded stream. A decoding device that decodes a coded stream obtained by hierarchical coding, comprising:
a base layer decoding unit that decodes the base layer encoded stream and outputs a main video area included in the input video;
an enhancement layer decoding unit that decodes an enhancement layer coded stream that can be used in combination with the base layer coded stream, and outputs a plurality of sub-pictures that constitute an image area other than the main image area of the input image;
a region synthesis unit that synthesizes the plurality of sub-pictures output by the enhancement layer decoding unit and outputs a video region other than the main video region;
A decoding device characterized by comprising a hierarchical synthesis unit that synthesizes the main video area output by the base hierarchical decoding unit with a video area other than the main video area output by the area synthesis unit to restore the input video. The enhancement layer decoding unit
a stream separator that separates the enhancement hierarchical coded stream into a plurality of coded streams corresponding to the plurality of sub-pictures and cut-out control information that specifies a position and a size of the main video area;
a plurality of sub-decoders each decoding the plurality of sub-pictures from the plurality of encoded streams,
4. The decoding device according to claim 3, wherein the area synthesis unit synthesizes the plurality of sub-pictures based on the cut-out control information, and outputs an image area other than the main image area. a first decoding unit that decodes the encoded stream and outputs a first video region included in the input video;
a second decoding unit that decodes an encoded stream that can be used in combination with the encoded stream, and outputs one or more sub-pictures that constitute an image area of the input video other than the first image area;
an acquisition unit that acquires control information that specifies a position and a size of the first video area;
a synthesis unit that synthesizes the first video area output by the first decoding unit and the one or more sub-pictures output by the second decoding unit based on the control information to restore the input video. A program that causes a computer to function as the encoding device according to claim 1 or 2. A program for causing a computer to function as the decoding device according to any one of claims 3 to 5 .

Images