CN109040762B - Decoding apparatus and decoding method - Google Patents

Abstract

The present technology relates to an encoding apparatus and an encoding method capable of reducing the amount of information related to information specifying a reference image. The coding unit generates a predicted image using the reference image. The transmission unit transmits inter_ref_pic_set_prediction_flag when the current coded picture is a picture other than the first picture of the GOP (group of pictures). Specifies information that the previous picture is a picture preceding the current coded picture in coding order. For example, the present technology can be applied to an encoding apparatus of the HEVC (High Efficiency Video Coding) scheme.

Description

Decoding device and decoding method

technical field

The present technology relates to an encoding apparatus and an encoding method, and more particularly, to an encoding apparatus and an encoding method capable of reducing the amount of information related to information specifying a reference image.

Background technique

Recently, image information is handled as digital data, and at this time, in order to efficiently transmit and store information, it is compatible to perform orthogonal transform such as discrete cosine transform by utilizing redundancy peculiar to image information, and utilize motion compensation Compressed MPEG (Moving Picture Experts Group) and other systems are widely used for information transmission in broadcasting stations, etc., and information reception in general households.

In particular, the MPEG2 (ISO/IEC 13818-2) method is defined as a general image coding method, and is currently widely used as a standard covering both interlaced and progressive images, as well as standard resolution images and high-definition images. Various applications for professional and consumer use. By using the MPEG2 method, for example, a code amount (bit rate) of 4-8 Mbps is allocated in the case of an interlaced image of a standard resolution of 720×480 pixels, and a code amount (bit rate) of 4 to 8 Mbps in the case of a high-definition interlaced image of 1920×1088 pixels Under the 18-22Mbps code volume, which can achieve high compression rate and improved image quality.

MPEG2 targets high image quality encoding, which is mainly suitable for broadcasting, but does not correspond to an encoding method with a lower code amount (bit rate) than that of MPEG1, in other words, a higher compression rate encoding method. With the popularization of mobile terminals, it is predicted that the demand for this encoding method will increase in the future, and in response to this, the standardization of the MPEG4 encoding method has been carried out. Related to the image encoding method of MPEG4, in December 1998, a specification was approved as the international standard ISO/IEC 14496-2.

In addition, recently, standardization of H.26L (ITU-T Q6/16 VCEG) is in progress for image coding for video conferences. Although H.26L requires larger encoding and decoding computations than conventional encoding methods such as MPEG2 or MPEG4, it is known that higher encoding efficiency can be achieved.

Furthermore, currently, as part of the activities of MPEG4, as the Enhanced Compression Video Coding Joint Mode, standardization of a specification based on H.26L that includes functions not supported in H.26L and achieves higher coding efficiency is being carried out. This standardization was internationally standardized in March 2003 based on the names of H.264 and MPEG-4 Part 10 (AVC (Advanced Video Coding)).

In addition, in February 2005, an encoding tool called RGB, 4:2:2 or 4:4:4 required for the service, and 8x8 DCT defined in MPEG-2, were completed including as extensions and FRExt (Fidelity Range Extension) normalization of the quantization matrix. Thus, AVC has become an encoding method that can also improve the expression of film noise contained in movies, and is a method used therein for various applications such as Blu-ray (registered trademark) discs.

However, recently, more and more attention has been paid to compressing an image of about 4000×2000 pixels (4 times that of a high vision image) and transmitting a high vision image in an environment with limited transmission capacity such as the Internet. The demand for high compression rate encoding has increased. Therefore, in the VCEG (Video Coding Experts Group) under the leadership of the ITU-T, the review on improving the coding efficiency is continuously carried out.

Meanwhile, in the HEVC (High Efficiency Video Coding) method, a short-term reference picture set (hereinafter referred to as RPS) containing reference picture designation information for identifying a reference picture in a decoding device is included in an SPS (Sequence Parameter Set) (for example, See Non-Patent Document 1).

FIG. 1 is a diagram illustrating an example of the syntax of RPS.

As illustrated in row 2 in FIG. 1 , in the RPS, inter_ref_pic_set_prediction_flag is included. Here, inter_ref_pic_set_prediction_flag is reference information indicating whether or not the reference picture specification information specifying the reference picture of the previous picture, which is within the GOP (Group of Pictures) of the current coded picture, is used as the reference picture specification information of the current coded picture, The picture that precedes the currently encoded picture in encoding order.

Here, inter_ref_pic_set_prediction_flag is "1" in the case where it indicates that the reference picture designation information of the reference picture designating the previous picture is used as the reference picture designation information of the current coded picture, and is "1" in the case where it indicates that the reference picture designating the previous picture's reference picture When the reference picture specification information is not used as the reference picture specification information of the current coded picture, it is "0".

As shown in the 3rd and 4th lines of FIG. 1 , in the case where inter_ref_pic_set_prediction_flag is "1", delta_idx_minus1 which is previous picture designation information for designating a previous picture is included in the RPS. More specifically, delta_idx_minus1 has a value obtained by subtracting 1 from a value obtained by subtracting the code number of the previous picture from the code number (coding order) of the current coded picture. Here, the code number is a number assigned to each picture in the GOP, starting from a smaller value in the coding order.

In addition, as illustrated in lines 13 to 23 in FIG. 1 , when inter_ref_pic_set_prediction_flag is “0”, parameter image designation information is included in the RPS.

FIG. 2 is a diagram illustrating an example of inter_ref_pic_set_prediction_flag and delta_idx_minus1.

In the example illustrated in FIG. 2, the reference picture designation information of the current coded picture whose code number is N and the reference of the previous picture whose code number is "N-1" preceding the current coded picture in the coding order The image specification information is the same.

In this case, inter_ref_pic_set_prediction_flag is set to "1", indicating that the reference picture specifying the information of the previous picture is used as the reference picture specifying the information of the current coded picture. In addition, delta_idx_minus1 is set by subtracting "N-1", which is the code number of the previous picture, from "N", which is the code number of the current encoded picture, and then from the value "1" obtained by the subtraction. , then subtract 1 to get "0".

Citation List

Non-patent literature

Non-patent document 1: Benjamin Bross, Woo-Jin Han, Jens-Rainer Ohm, Gary J. Sullivan, Thomas Wiegand, "High efficiency video coding (HEVC) textspecification draft7", JCTVC-I1003_d4, 2012.4.27-5.7.

SUMMARY OF THE INVENTION

However, the amount of information related to reference picture designation information such as RPS is not sufficiently reduced.

The present technology, which can reduce the amount of information related to information specifying a reference image, has been invented in consideration of such a situation.

According to an aspect of the present technology, there is provided an encoding apparatus including: a predicted image generation unit configured to generate a predicted image using a reference image; and a transmission unit configured to When the coded picture is a picture other than the first picture of the GOP (Group of Pictures), transmits reference information indicating whether the reference picture specifying information specifying the reference picture of the previous picture is used as the reference picture of the current coded picture Specifies information that the previous picture is a picture preceding the current coded picture in coding order.

An encoding method according to another aspect of the present technology corresponds to the encoding apparatus according to the one aspect of the present technology.

According to the one aspect of the present technology, a reference picture is used to generate a predicted picture; if the currently encoded picture is a picture other than the first picture of a GOP (group of pictures), reference information is transmitted, the reference information indicating the previous Whether the reference picture specification information of the specified reference picture of the picture, which is the picture preceding the current coded picture in the coding order, is used as the reference picture specification information of the current coded picture.

In addition, by causing a computer to execute the program, the encoding device according to the one aspect of the present technology can be realized.

Furthermore, in order to realize the encoding apparatus according to the one aspect of the present technology, by being transmitted via a transmission medium, or being recorded on a recording medium, a program executed by a computer may be provided.

According to the present technology, it is possible to reduce the amount of information related to information specifying a reference image.

Description of drawings

FIG. 1 is a diagram illustrating an example of the syntax of RPS.

FIG. 2 is a diagram illustrating an example of inter_ref_pic_set_prediction_flag and delta_idx_minus1.

FIG. 3 is a block diagram illustrating a configuration example of an encoding apparatus to which the present technology according to the first embodiment is applied.

FIG. 4 is a block diagram illustrating a structural example of the coding unit illustrated in FIG. 3 .

FIG. 5 is a diagram illustrating an example of the syntax of the SPS set by the setting unit 12 illustrated in FIG. 3 .

FIG. 6 is a diagram illustrating an example of the syntax of the RPS.

FIG. 7 is a diagram illustrating the information amount of the RPS set by the setting unit 12 illustrated in FIG. 3 .

FIG. 8 is a diagram illustrating the amount of information of a conventional RPS.

FIG. 9 is a diagram illustrating an example of the syntax of the slice header.

FIG. 10 is a flowchart illustrating generation processing by the encoding apparatus illustrated in FIG. 3 .

FIG. 11 is a flowchart illustrating in detail the RPS setting process illustrated in FIG. 10 .

FIG. 12 is a flowchart illustrating the encoding process illustrated in FIG. 10 in detail.

FIG. 13 is a flowchart illustrating the encoding process illustrated in FIG. 10 in detail.

FIG. 14 is a flowchart illustrating the RPS index determination process illustrated in FIG. 12 in detail.

15 is a block diagram illustrating a structural example of a decoding apparatus to which the present technology according to the first embodiment is applied.

FIG. 16 is a block diagram illustrating a structural example of the decoding unit illustrated in FIG. 15 .

FIG. 17 is a flowchart illustrating reception processing by the decoding apparatus illustrated in FIG. 15 .

FIG. 18 is a flowchart illustrating the RPS setting process illustrated in FIG. 17 in detail.

FIG. 19 is a flowchart illustrating the decoding process illustrated in FIG. 17 in detail.

20 is a block diagram illustrating a structural example of an encoding apparatus to which the present technology according to the second embodiment is applied.

FIG. 21 is a diagram illustrating an example of the syntax of the SPS set by the setting unit illustrated in FIG. 20 .

FIG. 22 is a diagram illustrating an example of the syntax of the RPS illustrated in FIG. 21 .

FIG. 23 is a diagram illustrating the information amount of the RPS set by the setting unit illustrated in FIG. 20 .

FIG. 24 is a diagram illustrating the information amount of the RPS set by the setting unit illustrated in FIG. 20 .

FIG. 25 is a diagram illustrating the amount of information of a conventional RPS.

FIG. 26 is a flowchart illustrating in detail the RPS setting process performed by the encoding apparatus illustrated in FIG. 20 .

27 is a block diagram illustrating a structural example of a decoding apparatus to which the present technology according to the second embodiment is applied.

FIG. 28 is a flowchart illustrating in detail the RPS setting process performed by the decoding apparatus illustrated in FIG. 27 .

FIG. 29 is a block diagram illustrating a structural example of an encoding apparatus to which the present technology according to the third embodiment is applied.

FIG. 30 is a diagram illustrating an example of the syntax of the SPS set by the setting unit illustrated in FIG. 29 .

FIG. 31 is a diagram of an example of the syntax of the RPS illustrated in FIG. 30 .

FIG. 32 is a diagram illustrating the information amount of the RPS set by the setting unit illustrated in FIG. 29 .

FIG. 33 is a flowchart illustrating in detail the RPS setting process performed by the encoding apparatus illustrated in FIG. 29 .

34 is a block diagram illustrating a structural example of a decoding apparatus to which the present technology according to the third embodiment is applied.

FIG. 35 is a flowchart illustrating in detail the RPS setting process performed by the decoding apparatus illustrated in FIG. 34 .

FIG. 36 is a block diagram illustrating a structural example of an encoding apparatus to which the present technology according to the fourth embodiment is applied.

FIG. 37 is a block diagram illustrating a structural example of the coding unit illustrated in FIG. 36 .

FIG. 38 is a diagram illustrating an example of the syntax of the PPS set by the setting unit illustrated in FIG. 36 .

FIG. 39 is a diagram illustrating an example of the syntax of the PPS set by the setting unit illustrated in FIG. 36 .

FIG. 40 is a diagram illustrating an example of the syntax of PPS in a conventional HEVC system.

FIG. 41 is a diagram illustrating an example of the syntax of PPS in a conventional HEVC system.

FIG. 42 is a diagram illustrating an example of syntax of a slice header added by the lossless coding unit illustrated in FIG. 37 .

FIG. 43 is a diagram illustrating an example of syntax of a slice header added by the lossless coding unit illustrated in FIG. 37 .

FIG. 44 is a diagram illustrating an example of syntax of a slice header added by the lossless coding unit illustrated in FIG. 37 .

FIG. 45 is a diagram illustrating an example of the syntax of a slice header in a conventional HEVC system.

FIG. 46 is a diagram illustrating an example of syntax of a slice header in a conventional HEVC system.

FIG. 47 is a diagram illustrating an example of syntax of a slice header in a conventional HEVC system.

FIG. 48 is a flowchart illustrating generation processing by the encoding apparatus illustrated in FIG. 36 .

FIG. 49 is a flowchart illustrating the encoding process illustrated in FIG. 48 in detail.

FIG. 50 is a flowchart illustrating the encoding process illustrated in FIG. 48 in detail.

FIG. 51 is a flowchart illustrating in detail the PPS setting process illustrated in FIG. 48 .

FIG. 52 is a block diagram illustrating a structural example of a decoding apparatus to which the present technology according to the fourth embodiment is applied.

FIG. 53 is a block diagram illustrating a structural example of the decoding unit illustrated in FIG. 52 .

FIG. 54 is a flowchart illustrating reception processing by the decoding apparatus illustrated in FIG. 52 .

FIG. 55 is a flowchart illustrating the decoding process illustrated in FIG. 54 in detail.

FIG. 56 is a diagram illustrating an example of a multi-view image encoding system.

FIG. 57 is a diagram illustrating an example of the main structure of a multi-viewpoint image encoding apparatus to which the present technology is applied.

FIG. 58 is a diagram illustrating an example of the main structure of a multi-viewpoint image decoding apparatus to which the present technology is applied.

FIG. 59 is a diagram illustrating an example of a layered image coding system.

FIG. 60 is a diagram illustrating an example of the main structure of a layered image encoding apparatus to which the present technology is applied.

FIG. 61 is a diagram illustrating an example of the main structure of a layered image decoding apparatus to which the present technology is applied.

FIG. 62 is a block diagram illustrating an example of a hardware configuration of a computer.

FIG. 63 is a diagram illustrating an example of a schematic structure of a television to which the present technology is applied.

FIG. 64 is a diagram illustrating an example of a schematic structure of a mobile phone to which the present technology is applied.

FIG. 65 is a diagram illustrating an example of a schematic structure of a recording and reproducing apparatus to which the present technology is applied.

FIG. 66 is a diagram illustrating an example of a schematic structure of an imaging apparatus to which the present technology is applied.

67 is a block diagram illustrating a use example of scalable encoding.

FIG. 68 is a block diagram illustrating yet another example of use of scalable coding.

FIG. 69 is a block diagram illustrating another example of use of scalable coding.

FIG. 70 is a diagram illustrating an example of a schematic structure of a video set to which the present technology is applied.

FIG. 71 is a diagram illustrating an example of a schematic structure of a video processor to which the present technology is applied.

FIG. 72 is a diagram illustrating another example of a schematic structure of a video processor to which the present technology is applied.

Detailed ways

<First Embodiment>

(Configuration example of the encoding apparatus according to the first embodiment)

FIG. 3 is a block diagram illustrating a configuration example of an encoding apparatus to which the present technology according to the first embodiment is applied.

The encoding apparatus 10 illustrated in FIG. 3 is constituted by an encoding unit 11, a setting unit 12, and a transmission unit 13, and encodes an image in the HEVC method.

More specifically, an image constituted in units of frames is input to the encoding unit 11 of the encoding device 10 as an input signal. The encoding unit 11 encodes the input signal in the HEVC manner by referring to the RPS supplied from the setting unit 12 , and then supplies the encoded data obtained by encoding to the setting unit 12 .

The setting unit 12 sets an RPS that does not include inter_ref_pic_set_prediction_flag, but includes reference picture designation information, and an RPS that includes inter_ref_pic_set_prediction_flag and reference picture designation information or delta_idx_minus1. The setting unit 12 assigns to each RPS an index as reference picture information specifying information specifying an RPS (reference picture information). Here, it is assumed that "0" is set not to include the inter_ref_pic_set_prediction_flag, but to include the index of the RPS of the reference picture designation information.

The setting unit 12 supplies the RPS to which the index has been assigned to the encoding unit 11 . The setting unit 12 sets SPS including RPS, PPS (Picture Parameter Set), and the like.

The setting unit 12 generates an encoded stream based on the set SPS and PPS and the encoded data supplied from the encoding unit 11 . The setting unit 12 supplies the encoded stream to the transmission unit 13 .

The transmission unit 13 transmits the encoded stream supplied from the setting unit 12 to a decoding device described later.

(Structure example of coding unit)

FIG. 4 is a block diagram illustrating a structural example of the encoding unit 11 illustrated in FIG. 3 .

The encoding unit 11 illustrated in FIG. 4 includes: A/D converter 31; screen rearrangement buffer 32; calculation unit 33; orthogonal transform unit 34; quantization unit 35; lossless encoding unit 36; accumulation buffer 37; Inverse quantization unit 38; inverse orthogonal transform unit 39; addition unit 40; deblocking filter 41; adaptive offset filter 42; adaptive loop filter 43; frame memory 44; switch 45; intra prediction unit 46 ; motion prediction/compensation unit 47 ; predicted image selection unit 48 ; reference image setting unit 49 ; and rate control unit 50 .

More specifically, the A/D converter 31 of the encoding unit 11 performs A/D conversion of the frame-by-frame image input as an input signal, and outputs the converted image to the screen rearrangement buffer 32 for saving in in. The screen rearrangement buffer 32 rearranges the stored images in units of frames in the display order according to the GOP structure in the coding order, and outputs the rearranged images to the calculation unit 33, the intra prediction unit 46, and the motion prediction/ Compensation unit 47 .

The calculation unit 33 functions as an encoding unit, and performs encoding by calculating the difference between the predicted image supplied from the predicted image selection unit 48 and the currently encoded image output from the screen rearrangement buffer 32 . More specifically, the calculation unit 33 performs encoding by subtracting the predicted image supplied from the predicted image selection unit 48 from the current encoded image output from the screen rearrangement buffer 32 . The calculation unit 33 outputs the image obtained as a result to the orthogonal transformation unit 34 as residual information. In addition, when the predicted image is not supplied from the predicted image selection unit 48 , the calculation unit 33 directly outputs the image read from the screen rearrangement buffer 32 to the orthogonal transformation unit 34 as residual information.

The orthogonal transform unit 34 performs orthogonal transform of the residual information output from the calculation unit 33 to generate orthogonal transform coefficients. The orthogonal transform unit 34 supplies the generated orthogonal transform coefficients to the quantization unit 35 .

The quantization unit 35 performs quantization of the orthogonal transform coefficient supplied from the orthogonal transform unit 34 by using the quantization parameter supplied from the rate control unit 50 . The quantization unit 35 inputs the coefficients obtained as a result thereof to the lossless encoding unit 36 .

The lossless encoding unit 36 obtains information indicating the optimum intra prediction mode (hereinafter referred to as intra prediction mode information) from the intra prediction unit 46 . In addition, the lossless encoding unit 36 obtains, from the motion prediction/compensation unit 47, information indicating the optimum inter prediction mode, motion vector, and the like (hereinafter referred to as inter prediction mode information). In addition, the lossless encoding unit 36 obtains the index of the RPS, the RPS, and the like from the reference image setting unit 49 , and obtains the quantization parameter from the rate control unit 50 .

In addition, the lossless encoding unit 36 obtains the storage flag, index or offset amount, and category information as offset filter information from the adaptive offset filter 42, and obtains filter coefficients from the adaptive loop filter 43.

The lossless encoding unit 36 performs lossless encoding such as variable-length encoding (for example, CAVLC (Context Adaptive Variable Length Coding) or the like), or arithmetic encoding (for example, CABAC (Context-Adaptive Binary Arithmetic Coding), on the quantized coefficients supplied from the quantization unit 35 ).

In addition, the lossless encoding unit 36 performs lossless encoding of quantization parameters such as intra prediction mode information or inter prediction mode information, motion vector, index of RPS, or RPS, offset filter information, and filter coefficients as encoding-related encoding information. The lossless encoding unit 36 supplies the encoding information and coefficients encoded in the lossless manner to the accumulation buffer 37 as encoded data so as to be held therein. In addition, the encoding information encoded in the lossless manner can be regarded as header information (slice header) of the coefficients encoded in the lossless manner.

The accumulation buffer 37 temporarily holds the encoded data supplied from the lossless encoding unit 36 . In addition, the accumulation buffer 37 supplies the held encoded data to the setting unit 12 illustrated in FIG. 3 .

In addition, the quantization coefficient output from the quantization unit 35 is also input to the inverse quantization unit 38 . The inverse quantization unit 38 performs inverse quantization of the coefficients quantized by the quantization unit 35 by using the quantization parameter supplied from the rate control unit 50 , and supplies the orthogonal transform coefficients obtained as a result to the inverse orthogonal transform unit 39 .

The inverse orthogonal transform unit 39 performs inverse orthogonal transform of the orthogonal transform coefficients supplied from the inverse quantization unit 38 . The inverse orthogonal transform unit 39 supplies the residual information obtained by the inverse orthogonal transform to the addition unit 40 .

The addition unit 40 adds the residual information supplied from the inverse orthogonal transform unit 39 and the predicted image supplied from the predicted image selection unit 48, thereby obtaining an image that has been locally decoded. In addition, in the case where the predicted image is not supplied from the predicted image selection unit 48, the addition unit 40 sets the residual information supplied from the inverse orthogonal transform unit 39 as a local decoded image. The adding unit 40 supplies the locally decoded image to the deblocking filter 41 and supplies the locally decoded image to the frame memory 44 for storage therein.

The deblocking filter 41 performs adaptive deblocking filter processing for removing block distortion on the locally decoded image supplied from the adding unit 40 , and supplies the image obtained as a result to the adaptive offset filter 42 .

The adaptive offset filter 42 performs an adaptive offset filter (SAO: Sampling Point Adaptive Offset) that mainly eliminates ringing of the image on the image after the adaptive deblocking filter process performed by the deblocking filter 41 . move) processing.

More specifically, the adaptive offset filter 42 determines the kind of adaptive offset filter processing for each LCU (largest coding unit) that is the largest coding unit, and obtains the offset amount used in the adaptive offset filter processing. . The adaptive offset filter 42 uses the obtained offset to perform a specific type of adaptive offset filter processing on the image after the adaptive deblocking filter processing. Then, the adaptive offset filter 42 supplies the image after the adaptive offset filter processing to the adaptive loop filter 43 .

In addition, the adaptive offset filter 42 has a buffer in which the offset is stored. For each LCU, adaptive offset filter 42 determines whether an offset for the adaptive deblocking filtering process has been stored in the buffer.

In a case where it is determined that the offset used for the adaptive deblocking filtering process has been stored in the buffer, the adaptive offset filter 42 puts a storage flag indicating whether the offset is stored in the buffer, Set to a value (here, "1") indicating that the offset is stored in the buffer.

Then, for each LCU, the adaptive offset filter 42 sets the storage flag to "1", the index indicating the storage position of the offset in the buffer, and the index indicating the performed adaptive offset filtering process. The category information of the category is supplied to the lossless encoding unit 36 .

On the other hand, in the case where the offsets used for the adaptive deblocking filtering process are not stored in the buffer, the adaptive offset filter 42 sequentially stores the offsets in the buffer. In addition, the adaptive offset filter 42 sets the storage flag to a value (here, "0") indicating that the offset is not stored in the buffer. Then, for each LCU, the adaptive offset filter 42 supplies the storage flag, the offset amount, and the category information set to "0" to the lossless encoding unit 36 .

For each LCU, the adaptive loop filter 43 performs, for example, adaptive loop filter (ALF: Adaptive Loop Filter) processing on the image after the adaptive offset filter processing supplied from the adaptive offset filter 42 . For example, as the adaptive loop filter processing, processing using a two-dimensional Wiener filter is used. Obviously filters other than Wiener filters can be used.

More specifically, for each LCU, the adaptive loop filter 43 calculates filter coefficients used in the adaptive loop filter process so as to be the original image which is the image output from the screen rearrangement buffer 32, and the adaptive loop filter. The residuals between the images after channel filtering are minimized. Then, for each LCU, the adaptive loop filter 43 performs adaptive loop filter processing on the image after the adaptive offset filter processing by using the calculated filter coefficients.

The adaptive loop filter 43 supplies the image after the adaptive loop filter processing to the frame memory 44 . In addition, the adaptive loop filter 43 supplies the filter coefficients to the lossless encoding unit 36.

Here, although it is assumed that the adaptive loop filter processing is performed on each LCU, the processing unit of the adaptive loop filter processing is not limited to the LCU. However, by matching the processing units of the adaptive offset filter 42 and the adaptive loop filter 43 with each other, processing can be performed efficiently.

The frame memory 44 holds the image supplied from the adaptive loop filter 43 and the image supplied from the adding unit 40 . The image stored in the frame memory 44 is output to the intra prediction unit 46 or the motion prediction/compensation unit 47 through the switch 45 as a reference image.

The intra prediction unit 46 performs intra prediction processing of all intra prediction modes that are candidates by using the reference image read from the frame memory 44 via the switch 45 .

In addition, the intra prediction unit 46 calculates cost function values of all intra prediction modes that are candidates based on the images read from the screen rearrangement buffer 32 and the predicted images generated by the intra prediction processing (details will be described later). . Subsequently, the intra prediction unit 46 determines the intra prediction mode whose cost function value is the smallest as the optimal intra prediction mode.

The intra prediction unit 46 supplies the predicted image generated in accordance with the optimal intra prediction mode, and the corresponding cost function value to the predicted image selection unit 48 . When the intra prediction unit 46 is informed of the selection of the predicted image generated in the optimum intra prediction mode from the predicted image selection unit 48 , the intra prediction unit 46 supplies the intra prediction mode information to the lossless encoding unit 36 .

The cost function value is also called RD (Rate Distortion) cost, and as defined in JM (Joint Model) which is reference software according to the H.264/AVC method, is according to the high complexity mode and the low complexity mode One of the technical computing ones.

More specifically, in the case of employing a high-complexity mode as a technique for calculating a cost function value, for all prediction modes that are candidates, decoding is temporarily performed, and for each prediction mode, the expression expressed by the following equation (1) is calculated: Cost function value.

cost(mode)=D+λ·R...(1)

Here, D is the difference (distortion) between the original image and the decoded image, R is the amount of generated coding that also includes the coefficients of the orthogonal transform, and λ is the Lagrangian multiplier given as a function of the quantization parameter QP.

On the other hand, in the case of employing the low-complexity mode as the technique for calculating the cost function value, the generation of the predicted image and the calculation of the encoding amount of the encoding information are performed for each of all the prediction modes that are candidates, and for each A cost function expressed by the following equation (2) is calculated for each prediction mode.

Cost(Mode)=D+QPtoQuant(QP)·Header_Bit...(2)

Here, D is the difference (distortion) between the original image and the decoded image, Header_Bit is the encoding amount of encoded information, and QPtoQuant is a function given as a function of the quantization parameter QP.

In the low-complexity mode, for all prediction modes, only the predicted image can be generated, and the decoded image does not have to be generated, so that the amount of computation is small.

The motion prediction/compensation unit 47 performs motion prediction/compensation processing of all inter prediction modes that are candidates. More specifically, the motion prediction/compensation unit 47 detects motion vectors of all inter prediction modes as candidates from the image supplied from the screen rearrangement buffer 32 and the reference image read from the frame memory 44 through the switch 45 . Subsequently, the motion prediction/compensation unit 47 functions as a predicted image generation unit, and generates a predicted image by performing compensation processing of the reference image according to the motion vector.

At this time, the motion prediction/compensation unit 47 calculates the cost function values of all the inter prediction modes that are candidates based on the images supplied from the screen rearrangement buffer 32 and the predicted images, and predicts the inter prediction with the smallest cost function value. mode is determined as the best inter prediction mode. Subsequently, the motion prediction/compensation unit 47 supplies the cost function value of the optimum inter prediction mode, and the corresponding predicted image, to the predicted image selection unit 48 . In addition, when the motion prediction/compensation unit 47 is notified of the selection of the predicted image generated in the optimum inter prediction mode from the predicted image selection unit 48, the motion prediction/compensation unit 47 converts the inter prediction mode information, the corresponding motion vector, etc. It is output to the lossless encoding unit 36 , and the reference image specifying information is output to the reference image setting unit 49 .

The predicted image selection unit 48 selects one of the optimal intra prediction mode and the optimal inter prediction mode corresponding to the smaller cost function value according to the cost function value supplied from the intra prediction unit 46 and the motion prediction/compensation unit 47, Determined as the best prediction mode. Subsequently, the predicted image selection unit 48 supplies the predicted image of the optimum prediction mode to the calculation unit 33 and the addition unit 40 . In addition, the prediction image selection unit 48 notifies the intra prediction unit 46 or the motion prediction/compensation unit 47 of the selection of the prediction image of the optimal prediction mode.

The reference image setting unit 49 holds the reference image designation information supplied from the motion prediction/compensation unit 47 corresponding to the GOP. When the current coded picture is the first picture of the GOP, the reference picture setting unit 49 supplies "0" which is the index of the RPS, and an RPS flag indicating that the RPS of the current coded picture is the RPS included in the SPS to the lossless encoding unit 36 .

On the other hand, when the current coded picture is a picture other than the first picture of the GOP, the reference picture setting section 49 compares the reference picture specifying information of the previous picture and the reference picture specifying information of the current coded picture with each other. , and according to the comparison result, determine inter_ref_pic_set_prediction_flag and delta_idx_minus1. Subsequently, the reference picture setting unit 49 sets the RPS including the determined inter_ref_pic_set_prediction_flag, and the reference picture designation information of the current coded picture or delta_idx_minus1, as the RPS of the current coded picture.

Subsequently, in the case where the same RPS as the RPS of the current coded picture is supplied from the setting unit 12, the reference picture setting unit 49 sets the index of the RPS, and the RPS flag indicating that the RPS of the current coded picture is the RPS contained in the SPS Provided to the lossless encoding unit 36 . On the other hand, when the same RPS as the RPS of the current coded picture is not supplied from the setting unit 12, the reference picture setting unit 49 sets the RPS of the current coded picture and the RPS indicating that the current coded picture is not included in the SPS The RPS flag of the RPS is supplied to the lossless encoding unit 36 .

The rate control unit 50 determines the quantization parameter used by the quantization unit 35 based on the encoded data stored in the accumulation buffer 37 so that overflow or underflow does not occur. The rate control unit 50 supplies the determined quantization parameter to the quantization unit 35, the lossless encoding unit 36 and the inverse quantization unit 38.

(SPS syntax example)

FIG. 5 is a diagram illustrating an example of the syntax of the SPS set by the setting unit 12 illustrated in FIG. 3 .

As illustrated in line 18 in FIG. 5 , the RPS of each index (i) is contained in the SPS.

(Syntax example of RPS)

FIG. 6 is a diagram illustrating an example of the syntax of the RPS.

Although not illustrated in the figure, the description of the sixth and subsequent rows illustrated in FIG. 6 is the same as that of the third and subsequent rows illustrated in FIG. 1 .

As illustrated in lines 2 and 3 in FIG. 6 , in an RPS whose index (idx) is 0, inter_ref_pic_set_prediction_flag is not included, but reference picture designation information included when inter_ref_pic_set_prediction_flag is "0" is included.

On the other hand, as illustrated in lines 4 and 5, in the RPS whose index (idx) is different from "0", inter_ref_pic_set_prediction_flag is included. Subsequently, when inter_ref_pic_set_prediction_flag is "0", reference picture designation information is included. On the other hand, when inter_ref_pic_set_prediction_flag is "1", delta_idx_minus1 is included.

(Explanation of the advantages of the present technology)

FIG. 7 is a diagram illustrating the information amount of the RPS set by the setting unit 12 illustrated in FIG. 3 , and FIG. 8 is a diagram illustrating the information amount of the conventional RPS.

In the examples illustrated in Figs. 7 and 8, the reference picture designation information of the 2nd picture and the 8th picture from the beginning within the GOP is the same as the reference picture designation information of the previous picture in coding order.

In this case, as illustrated in FIG. 7 , the setting unit 12 sets the reference image designation information of the first picture of the GOP as the RPS whose index is “0”. In addition, as the RPS whose index is "1", the setting unit 12 sets "1" as inter_ref_pic_set_prediction_flag, and sets "0" as delta_idx_minus1. Therefore, the index of the RPS of the first picture of the GOP is set to "0", and the indices of the RPS of the second picture and the eighth picture are set to "1".

In contrast, as illustrated in FIG. 8 , in the conventional case, for example, as an RPS whose index is “0”, “0” as inter_ref_pic_set_prediction_flag is set, and reference picture designation information of the 1st picture of the GOP is set. In addition, similarly to the case of the setting unit 12, the RPS whose index is "1" is set. Therefore, the index of the first picture of the GOP is set to "0", and the indices of the RPS of the second picture and the eighth picture are set to "1".

As described above, the setting unit does not set inter_ref_pic_set_prediction_flag which is the RPS whose index is "0" used as the RPS of the first picture. In other words, since the 1st picture of the GOP does not have any preceding pictures in coding order, inter_ref_pic_set_prediction_flag must be "0". Therefore, the setting section 12 does not set the inter_ref_pic_set_prediction_flag of the RPS whose index is "0" serving as the RPS of the first picture, but sets only the reference picture specifying information because the inter_ref_pic_set_prediction_flag is "0". As a result, the information amount of the RPS can be reduced by the amount corresponding to the inter_ref_pic_set_prediction_flag of the first picture compared to the conventional case.

(Example of the syntax of the slice header)

FIG. 9 is a diagram illustrating an example of the syntax of the slice header.

As illustrated in row 5 in Fig. 9, in the slice header, the RPS flag (short_term_ref_pic_set_sps_flag) of the corresponding coefficient is included. In addition, as illustrated in the 6th row and the 7th row in FIG. 9, when the RPS flag is "0", indicating that the RPS of the current coded picture is not the RPS included in the SPS, in the slice header, including The RPS of the corresponding coefficient, as short_term_ref_pic_set(num_short_term_ref_pic_sets).

On the other hand, as illustrated in lines 8 and 9 in FIG. 9 , when the RPS flag is "1", indicating that the RPS of the currently encoded picture is the RPS contained in the SPS, in the slice header , containing the index of the RPS of the corresponding coefficient, as short_term_ref_pic_set_idx (num_short_term_ref_pic_sets).

(Explanation of the processing of the encoding device)

FIG. 10 is a flowchart illustrating generation processing performed by the encoding apparatus 10 illustrated in FIG. 3 .

In step S11 illustrated in FIG. 10 , the setting unit 12 of the encoding device 10 performs an RPS setting process of setting the RPS. This RPS setting process will be described in detail later with reference to Fig. 11 described later. In step S12, the encoding section 11 performs an encoding process of encoding an image that is externally input as an input signal and configured in frame units according to the HEVC method. This encoding process will be described in detail later with reference to FIGS. 12 and 13 described later.

In step S13, the setting unit 12 sets the SPS including the RPS to which the index is assigned. In step S14, the setting unit 12 sets the PPS. In step S15, the setting unit 12 generates an encoded stream based on the set SPS and PPS and the encoded data supplied from the encoding unit 12. The setting unit 12 supplies the encoded stream to the transmission unit 13.

In step S16, the transmission unit 13 transmits the encoded stream supplied from the setting unit 12 to the decoding device described later, and then ends the processing.

FIG. 11 is a flowchart illustrating in detail the RPS setting process represented in step S11 illustrated in FIG. 10 .

In step S21 illustrated in Fig. 11, the setting unit 12 sets the index i of the RPS to "0". In step S22, it is determined whether or not the index i of the RPS is "0". When the index i of the RPS is determined to be "0" in step S22, the setting unit 12 sets the inter_ref_pic_set_prediction_flag to "0" in step S23, and the process proceeds to step S25.

On the other hand, when the index i of the RPS is determined not to be "0" in step S22, the setting unit 12 sets the RPS of the index i to inter_ref_pic_set_prediction_flag in step S24, and the process proceeds to step S25.

In step S25, the setting unit 12 determines whether or not inter_ref_pic_set_prediction_flag is "1". When it is determined in step S25 that inter_ref_pic_set_prediction_flag is "1", in step S26 the setting unit 12 sets delta_idx_minus1 as the RPS of the index i, and the process proceeds to step S28.

On the other hand, when it is determined in step S25 that inter_ref_pic_set_prediction_flag is not "1", in other words, when inter_ref_pic_set_prediction_flag is "0", in step S27 the setting unit 12 sets the reference picture specifying information, and the process proceeds to step S28 .

In step S28, the setting unit 12 increments the index i by one. In step S29, the setting unit 12 determines whether or not the index i is equal to or greater than the number num_short_term_ref_pic_sets of the RPS contained in the SPS.

When it is determined in step S29 that the index i is not equal to or greater than the number num_short_term_ref_pic_sets, the process returns to step S22, and the processes of steps S22-S29 are repeated until the index i is equal to or greater than the number num_short_term_ref_pic_sets.

On the other hand, when it is determined in step S29 that the index i is equal to or greater than the number num_short_term_ref_pic_sets, the process returns to step S11 illustrated in Fig. 10 and then proceeds to step S12.

12 and 13 show flowcharts illustrating the encoding process of step S12 illustrated in FIG. 10 in detail.

In step S31 illustrated in FIG. 12 , the A/D converter 31 of the encoding unit 11 performs A/D conversion of the frame-unit image input as an input signal, and then outputs the converted image to screen rearrangement buffer 32 to be stored in it.

In step S32, the screen rearrangement buffer 32 rearranges the saved frame images arranged in the display order according to the structure of the GOP and in the encoding order. The screen rearrangement buffer 32 supplies the rearranged image constituted in units of frames to the calculation unit 33 , the intra prediction unit 46 and the motion prediction/compensation unit 47 .

In step S33, the intra prediction unit 46 performs intra prediction processing of all intra prediction modes that are candidates. In addition, the intra prediction unit 46 calculates cost function values of all intra prediction modes that are candidates based on the image read from the screen rearrangement buffer 32 and the predicted image generated by the intra prediction process. Subsequently, the intra prediction unit 46 determines the intra prediction mode whose cost function value is the smallest as the optimal intra prediction mode. The intra prediction unit 46 supplies the predicted image generated in accordance with the optimal intra prediction mode, and the corresponding cost function value to the predicted image selection unit 48 .

In addition, the motion prediction/compensation unit 47 performs motion prediction/compensation processing of all the inter prediction modes that are candidates. Further, the motion prediction/compensation unit 47 calculates the cost function values of all the inter prediction modes as candidates based on the images supplied from the screen rearrangement buffer 32, and the predicted images, and predicts the inter prediction whose cost function value is the smallest mode is determined as the best inter prediction mode. Subsequently, the motion prediction/compensation unit 47 supplies the cost function value of the optimum inter prediction mode, and the corresponding predicted image, to the predicted image selection unit 48 .

In step S34, the predicted image selection unit 48 uses the cost function values supplied from the intra prediction unit 46 and the motion prediction/compensation unit 47 in the process of step S33 to select the optimal intra prediction mode whose cost function value is the smallest and the One of the best inter prediction modes is determined as the best prediction mode. Subsequently, the predicted image selection unit 48 supplies the predicted image of the optimum prediction mode to the calculation unit 33 and the addition unit 40 .

In step S35, the predicted image selection unit 48 determines whether or not the optimum prediction mode is the optimum inter prediction mode. When it is determined in step S35 that the optimal prediction mode is the optimal inter prediction mode, the prediction image selection unit 48 notifies the motion prediction/compensation unit 47 of the selection of the prediction image generated in the optimal inter prediction mode.

Subsequently, in step S36, the motion prediction/compensation unit 47 supplies the inter prediction mode information and the corresponding motion vector to the lossless encoding unit 36. The motion prediction/compensation unit 47 supplies the reference image designation information to the reference image setting unit 49 .

In step S37, the reference image setting unit 49 performs RPS index determination processing of determining the index of the RPS. This RPS index determination process will be described in detail later with reference to FIG. 14 described later.

On the other hand, in step S35, when it is determined that the optimal prediction mode is not the optimal inter prediction mode, in other words, when the optimal prediction mode is the optimal intra prediction mode, the predicted image selection unit 48 selects the optimal prediction mode according to the optimal prediction mode. The selection of the predicted image generated by the intra prediction mode is notified to the intra prediction unit 46 . Subsequently, in step S38, the intra prediction unit 46 supplies the intra prediction mode information to the lossless encoding unit 36, and the process proceeds to step S39.

In step S39, the calculation unit 33 subtracts the predicted image supplied from the predicted image selection unit 48 from the image supplied from the screen rearrangement buffer 32, thereby performing encoding. The calculation unit 33 outputs the image obtained as a result to the orthogonal transformation unit 34 as residual information.

In step S40 , the orthogonal transform unit 34 orthogonally transforms the residual information output from the calculation unit 33 , and supplies the orthogonal transform system obtained as a result to the quantization unit 35 .

In step S41 , the quantization unit 35 quantizes the coefficient supplied from the orthogonal transform unit 34 by using the quantization parameter supplied from the rate control unit 50 . The quantized coefficients are input to the lossless encoding unit 36 and the inverse quantization unit 38 .

In step S42 illustrated in Fig. 13 , the inverse quantization unit 38 performs inverse quantization of the quantization coefficient supplied from the quantization unit 35 by using the quantization parameter supplied from the rate control unit 50, and transforms the orthogonal transform obtained as a result thereof The coefficients are supplied to the inverse orthogonal transform unit 39 .

In step S43 , the inverse orthogonal transform unit 39 performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization unit 38 , and supplies the residual information obtained as a result to the addition unit 40 .

In step S44, the adding unit 40 adds the residual information supplied from the inverse orthogonal transform unit 39 and the predicted image supplied from the predicted image selection unit 48, thereby obtaining a local decoded image. The adding unit 40 supplies the obtained image to the deblocking filter 41 and the frame memory 44 .

In step S45 , the deblocking filter 41 performs deblocking filter processing on the locally decoded image supplied from the adding unit 40 . The deblocking filter 41 supplies the image obtained as a result thereof to the adaptive offset filter 42 .

In step S46, the adaptive offset filter 42 performs adaptive offset filter processing on the image supplied from the deblocking filter 41 for each LCU. The adaptive offset filter 42 supplies the image obtained as a result thereof to the adaptive loop filter 43 . In addition, for each LCU, the adaptive offset filter 42 supplies the storage flag, the index or offset, and the category information to the lossless encoding unit 36 as offset filter information.

In step S47, the adaptive loop filter 43 performs adaptive loop filter processing on the image supplied from the adaptive offset filter 42 for each LCU. The adaptive loop filter 43 supplies the image obtained as a result thereof to the frame memory 44 . In addition, the adaptive loop filter 43 supplies the lossless encoding unit 36 with filter coefficients used in the adaptive loop filter processing.

In step S48, the frame memory 44 holds the image supplied from the adaptive loop filter 43 and the image supplied from the adding unit 40. The image stored in the frame memory 44 is output to the intra prediction unit 46 or the motion prediction/compensation unit 47 through the switch 45 as a reference image.

In step S49, the lossless encoding unit 36 compares the quantization parameters, offset filter information and filter coefficients supplied from the rate control unit 50, such as intra prediction mode information or inter prediction mode information, motion vector, index of RPS or RPS, etc., Perform lossless encoding as encoded information.

In step S50 , the lossless encoding unit 36 performs lossless encoding on the quantized coefficient supplied from the quantization unit 35 . Subsequently, the lossless encoding unit 36 generates encoded data based on the encoding information and coefficients encoded in the lossless manner in step S49.

In step S51, the accumulation buffer 37 temporarily holds the encoded data supplied from the lossless encoding unit 36.

In step S52, the rate control unit 50 determines the quantization parameter used by the quantization unit 35 based on the encoded data stored in the accumulation buffer 37 so that overflow or underflow does not occur. The rate control unit 50 supplies the determined quantization parameters to the quantization unit 35 , the lossless encoding unit 36 and the inverse quantization unit 38 .

In step S53, the accumulation buffer 37 outputs the held encoded data to the setting unit 12 illustrated in Fig. 3 .

In the encoding process illustrated in FIGS. 12 and 13 , in order to simplify the explanation, although both the intra prediction process and the motion prediction/compensation process are configured to be continuously performed, in practice, it is possible to etc., do only one of them.

FIG. 14 is a flowchart illustrating in detail the RPS index determination process represented in step S37 illustrated in FIG. 12 .

In step S71 illustrated in FIG. 14 , the reference image setting unit 49 holds the reference image designation information supplied from the motion prediction/compensation unit 47 in correspondence with the GOP. In step S72, the reference image setting unit 49 determines whether or not the currently encoded image is the first image of the GPS.

When it is determined in step S72 that the currently encoded picture is the first picture in the GOP, in step S73 the reference picture setting unit 49 sets the RPS flag to "1". In step S74, the reference image setting unit 49 sets the index of the RPS to "0", and the process proceeds to step S79.

On the other hand, when it is determined in step S72 that the current coded picture is a picture other than the first picture in the GOP, in step S75, the reference picture setting unit 49 generates the RPS of the currently coded picture.

More specifically, the reference picture setting unit 49 determines whether or not the reference picture specification information of the held previous picture and the reference picture specification information of the currently encoded picture are the same. When it is determined that the reference picture specifying information of the previous picture held and the reference picture specifying information of the current coded picture are the same, the reference picture setting unit 49 generates the RPS of the current coded picture which contains "1" as inter_ref_pic_set_prediction_flag and contains delta_idx_minus1 .

On the other hand, when it is determined that the reference picture specifying information of the held previous picture and the reference picture specifying information of the current coded picture are not identical, the reference picture setting unit 49 generates the RPS of the current coded picture including "0" as inter_ref_pic_set_prediction_flag.

In step S76 , the reference picture setting unit 49 determines whether or not the RPS of the currently encoded picture is the same as the RPS contained in the SPS supplied from the setting unit 12 . When it is determined in step S76 that the RPS of the currently encoded picture and the RPS contained in the SPS are the same, in step S77 the reference picture setting unit 49 sets the RPS flag to "1".

In step S78, the reference picture setting unit 49 identifies the index of the same RPS contained in the SPS as the RPS of the currently encoded picture, and the process proceeds to step S79. In step S79, the reference picture setting unit 49 supplies the RPS flag set in step S73 or step S77, and the index of the RPS set in step S74, or the index of the RPS identified in step S78 to lossless encoding unit 36. Subsequently, the process returns to step S37 illustrated in Fig. 12 , and then the process proceeds to step S39.

On the other hand, when it is determined in step S76 that the RPS of the currently encoded picture and the RPS contained in the SPS are different, the reference picture setting unit 49 sets the RPS flag to "0". In step S81, the reference image setting unit 49 supplies the RPS flag set in step S80 and the RPS generated in step S75 to the lossless encoding unit 36. Subsequently, the process returns to step S37 illustrated in Fig. 12 , and then the process proceeds to step S39.

As above, when the currently encoded picture is a picture other than the first picture in the GOP, the encoding device 10 transmits the inter_ref_pic_set_prediction_flag. In other words, the encoding device 10 does not transmit the inter_ref_pic_set_prediction_flag when the currently encoded picture is the first picture of the GOP. Therefore, the information amount of the RPS related to the reference picture designation information can be reduced by the amount corresponding to the inter_ref_pic_set_prediction_flag of the first picture of the GOP.

(Configuration example of decoding apparatus according to first embodiment)

FIG. 15 is a block diagram illustrating a structural example of a decoding apparatus that decodes an encoded stream transmitted from the encoding apparatus 10 illustrated in FIG. 3 to which the present technology according to the first embodiment is applied.

The decoding apparatus 110 illustrated in FIG. 15 is constituted by a receiving unit 111 , an extracting unit 112 , and a decoding unit 113 .

The receiving unit 111 of the decoding device 110 receives the encoded stream transmitted from the encoding device 10 illustrated in FIG. 3 , and supplies the received encoded stream to the extraction unit 112 .

The extraction unit 112 extracts SPS, PPS, encoded data, and the like from the encoded stream supplied from the reception unit 111 . The extraction unit 112 supplies the encoded data to the decoding unit 113 . In addition, the extraction unit 112 obtains inter_ref_pic_set_prediction_flag of each SPS, and delta_idx_minus1 or reference picture designation information from the SPS, and supplies the obtained information to the decoding unit 113. In addition, the extracting unit 112 supplies information other than the RPS, PPS, etc., contained in the SPS, to the decoding unit 113, as necessary.

Based on the inter_ref_pic_set_prediction_flag and delta_idx_minus1 of each RPS supplied from the extraction unit 112 or the reference picture designation information, the decoding unit 113 decodes the encoded data supplied from the extraction unit 112 according to the HEVC method. At this time, the decoding unit 113 refers to information other than the RPS, the PPS, and the like contained in the SPS, as necessary. The decoding unit 113 outputs the image obtained by decoding as an output signal.

(Example of structure of decoding unit)

FIG. 16 is a block diagram illustrating a structural example of the decoding unit 13 illustrated in FIG. 15 .

The decoding unit 113 illustrated in FIG. 16 is composed of the following components: an accumulation buffer 131; a lossless decoding unit 132; an inverse quantization unit 133; an inverse orthogonal transform unit 134; an addition unit 135; a deblocking filter 136; Shift filter 137; Adaptive loop filter 138; Screen rearrangement buffer 139; D/A converter 140; Frame memory 141; Switch 142; Intra prediction unit 143; Reference image setting unit 144; Motion compensation unit 145; and switch 146.

The accumulation buffer 131 of the decoding unit 113 receives encoded data from the extraction unit 112 illustrated in FIG. 15 and holds the received encoded data. The accumulation buffer 131 supplies the stored encoded data to the lossless decoding unit 132 .

The lossless decoding unit 132 performs lossless decoding such as variable-length decoding or arithmetic decoding on the encoded data supplied from the accumulation buffer 131, thereby obtaining quantized coefficients and encoding information. The lossless decoding unit 132 supplies the quantized coefficients to the inverse quantization unit 133 . In addition, the lossless decoding unit 132 supplies intra prediction mode information and the like to the intra prediction unit 143 as encoding information, and supplies the motion vector, inter prediction mode information, and the like to the motion compensation unit 145 . The lossless decoding unit 132 supplies the RPS flag and the RPS index or RPS to the reference image setting unit 144 as encoding information.

In addition, the lossless decoding unit 132 supplies intra prediction mode information or inter prediction mode information to the switch 146 as encoding information. The lossless decoding unit 132 supplies the offset filter information as encoded information to the adaptive offset filter 137 and supplies the filter coefficients to the adaptive loop filter 138 .

Inverse quantization unit 133, inverse orthogonal transform unit 134, addition unit 135, deblocking filter 136, adaptive offset filter 137, adaptive loop filter 138, frame memory 141, switch 142, intra prediction unit 13 and the motion compensation unit 145 perform the inverse quantization unit 38, the inverse orthogonal transform unit 39, the addition unit 40, the deblocking filter 41, the adaptive offset filter 42, the adaptive loop filter illustrated in FIG. 4 43, the frame memory 44, the switch 45, the intra prediction unit 46, and the motion prediction/compensation unit 47 are processed similarly, whereby the image is decoded.

More specifically, the inverse quantization unit 133 performs inverse quantization of the quantized coefficients supplied from the lossless decoding unit 132 , and supplies the orthogonal transform coefficients obtained as a result to the inverse orthogonal transform unit 134 .

Inverse orthogonal transform unit 134 performs inverse orthogonal transform on the orthogonal transform coefficients supplied from inverse quantization unit 133 . The inverse orthogonal transform unit 134 supplies the residual information obtained by the inverse orthogonal transform to the addition unit 135 .

The adding unit 135 functions as a decoding unit, and performs decoding by adding, as the current decoded image, the residual information supplied from the inverse orthogonal transform unit 134 , and the predicted image supplied from the switch 146 . The adding unit 135 supplies the image obtained by decoding to the deblocking filter 136 and the frame memory 141 . In addition, when the predicted image is not supplied from the switch 146, the adding unit 135 supplies the image that is the residual information supplied from the inverse orthogonal transform unit 134 to the deblocking filter 136 as an image obtained by decoding, and adds the image to the deblocking filter 136. Provided to the frame memory 141 to be stored therein.

The deblocking filter 136 performs adaptive deblocking filter processing on the image supplied from the adding unit 135 , and supplies the image obtained as a result to the adaptive offset filter 137 .

The adaptive offset filter 137 has a buffer that sequentially holds the offsets supplied from the lossless decoding unit 132 . In addition, the adaptive offset filter 137 performs adaptive offset filtering for each LCU on the image after the adaptive deblocking filter processing performed by the deblocking filter 136 based on the offset filter information supplied from the lossless decoding unit 132 deal with.

More specifically, when the storage flag included in the offset filter information is "0", the adaptive offset filter 137 performs the processing performed in the LCU unit by using the offset amount included in the offset filter information. The image after the deblocking filter process is subjected to the adaptive offset filter process of the type indicated by the type information.

On the other hand, when the storage flag included in the offset filter information is "1", the adaptive offset filter 137 reads the image after deblocking The offset of the position indicated by the index in the offset filter information. Then, the adaptive offset filter 137 performs adaptive offset filter processing of the category indicated by the category information by using the read offset. The adaptive offset filter 137 supplies the image after the adaptive offset filter processing to the adaptive loop filter 138 .

The adaptive loop filter 138 performs adaptive loop filter processing for each LCU on the image supplied from the adaptive offset filter 137 by using the filter coefficient supplied from the lossless decoding unit 132 . The adaptive loop filter 138 supplies the image obtained as a result thereof to the frame memory 141 and the screen rearrangement buffer 139 .

The screen rearrangement buffer 139 holds the image supplied from the adaptive loop filter 138 on a frame-by-frame basis. The screen rearrangement buffer 139 rearranges the stored images in units of frames arranged in the encoding order in the original order, and supplies the rearranged images to the D/A converter 140 .

The D/A converter 140 performs D/A conversion of the image supplied from the screen rearrangement buffer 139 constituted in units of frames, and outputs the converted image as an output signal. The frame memory 141 holds the image supplied from the adaptive loop filter 138 and the image supplied from the adding unit 135 . The image stored in the frame memory 141 is read as a reference image and supplied to the motion compensation unit 145 or the intra prediction unit 143 through the switch 142.

The intra prediction unit 143 performs intra prediction processing of the intra prediction mode indicated by the intra prediction mode information supplied from the lossless decoding unit 132 by using the reference image read from the frame memory 141 via the switch 142 . The intra prediction unit 143 supplies the predicted image generated as a result thereof to the switch 146 .

The reference picture setting unit 144 holds, as RPS information, inter_ref_pic_set_prediction_flag and delta_idx_minus1 of each RPS supplied from the extraction unit 112 illustrated in FIG. 15 or reference picture specifying information. In addition, the reference picture setting section 144 generates reference picture specifying information of the currently decoded picture based on the RPS flag and the RPS index or RPS supplied from the lossless decoding section 132, and the RPS information of each RPS. The reference image setting unit 144 supplies the generated reference image designation information to the motion compensation unit 145, and holds the reference image designation information.

The motion compensation unit 145 reads the reference image designated by the reference image designation information from the frame memory 141 through the switch 142 based on the reference image designation information supplied from the reference image setting unit 144 . The motion compensation unit 145 functions as a predicted image generation unit, and performs motion compensation processing of the optimum inter prediction mode indicated by the inter prediction mode information by using the motion vector and the reference image. The motion compensation unit 145 supplies the predicted image generated as a result thereof to the switch 146 .

When the intra prediction mode information is supplied from the lossless decoding unit 132 , the switch 146 supplies the prediction image supplied from the intra prediction unit 143 to the adding unit 135 . On the other hand, when the inter prediction mode information is supplied from the lossless decoding unit 132, the switch 146 supplies the prediction image supplied from the motion compensation unit 145 to the adding unit 135.

(Explanation of the processing of the decoding device)

FIG. 17 is a flowchart illustrating reception processing by the decoding apparatus 110 illustrated in FIG. 15 .

In step S111 illustrated in FIG. 17 , the receiving unit 111 of the decoding device 110 receives the encoded stream transmitted from the encoding device 10 illustrated in FIG. 3 and supplies the received encoded stream to the extraction unit 112 .

In step S112 , the extraction unit 112 extracts SPS, PPS, encoded data, and the like from the encoded stream supplied from the reception unit 111 . The extraction unit 112 supplies the encoded data to the decoding unit 113. In addition, the extracting unit 112 supplies information other than the RPS, PPS, etc. contained in the SPS to the decoding unit 113 as necessary.

In step S113 , the extraction unit 112 obtains, as RPS information, inter_ref_pic_set_prediction_flag of each RPS, and delta_idx_minus1 or reference picture specifying information according to the SPS, and supplies the obtained information to the decoding unit 113 .

In step S114, the decoding unit 113 performs decoding processing of decoding the encoded data supplied from the extraction unit 112 according to the HEVC method based on the RPS information of each RPS supplied from the extraction unit 112. This decoding process will be described in detail with reference to FIG. 19 described later. Then, the processing ends.

FIG. 18 is a flowchart illustrating in detail the RPS setting process represented in step S113 illustrated in FIG. 17 .

In step S120 illustrated in Fig. 18, the extraction unit 112 obtains num_short_term_ref_pic_sets contained in the SPS (Fig. 5). In step S121, the extraction unit 112 sets the index i of the RPS corresponding to the generated RPS information to "0". In step S122, it is determined whether or not the index i of the RPS is "0".

When it is determined in step S122 that the index i is "0", in step S123, the extraction unit 112 sets the inter_ref_pic_set_prediction_flag contained in the RPS information of the RPS of the index i to "0", and the process proceeds to step S125.

On the other hand, when it is determined in step S122 that the index i is not "0", in step S124 the extraction unit 112 obtains the inter_ref_pic_set_prediction_flag included in the RPS of the index i included in the SPS. Subsequently, the extraction unit 112 sets the obtained inter_ref_pic_set_prediction_flag as the inter_ref_pic_set_prediction_flag included in the RPS information of the RPS of the index i, and the process proceeds to step S125.

In step S125, the extraction unit 112 determines whether or not inter_ref_pic_set_prediction_flag is "1". When it is determined in step S125 that the inter_ref_pic_set_prediction_flag is "1", in step S126, the extraction unit 112 obtains delta_idx_minus1 included in the RPS of the index i included in the SPS. Subsequently, the extraction unit 112 sets the obtained delta_idx_minus1 as delta_idx_minus1 included in the RPS information of the RPS of index i, and the process proceeds to step S128.

On the other hand, when it is determined in step S125 that the inter_ref_pic_set_prediction_flag is not "1", in step S127 the extraction unit 112 obtains the reference picture designation information included in the RPS of the index i included in the SPS. Subsequently, the extraction unit 112 sets the obtained reference image designation information as the reference image designation information contained in the RPS information of the RPS of index i, and the process proceeds to step S128.

In step S128, the extraction unit 112 increments the index i by 1. In step S129, the extraction unit 112 determines whether the index i is equal to or greater than num_short_term_ref_pic_sets obtained in step S120.

When it is determined in step S129 that the index i is not equal to or greater than num_short_term_ref_pic_sets, the process returns to step S122, and the processes of steps S122-S129 are repeated until the index i is equal to or greater than num_short_term_ref_pic_sets.

On the other hand, when it is determined in step S129 that the index i is equal to or greater than num_short_term_ref_pic_sets, in step S130, the extraction unit 112 supplies the RPS information of the RPS whose number is the set num_short_term_ref_pic_sets. Subsequently, the process returns to step S113 illustrated in Fig. 17, and then the process proceeds to step S114.

FIG. 19 is a flowchart illustrating in detail the decoding process represented in step S114 illustrated in FIG. 17 .

In step S131 illustrated in FIG. 19 , the accumulation buffer 131 of the decoding unit 113 receives encoded data constituted in units of frames from the extraction unit 112 illustrated in FIG. 15 , and holds the received encoded data. The accumulation buffer 131 supplies the stored encoded data to the lossless decoding unit 132 .

In step S132, the lossless decoding unit 132 performs lossless decoding of the encoded data supplied from the accumulation buffer 131, thereby obtaining quantization coefficients and encoding information. The lossless decoding unit 132 supplies the quantized coefficients to the inverse quantization unit 133 . In addition, the lossless decoding unit 132 supplies intra prediction mode information and the like as encoding information to the intra prediction unit 143, and supplies motion vector, inter prediction mode information, RPS flag, RPS index or RPS, etc. to motion compensation unit 145.

In addition, the lossless decoding unit 132 supplies the inter prediction mode information or the intra prediction mode information to the switch 146 as encoding information. The lossless decoding unit 132 supplies the offset filter information as encoded information to the adaptive offset filter 137 and supplies the filter coefficients to the adaptive loop filter 138 .

In step S133 , the inverse quantization unit 133 performs inverse quantization of the quantized coefficients supplied from the lossless decoding unit 132 , and supplies the orthogonal transform coefficients obtained as a result to the inverse orthogonal transform unit 134 .

In step S134 , the motion compensation unit 145 determines whether the inter prediction mode information is supplied from the lossless decoding unit 132 . When it is determined in step S134 that the inter prediction mode information is supplied, the process proceeds to step S135.

In step S135, the reference picture setting unit 144 generates the reference picture specifying information of the current decoded picture based on the RPS information of each RPS supplied from the extraction unit 112, and the RPS flag and the index or RPS of the RPS supplied from the lossless decoding unit 132, And keep the generated reference image specifying information.

More specifically, the reference image setting unit 144 holds the RPS information of each RPS supplied from the extraction unit 112 . When the RPS flag is "1", the reference image setting unit 144 reads the index RPS information of the RPS contained in the held RPS information. Subsequently, when the inter_ref_pic_set_prediction_flag contained in the read RPS information is "0", the reference picture setting unit 144 generates the reference picture specifying information contained in the RPS information as the reference picture specifying information of the current decoded picture, and keeps generating The reference image specification information for .

On the other hand, when inter_ref_pic_set_prediction_flag is "1", the reference picture setting unit 144 reads the reference picture specifying information of the preceding picture specified by delta_idx_minus1 included in the RPS information from among the held reference picture specifying information. Subsequently, the reference image setting unit 144 generates and holds the read reference image designation information of the previous image as the reference image designation information of the current decoded image.

In addition, when the RPS flag is "0" and the inter_ref_pic_set_prediction_flag contained in the RPS supplied from the lossless decoding unit 132 together with the RPS flag is "0", the reference picture setting unit 144 generates reference picture designation information contained in the RPS , as the reference picture specification information of the current decoded picture, and maintain the generated reference picture specification information. On the other hand, when inter_ref_pic_set_prediction_flag is "1", the reference picture setting section 144 reads the reference picture specification information of the preceding picture specified by delta_idx_minus1 included in the RPS from the held reference picture specification information. Subsequently, the reference image setting unit 144 generates the read reference image designation information of the preceding image as the reference image designation information of the currently decoded image, and holds the generated reference image designation information.

In step S136, the motion compensation unit 145 reads the reference image based on the reference image designation information supplied from the reference image setting unit 144, and performs an optimal inter frame represented by the inter prediction mode information by using the motion vector and the reference image Motion compensation processing for prediction mode. The motion compensation unit 145 supplies the predicted image generated as a result thereof to the addition unit 135 through the switch 146, and the process proceeds to step S138.

On the other hand, when it is determined in step S134 that the inter prediction mode information is not supplied, in other words, when the intra prediction mode information is supplied to the intra prediction unit 143, the process proceeds to step S137.

In step S137, the intra prediction unit 143 performs intra prediction processing of the intra prediction mode indicated by the intra prediction mode information by using the reference image read from the frame memory 141 via the switch 142. The intra prediction unit 143 supplies the predicted image generated by the intra prediction process to the addition unit 135 through the switch 146, and the process proceeds to step S138.

In step S138 , the inverse orthogonal transform unit 134 performs inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization unit 133 , and supplies the residual information obtained as a result to the addition unit 135 .

In step S139 , the adding unit 135 adds the residual information supplied from the inverse orthogonal transform unit 134 and the predicted image supplied from the switch 146 . The adding unit 135 supplies the image obtained as a result thereof to the deblocking filter 136 and supplies the obtained image to the frame memory 141 .

In step S140, the deblocking filter 136 performs deblocking filter processing on the image supplied from the adding unit 135, thereby eliminating block distortion. The deblocking filter 136 supplies the image obtained as a result thereof to the adaptive offset filter 137 .

In step S141, the adaptive offset filter 137 performs adaptive offset filtering for each LCU on the image after the deblocking filter processing performed by the deblocking filter 136 based on the offset filter information supplied from the lossless decoding unit 132 deal with. The adaptive offset filter 137 supplies the image after the adaptive offset filter processing to the adaptive loop filter 138 .

In step S142, the adaptive loop filter 138 performs adaptive loop filter processing for each LCU on the image supplied from the adaptive offset filter 137 by using the filter coefficient supplied from the lossless decoding unit 132. The adaptive loop filter 138 supplies the image obtained as a result thereof to the frame memory 141 and the screen rearrangement buffer 139 .

In step S143 , the frame memory 141 holds the image supplied from the adding unit 135 and the image supplied from the adaptive loop filter 138 . The image stored in the frame memory 141 is supplied to the motion compensation unit 145 or the intra prediction unit 143 through the switch 142 as a reference image.

In step S144, the screen rearrangement buffer 139 stores the images supplied from the adaptive loop filter 138 in units of frames, and rearranges the stored images in units of frames in the encoding order in the original display order, The rearranged image is then supplied to the D/A converter 140 .

In step S145, the D/A converter 140 performs D/A conversion on the image supplied from the screen rearrangement buffer 139 constituted in units of frames, and outputs the converted image as an output signal. Subsequently, the process returns to step S114 illustrated in FIG. 17 , and then the process ends.

As above, the decoding apparatus 110 receives the inter_ref_pic_set_prediction_flag transmitted when the current encoded picture is a picture other than the first picture of the GOP. When receiving the inter_ref_pic_set_prediction_flag, the decoding apparatus 110 generates reference picture designation information of the current decoded picture according to the inter_ref_pic_set_prediction_flag. On the other hand, when the inter_ref_pic_set_prediction_flag is not received, the decoding device 110 generates the reference picture designation information of the current decoded picture based on "0" as the inter_ref_pic_set_prediction_flag.

As a result, the decoding apparatus 110 can decode the encoded stream in which the information amount of the RPS is reduced by the amount corresponding to the inter_ref_pic_set_prediction_flag of the first picture of the GOP.

<Second Embodiment>

(Example of the structure of the encoding apparatus according to the second embodiment)

20 is a block diagram illustrating a structural example of an encoding apparatus to which the present technology according to the second embodiment is applied.

Here, the same reference numerals are assigned to each of the structures illustrated in FIG. 20 that are the same as those illustrated in FIG. 3 , and repeated explanations thereof are omitted.

The structure of the encoding device 150 illustrated in FIG. 20 differs from the structure of the encoding device 10 illustrated in FIG. 3 in that the setting unit 151 is set instead of the setting unit 12 . The encoding apparatus 150 sets the SPS so that inter_ref_pic_set_prediction_flag and delta_idx_minus1 can be shared in units of GOPs.

More specifically, the setting unit 151 sets an RPS including inter_ref_pic_set_prediction_flag, delta_idx_minus1, reference picture designation information, and the like, as necessary, and assigns an index to each RPS. The setting unit 151 supplies the indexed RPS to the encoding unit 11 . In addition, setting section 151 sets the SPS including delta_idx_minus1 common to all pictures in the GOP including non-reference information indicating whether inter_ref_pic_set_prediction_flag is "0" in all pictures in the RPS and GOP, if necessary. The setting unit 151 sets PPS and the like.

In addition, similar to the setting unit 12 illustrated in FIG. 3 , the setting unit 151 generates an encoded stream based on the SPS and PPS that have been set, and the encoded data supplied from the encoding unit 11 . Similar to the setting unit 12 , the setting unit 151 supplies the encoded stream to the transmission unit 13 .

(SPS syntax example)

FIG. 21 is a diagram illustrating an example of the syntax of the SPS set by the setting unit 151 illustrated in FIG. 20 .

As illustrated in the 4th row in FIG. 21 , unreferenceable information (disable_rps_prediction_flag) is included in the SPS. In addition, as illustrated in the 5th and 6th lines, when the unreferenceable information is "0" that does not indicate that inter_ref_pic_set_prediction_flag is "0" in all pictures within the GOP, the SPS includes the information that is within the GOP. Consistency information (unified_rps_prediction_control_present_flag) whether delta_idx_minus1 is the same in all pictures.

Also, as illustrated in the 7th and 8th lines, when the coincidence information is "1", indicating that delta_idx_minus1 is the same in all pictures within the GOP, the delta_idx_minus1 that is common to all pictures within the GOP is included in the SPS unified_delta_idx_minus1. In addition, as illustrated in line 11, the RPS of each index (i) is included in the SPS.

(Syntax example of RPS)

FIG. 22 is a diagram illustrating an example of the syntax of the RPS.

The description of the 11th and subsequent rows illustrated in FIG. 22 is the same as that of the fifth and subsequent rows illustrated in FIG. 1 .

As illustrated in lines 2 and 3 in FIG. 22 , when the disable_rps_prediction_flag is "1", in the RPS, the inter_ref_pic_set_prediction_flag is not included, but the reference picture designation included when the inter_ref_pic_set_prediction_flag is "0" is included information.

On the other hand, as illustrated in lines 4 and 5, when disable_rps_prediction_flag is "0", the RPS includes inter_ref_pic_set_prediction_flag. Also, as illustrated in lines 6 to 8, when inter_ref_pic_set_prediction_flag and unified_rps_prediction_control_present_flag are "1", respectively, in RPS, delta_idx_minus1 is not included, and delta_idx_minus1 is unified_delta_idx_minus1.

Also, as illustrated in the 9th and 10th lines, in the case where inter_ref_pic_set_prediction_flag is "1" and unified_rps_prediction_control_present_flag is "0", delta_idx_minus1 is included in the RPS.

(Explanation of the advantages of the present technology)

23 and 24 are diagrams illustrating the information amount of the RPS set by the setting unit 151 illustrated in FIG. 20, and FIG. 25 is a diagram illustrating the information amount of the conventional RPS.

In the example illustrated in FIG. 23 , the reference picture designation information of each of the 2nd picture and the 8th picture from the beginning within the GOP is the same as the reference picture designation information of the corresponding previous picture in coding order.

In this case, as illustrated in FIG. 23, the setting unit 151 sets "0" as disable_rps_prediction_flag, and sets "1" as unified_rps_prediction_control_present_flag. In addition, the setting unit 151 sets "0" as unified_delta_idx_minus1.

Further, for example, as the RPS whose index is "0", the setting section 151 sets "0" as the inter_ref_pic_set_prediction_flag, and sets the reference picture specification information of the first picture of the GOP. In addition, as the RPS whose index is "1", the setting unit 151 sets "1" as inter_ref_pic_set_prediction_flag. Therefore, the index of the RPS of the first picture of the GOP is set to "0", and the indices of the RPS of the second picture and the eighth picture are set to "1".

As above, the setting unit 151 sets delta_idx_minus1 common to all pictures within the GOP to unified_delta_idx_minus1. Therefore, the setting unit 151 can set delta_idx_minus1 in units of GOPs.

In addition, in the examples illustrated in FIGS. 24 and 25 , the reference picture designation information of all pictures within the GOP is not the same as the reference picture designation information of the corresponding previous picture in encoding order.

In this case, as illustrated in FIG. 24 , the setting unit 151 sets "1" as the disable_rps_prediction_flag, and as the RPS corresponding to each picture within the GOP, sets the reference picture designation information of the picture. In contrast, in the conventional case, as illustrated in FIG. 25 , as the RPS corresponding to each picture within the GOP, "0" is set as inter_ref_pic_set_prediction_flag, and the reference picture designation information of the picture is set.

As described above, the setting section 151 sets "0" which is inter_ref_pic_set_prediction_flag common to all pictures in the GOP to disable_rps_prediction_flag. Thus, in the case where the disable_rps_prediction_flag is "1", the information amount of the RPS can be reduced by the amount corresponding to the inter_ref_pic_set_prediction_flag compared to the normal case.

(Explanation of the processing of the encoding device)

The generation processing by the encoding device 150 illustrated in FIG. 20 is the same as the generation processing illustrated in FIG. 10 except for the RPS setting processing, so that only the RPS setting processing will be described below.

FIG. 26 is a flowchart illustrating the RPS setting process performed by the setting unit 151 of the encoding device 150 in detail.

In step S161 illustrated in FIG. 26 , the setting unit 151 sets disable_rps_prediction_flag as the SPS. In step S162, the setting unit 151 determines whether the disable_rps_prediction_flag is "1". When it is determined in step S162 that disable_rps_prediction_flag is not "1", in step S163, the setting unit 151 sets unified_rps_prediction_control_present_flag as the SPS.

In step S164, the setting unit 151 determines whether or not unified_rps_prediction_control_present_flag is "1". When it is determined in step S164 that unified_rps_prediction_control_present_flag is "1", in step S165, the setting unit 151 sets unified_delta_idx_minus1 as the SPS, and the process proceeds to step S166.

When it is determined in step S162 that disable_rps_prediction_flag is "1", or when it is determined that unified_rps_prediction_control_present_flag is "0" in step S164, the process proceeds to step S166.

In step S166, the setting unit 151 sets the index i of the RPS to "0". In step S167, the setting unit 151 determines whether the disable_rps_prediction_flag is "1". When it is determined in step S167 that disable_rps_prediction_flag is "1", in step S168, the setting unit 151 sets the inter_ref_pic_set_prediction_flag to "0", and the process proceeds to step S170.

On the other hand, when it is determined in step S167 that disable_rps_prediction_flag is not "1", in step S169, the setting unit 151 sets inter_ref_pic_set_prediction_flag of the RPS as index i, and the process proceeds to step S170.

In step S170, the setting unit 151 determines whether or not inter_ref_pic_set_prediction_flag is "1". When it is determined in step S170 that inter_ref_pic_set_prediction_flag is "1", in step S171, the setting unit 151 determines whether unified_rps_prediction_control_present_flag is "1".

When it is determined in step S171 that unified_rps_prediction_control_present_flag is "1", the process proceeds to step S174. On the other hand, when it is determined in step S171 that unified_rps_prediction_control_present_flag is not "1", in step S172, the setting unit 151 sets delta_idx_minus1 as the RPS of index i, and the process proceeds to step S174.

In addition, when it is determined in step S170 that inter_ref_pic_set_prediction_flag is not "1", in step S173, the setting unit 151 sets the reference picture specifying information as the RPS of the index i, and the process proceeds to step S174.

In step S174, the setting unit 151 increments the index i by 1. In step S175, the setting unit 151 determines whether the index i is equal to or greater than the number num_short_term_ref_pic_sets of RPS included in the SPS.

When it is determined in step S175 that the index i is not equal to or greater than num_short_term_ref_pic_sets, the process returns to step S167, and the processes of steps S167-S175 are repeated until the index i is equal to or greater than the number num_short_term_ref_pic_sets.

On the other hand, when it is determined in step S175 that the index i is num_short_term_ref_pic_sets, the RPS setting process ends.

As above, since the encoding apparatus 150 sets disable_rps_prediction_flag, when disable_rps_prediction_flag is "1", the information amount of the RPS related to the reference picture designation information can be reduced by the amount corresponding to inter_ref_pic_set_prediction_flag compared to the normal case. In addition, inter_ref_pic_set_prediction_flag can be set in GOP units.

Furthermore, since the encoding apparatus 150 sets delta_idx_minus1 common to all pictures within a GOP as unified_delta_idx_minus1, it is possible to set delta_idx_minus1 in units of GOPs.

(Configuration example of decoding apparatus according to second embodiment)

FIG. 27 is a block diagram illustrating a structural example of a decoding apparatus that decodes an encoded stream transmitted from the encoding apparatus 150 illustrated in FIG. 20 to which the present technology according to the second embodiment is applied.

Here, the same reference numerals are assigned to each of the structures illustrated in FIG. 27 that are the same as those illustrated in FIG. 15 , and repeated explanations thereof are omitted.

The structure of the decoding apparatus 170 illustrated in FIG. 27 differs from the structure of the decoding apparatus 110 illustrated in FIG. 15 in that the extraction unit 171 is provided instead of the extraction unit 112 . The decoding device 170 sets the RPS information of each RPS according to the SPS illustrated in FIG. 21 .

More specifically, similar to the extraction unit 112 illustrated in FIG. 15 , the extraction unit 171 of the decoding device 170 extracts SPS, PPS, encoded data, and the like from the encoded stream supplied from the reception unit 111 . Similar to the extraction unit 112 , the extraction unit 171 supplies the encoded data to the decoding unit 113 . In addition, according to the SPS illustrated in Fig. 21 , the extraction unit 171 obtains the RPS information of each RPS, and supplies the obtained RPS information to the decoding unit 113. Further, similarly to the extraction unit 112, the extraction unit 171 also supplies the information other than the RPS, PPS, etc. contained in the SPS to the decoding unit 113 as necessary.

(Explanation of the processing of the decoding device)

The reception processing performed by the decoding apparatus 170 illustrated in FIG. 27 is the same as the reception processing illustrated in FIG. 17 except for the RPS setting processing, so that only the RPS setting processing will be described below.

FIG. 28 is a flowchart illustrating in detail the RPS setting process performed by the decoding apparatus 170 illustrated in FIG. 27 .

In step S191 illustrated in Fig. 28, the extraction unit 171 obtains num_short_term_ref_pic_sets contained in the SPS (Fig. 21). In step S192, the extraction unit 171 obtains the disable_rps_prediction_flag included in the SPS. In step S193, the extraction unit 171 determines whether the obtained disable_rps_prediction_flag is "1".

When, in step S193, it is determined that the disable_rps_prediction_flag is not "1", in step S194, the extraction unit 171 obtains the unified_rps_prediction_control_present_flag included in the SPS. In step S195, the extraction unit 171 determines whether or not the obtained unified_rps_prediction_control_present_flag is "1".

When it is determined in step S195 that unified_rps_prediction_control_present_flag is "1", the extraction unit 171 obtains unified_delta_idx_minus1 included in the SPS, and then the process proceeds to step S197.

On the other hand, when it is determined in step S195 that unified_delta_idx_minus1 is not "1", the process proceeds to step S197. In addition, when it is determined in step S193 that disable_rps_prediction_flag is 1, the process proceeds to step S197.

In step S197, the extraction unit 171 sets the index i of the RPS corresponding to the generated RPS information to "0". In step S198, the extraction unit 171 determines whether the disable_rps_prediction_flag obtained in step S192 is "1".

When it is determined in step S198 that disable_rps_prediction_flag is "1", in step S199, the extraction unit 171 sets the inter_ref_pic_set_prediction_flag contained in the RPS information of the RPS of index i to "0", and the process proceeds to step S201.

On the other hand, when it is determined in step S198 that disable_rps_prediction_flag is not "1", in step S200, the extraction unit 171 obtains the inter_ref_pic_set_prediction_flag included in the RPS of the index i included in the SPS. Subsequently, the extraction unit 171 sets the obtained inter_ref_pic_set_prediction_flag as the inter_ref_pic_set_prediction_flag included in the RPS information of the RPS of index i, and the process proceeds to step S201.

In step S201, the extraction unit 171 determines whether or not inter_ref_pic_set_prediction_flag is "1". When it is determined in step S201 that inter_ref_pic_set_prediction_flag is "1", in step S202, the extraction unit 171 determines whether the unified_rps_prediction_control_present_flag obtained in step S194 is "1".

When it is determined in step S202 that unified_rps_prediction_control_present_flag is "1", the process proceeds to step S203. In step S203, the extraction unit 171 sets unified_delta_idx_minus1 obtained in step S196 as unified_delta_idx_minus1 included in the RPS information of the RPS of index i, and the process proceeds to step S206.

On the other hand, when it is determined in step S202 that unified_rps_prediction_control_present_flag is not "1", in step S204, the extraction unit 171 obtains delta_idx_minus1 included in the RPS of index i included in the SPS. Subsequently, the extraction unit 171 sets the obtained delta_idx_minus1 as delta_idx_minus1 included in the RPS information of the RPS of index i, and the process proceeds to step S206.

On the other hand, when it is determined in step S201 that inter_ref_pic_set_prediction_flag is not "1", in step S205, the extraction unit 171 obtains the reference picture designation information included in the RPS of the index i included in the SPS. Subsequently, the extraction unit 171 sets the obtained reference image designation information to the reference image designation information contained in the RPS information of the RPS of index i, and the process proceeds to step S206.

The processing of steps S206 to S208 is similar to the processing of steps S128 to S130 illustrated in FIG. 18 , so that the description thereof is omitted.

As above, the decoding apparatus 170 receives the disable_rps_prediction_flag, and generates the reference picture specifying information of the current decoded picture according to the disable_rps_prediction_flag. As a result, in the case where the disable_rps_prediction_flag is "1", the decoding apparatus 170 can decode the encoded stream in which the information amount of the RPS is reduced by the amount corresponding to inter_ref_pic_set_prediction_flag.

In addition, the decoding apparatus 170 receives delta_idx_minus1 common to all pictures within the GOP as unified_delta_idx_minus1, and generates reference picture specifying information of the current decoded picture according to unified_delta_idx_minus1. As a result, the decoding apparatus 170 can decode the encoded stream in which delta_idx_minus1 is set in units of GOPs.

<Third Embodiment>

(Example of the structure of the encoding apparatus according to the third embodiment)

FIG. 29 is a block diagram illustrating a structural example of an encoding apparatus to which the present technology according to the third embodiment is applied.

Here, the same reference numerals are assigned to each of the structures illustrated in FIG. 29 that are the same as those illustrated in FIG. 3 , and repeated explanations thereof are omitted.

The structure of the encoding device 190 illustrated in FIG. 29 differs from the structure of the encoding device 10 illustrated in FIG. 3 in that the setting unit 191 is set instead of the setting unit 12 . The encoding device 190 is obtained by combining the encoding device 10 illustrated in FIG. 3 and the encoding device 150 illustrated in FIG. 20 .

More specifically, the setting unit 191 of the encoding device 190 sets an RPS including a PRS that does not include inter_ref_pic_set_prediction_flag, but includes reference picture designation information, and, if necessary, an RPS that includes inter_ref_pic_set_prediction_flag, delta_idx_minus1, reference picture designation information, and the like. In addition, the setting unit 191 assigns an index to each RPS. Here, "0" is assigned as an index of an RPS that does not include inter_ref_pic_set_prediction_flag but includes reference picture designation information.

The setting unit 191 supplies the indexed RPS to the encoding unit 11 . In addition, the setting unit 191 sets the SPS including the RPS and disable_rps_prediction_flag and, if necessary, unified_rps_prediction_control_present_flag and unified_delta_idx_minus1. The setting unit 191 sets PPS and the like.

In addition, similarly to the setting unit 12 illustrated in FIG. 3 , the setting unit 191 generates an encoded stream based on the set SPS and PPS, and the encoded data supplied from the encoding unit 11 . Similar to the setting unit 12 , the setting unit 191 supplies the encoded stream to the transmission unit 13 .

(SPS syntax example)

FIG. 30 is a diagram illustrating an example of the syntax of the SPS set by the setting unit 191 illustrated in FIG. 29 .

The structure illustrated in FIG. 30 is the same as the structure illustrated in FIG. 21 , so that the description thereof is omitted.

(Syntax example of RPS)

FIG. 31 is a diagram illustrating an example of the syntax of the RPS.

Although not illustrated in the drawing, the description of the 11th and subsequent rows illustrated in FIG. 31 is the same as that of the fifth and subsequent rows illustrated in FIG. 1 .

As illustrated in the 2nd and 3rd rows illustrated in FIG. 31, when the index (idx) is "0", or when the disable_rps_prediction_flag is "1", in the RPS, the inter_ref_pic_set_prediction_flag is not included, but Contains reference picture specification information included when inter_ref_pic_set_prediction_flag is "0".

The description of the 4th row to the 10th row is the same as that of the 4th row to the 10th row illustrated in FIG. 22 , so that the description thereof is omitted.

(Explanation of the advantages of the present technology)

FIG. 32 is a diagram illustrating the information amount of the RPS set by the setting unit 191 illustrated in FIG. 29 .

In the example illustrated in FIG. 32 , the reference picture designation information of the 2nd picture and the 8th picture from the beginning within the GOP is the same as the reference picture designation information of the previous picture in coding order.

In this case, as illustrated in FIG. 32, the setting unit 191 sets "0" as disable_rps_prediction_flag, and sets "1" as unified_rps_prediction_control_present_flag. In addition, the setting unit 191 sets "0" as unified_delta_idx_minus1.

Further, for example, the setting unit 191 sets the reference image designation information of the first picture of the GOP to the RPS whose index is "0". In addition, the setting unit 191 sets "1" as inter_ref_pic_set_prediction_flag to the RPS whose index is "1". Therefore, the index of the RPS of the first picture of the GOP is set to "0", and the index of the RPS of the second picture and the eighth picture is set to "1".

As above, the setting unit 191 does not set the inter_ref_pic_set_prediction_flag to the RPS whose index is "0" serving as the RPS of the first picture. Thus, compared to the conventional case illustrated in FIG. 8 , the information amount of the RPS can be reduced by the amount corresponding to the inter_ref_pic_set_prediction_flag of the 1st picture.

In addition, the setting unit 191 sets delta_idx_minus1 common to all pictures in the GOP to unified_delta_idx_minus1. Therefore, delta_idx_minus1 can be set in units of GOPs.

Although not shown in the figure, the setting unit 191 sets "0", which is inter_ref_pic_set_prediction_flag common to all pictures in the GOP, to disable_rps_prediction_flag. Therefore, when the disable_rps_prediction_flag is "1", the information amount of the RPS can also be reduced by the amount corresponding to the inter_ref_pic_set_prediction_flag of pictures other than the 1st picture, compared to the normal case.

(Explanation of the processing of the encoding device)

The generation processing performed by the encoding device 190 illustrated in FIG. 29 is the same as the generation processing illustrated in FIG. 10 except for the RPS setting processing, so that only the RPS setting processing will be described below.

33 is a flowchart illustrating the RPS setting process performed by the setting unit 191 of the encoding device 190.

The processing of steps S221 to S226 illustrated in FIG. 33 is similar to the processing of steps S161 to S166 illustrated in FIG. 26 , so that the description thereof is omitted.

In step S227, the setting unit 191 determines whether the disable_rps_prediction_flag is "1", or whether the index i is "0". When it is determined in step S227 that the disable_rps_prediction_flag is "1" or the index i is "0", the process proceeds to step S228. On the other hand, when it is determined in step S227 that the disable_rps_prediction_flag is not "1" and the index i is not "0", the process proceeds to step S229.

The processing of steps S228-S235 is similar to the processing of steps S168-S175 illustrated in FIG. 26, so that the description thereof is omitted.

(Configuration example of decoding apparatus according to third embodiment)

FIG. 34 is a block diagram illustrating a structural example of a decoding apparatus that decodes an encoded stream transmitted from the encoding apparatus 190 illustrated in FIG. 29 to which the present technology according to the third embodiment is applied.

Here, the same reference numerals are assigned to each of the structures illustrated in FIG. 34 that are the same as those illustrated in FIG. 15 , and repeated explanations thereof are omitted.

The structure of the decoding apparatus 210 illustrated in FIG. 34 differs from the structure of the decoding apparatus 110 illustrated in FIG. 15 in that the extraction unit 211 is provided instead of the extraction unit 112 . The decoding device 210 sets the RPS information of each RPS according to the SPS illustrated in FIG. 30 including the RPS illustrated in FIG. 31 .

More specifically, similar to the extraction unit 112 illustrated in FIG. 15 , the extraction unit 211 of the decoding device 210 extracts SPS, PPS, encoded data, and the like from the encoded stream supplied from the reception unit 111 . Similar to the extraction unit 112 , the extraction unit 211 supplies the encoded data to the decoding unit 113 . In addition, the extraction unit 211 obtains the RPS information of each RPS from the SPS illustrated in FIG. 30 including the RPS illustrated in FIG. 31 , and supplies the obtained RPS information to the decoding unit 113. Further, similarly to the extraction unit 112, the extraction unit 211 also supplies the information other than the RPS, the PPS, etc. contained in the SPS to the decoding unit 113 as necessary.

(Explanation of the processing of the decoding device)

The reception processing by the decoding device 210 illustrated in FIG. 34 is the same as the reception processing illustrated in FIG. 17 except for the RPS setting processing, so that only the RPS setting processing will be described below.

FIG. 35 is a flowchart illustrating in detail the RPS setting process performed by the decoding apparatus 210 illustrated in FIG. 34 .

The processing of steps S251 to S257 illustrated in FIG. 35 is similar to the processing of steps S191 to S197 illustrated in FIG. 28 , so that the description thereof is omitted.

In step S258, the extraction unit 211 determines whether the disable_rps_prediction_flag obtained in step S252 is "1", or whether the index i is "0".

When it is determined in step S258 that the disable_rps_prediction_flag is "1", or the index i is "0", the process proceeds to step S259. On the other hand, when it is determined in step S258 that disable_rps_prediction_flag is not "1" and the index i is not "0", the process proceeds to step S260.

The processing of steps S259-S268 is similar to the processing of steps S199-S208 illustrated in FIG. 28, and thus the description thereof is omitted.

<Fourth Embodiment>

(Configuration example of encoding apparatus according to fourth embodiment)

FIG. 36 is a block diagram illustrating a structural example of an encoding apparatus to which the present technology according to the fourth embodiment is applied.

Here, the same reference numerals are assigned to each of the structures illustrated in FIG. 36 that are the same as those illustrated in FIG. 3 , and repeated explanations thereof are omitted.

The structure of the encoding device 230 illustrated in FIG. 36 differs from the structure of the encoding device 10 illustrated in FIG. 3 in that the encoding unit 231 is set instead of the encoding unit 11, and the setting unit 232 is set instead of the set Fixed unit 12. Depending on the type of slice within the screen, the encoding device 230 does not set information related to the reference image that is unnecessary for the type of slice.

More specifically, an image constituted in units of frames is input to the encoding unit 231 of the encoding device 230 as an input signal. The encoding section 231 encodes the input signal according to the HEVC method by referring to the RPS, PPS, etc. supplied from the setting section 232 . At this time, when necessary, the encoding unit 231 performs weighted prediction (weighted prediction) on the reference image in the inter prediction.

Here, the weighted prediction is a process of generating a predicted image by weighting a reference image. More specifically, for example, in the case where the decoded images of the two frames Y ₁ and Y ₀ preceding the current coded frame X in the coding order are used as reference images, in the weighted prediction, using the following formula (3), the frame is obtained: Predicted image X' of X.

X'=w ₀ ×Y ₀ +w ₀ ×Y ₁ +d...(3)

Here, in Equation (3), w ₀ and w ₁ are weighting coefficients, and d is an offset value. These weighting coefficients and offset values are transmitted as contained in the encoded stream.

By performing the weighted prediction, even if a change in luminance occurs between the reference image and the currently encoded image due to fade-in, fade-out, cross-fading, etc., the difference between the predicted image and the currently encoded image can be reduced. As a result, encoding efficiency can be improved.

Conversely, without weighted prediction, the change in luminance that occurs between the reference image and the currently encoded image directly becomes the difference between the predicted image and the currently encoded image due to fade-in, fade-out, cross-fading, etc., Thereby reducing the coding efficiency.

The encoding unit 231 supplies the encoded data obtained by the encoding process to the setting unit 232 .

Similar to the setting unit 12 illustrated in FIG. 3 , the setting unit 232 sets an RPS that does not include inter_ref_pic_set_prediction_flag, but includes reference picture specification information, and an RPS that includes inter_ref_pic_set_prediction_flag and reference picture specification information or delta_idx_minus1. Similar to the setting unit 12, the setting unit 232 assigns an index to each RPS.

The setting unit 232 sets SPS, PPS, and the like including RPS. The setting unit 232 supplies the indexed RPS and the PPS to the encoding unit 231 . The setting unit 232 generates an encoded stream based on the set SPS and PPS and the encoded data supplied from the encoding unit 231 . The setting unit 232 supplies the encoded stream to the transmission unit 13 .

(Configuration example of encoding device)

FIG. 37 is a block diagram illustrating a structural example of the encoding unit 231 illustrated in FIG. 36 .

Here, the same reference numerals are assigned to each of the structures illustrated in FIG. 37 that are the same as those illustrated in FIG. 4 , and repeated explanations thereof are omitted.

The structure of the encoding unit 231 illustrated in FIG. 37 differs from the structure of the encoding unit 11 illustrated in FIG. 4 in that the motion prediction/compensation unit 251 is provided instead of the motion prediction/compensation unit 47, lossless encoding is provided unit 252 instead of lossless encoding unit 36.

According to the PPS supplied from the setting unit 232 illustrated in FIG. 36 , the motion prediction/compensation unit 251 performs motion prediction/compensation processing using weighted prediction of all inter prediction modes as candidates. More specifically, the motion prediction/compensation unit 251 detects motion vectors of all inter prediction modes as candidates from the image supplied from the screen rearrangement buffer 32 and the reference image read from the frame memory 44 through the switch 45 . Subsequently, the motion prediction/compensation unit 251 performs compensation processing on the reference image based on the detected motion vector.

Subsequently, the motion prediction/compensation unit 251 calculates weighting information composed of the weighting coefficient and the offset value in the weighted prediction. The motion prediction/compensation unit 251 functions as a generation unit, and performs weighted prediction on the reference image after compensation processing based on the calculated weighting information, thereby generating a predicted image.

At this time, similarly to the motion prediction/compensation unit 47 illustrated in FIG. 4 , the motion prediction/compensation unit 251 calculates all inter predictions as candidates from the images supplied from the screen rearrangement buffer 32 , and the predicted images The cost function value of the mode. Then, similarly to the motion prediction/compensation unit 47, the motion prediction/compensation unit 251 determines the inter prediction mode whose cost function value is the smallest as the optimum inter prediction mode.

Then, similarly to the motion prediction/compensation unit 47 , the motion prediction/compensation unit 251 supplies the cost function value of the optimum inter prediction and the corresponding predicted image to the predicted image selection unit 48 . In addition, when the motion prediction/compensation unit 251 is notified from the predicted image selection unit 48 of the selection of the predicted image generated in accordance with the optimal inter prediction mode, the motion prediction/compensation unit 251 converts the inter prediction mode information to the corresponding motion The vector, weighting information, etc., are output to the lossless encoding unit 252 . In addition, the motion prediction/compensation unit 251 outputs the reference image designation information to the reference image setting unit 49 .

The lossless encoding unit 252 generates a slice type indicating the type of the slice of the currently encoded image based on the PPS supplied from the setting unit 232 illustrated in Fig. 36 . In addition, similar to the lossless encoding unit 36 illustrated in FIG. 4 , the lossless encoding unit 252 obtains intra prediction mode information from the intra prediction unit 46. Further, the lossless encoding unit 252 obtains inter prediction mode information, motion vector, weighting information, and the like from the motion prediction/compensation unit 251 . In addition, similarly to the lossless encoding unit 36, the lossless encoding unit 252 obtains the index of the RPS or the RPS or the like from the reference image setting unit 49, and obtains the quantization parameter from the rate control unit 50.

In addition, similarly to the lossless encoding unit 36, the lossless encoding unit 252 obtains the storage flag, index or offset amount and kind information from the adaptive offset filter 42 as offset filter information, and obtains from the adaptive loop filter 43 Get filter coefficients.

Similar to the lossless encoding unit 36 , the lossless encoding unit 252 performs lossless encoding of the quantized coefficients supplied from the quantization unit 35 . In addition, the lossless encoding unit 252 performs lossless encoding of quantization parameters, offset filter information and filter coefficients such as slice type, intra prediction mode information or inter prediction mode information, motion vector, weight information, and RPS index or RPS, as coded information.

The lossless encoding unit 252 adds the encoding information encoded in the lossless manner as a slice header to the coefficients encoded in the lossless manner, thereby generating encoded data. The lossless encoding unit 252 supplies the encoded data to the accumulation buffer 37 so as to be held therein.

(Syntax example of PPS)

38 and 39 are diagrams illustrating an example of the syntax of the PPS set by the setting unit 232 illustrated in FIG. 36 . 40 and 41 are diagrams illustrating an example of the syntax of PPS in the conventional HEVC method.

As illustrated in row 6 in FIG. 38 , in the PPS set by the setting unit 232, a unified flag unified_slice_type_flag indicating whether or not the kinds of all slices within the corresponding picture are the same is included. In addition, as illustrated in the seventh and eighth lines, when the unified flag is "1", the PPS includes an I flag (all_intra_slice_flag) indicating whether the types of all slices in the corresponding picture are I slices or not. ).

In addition, as illustrated in the ninth and tenth lines, when the I flag is not "1", in other words, when a P slice or a B slice is included in the screen, in the PPS, the Indicates whether the B-not-present flag no_b_slice_flag of the B slice exists in the corresponding picture.

As shown in the 11th and 12th lines, when the I flag is not "1", the PPS includes the RPSL0 number num_ref_idx_l0_default_active_minus1, and the RPSL0 number num_ref_idx_l0_default_active_minus1 is used for the display time as information related to the reference picture. The maximum number of RPS in the forward prediction (L0 prediction) of the reference picture earlier than the display time of the corresponding picture.

As illustrated in lines 13 and 14, in the case where the B-not-present flag is "0", in other words, in the case where the B slice is included in the picture, in the PPS, the RPSL1 number ( num_ref_idx_l1_default_active_minus1), as information related to a reference picture, the number of RPSL1 (num_ref_idx_l1_default_active_minus1) is the maximum number of RPS in backward prediction (L1 prediction) using a reference picture whose display time is later than that of the corresponding picture.

As illustrated in the 25th and 26th lines, when the I flag is not "1", in the PPS, the P prediction flag weighted_pred_flag is included as information related to the reference picture, and the P prediction flag weighted_pred_flag indicates that the P slice, whether to perform weighted prediction. In addition, when the B-not-present flag is not "1", the PPS includes the B prediction flag weighted_bipred_flag as information related to the reference picture, and the B prediction flag weighted_bipred_flag indicates whether weighted prediction is performed for the B slice.

As above, in the PPS illustrated in FIGS. 38 and 39 , in the case where the corresponding picture is composed of only I slices, the number of RPSL0, the number of RPSL1, the P prediction flag, and the B prediction flag are not set. In addition, in the case where the corresponding picture includes slices other than the I slice, the RPSL1 number and the B prediction flag are not set. Therefore, compared with the case where the number of RPSL0, the number of RPSL1, the P prediction flag, and the B prediction flag are set for all pictures regardless of the type of slice in the picture, encoding efficiency can be improved.

In addition, in the decoding device, in the case where the picture consists of only I slices, the RPSL0 number and the RPSL1 number are recognized as "0", and in the case where the picture includes slices other than the I slice, the RPSL1 number is recognized as "0".

In contrast, in the PPS of the conventional HEVC mode illustrated in FIGS. 40 and 41, as illustrated in the 6th, 7th, 17th, and 18th rows of FIG. 40, regardless of the number of slices within the picture Type, the number of RPSL0, the number of RPSL1, the P prediction flag and the B prediction flag are set.

In addition, when the picture is composed of only B slices, the P prediction flag may not be set.

(Example of the syntax of the slice header)

42-44 are diagrams illustrating examples of syntax of the slice header added by the lossless encoding unit 252 illustrated in FIG. 37 . In addition, FIGS. 45-47 are diagrams illustrating an example of the syntax of the slice header in the conventional HEVC method.

As illustrated in the second row of FIG. 42 , in the slice header added by the lossless encoding unit 252, a first flag first_slice_in_pic_flag indicating whether the corresponding slice is the first slice in the picture is included. In addition, as illustrated in lines 11 and 12, when the uniform flag is "0", or when the uniform flag is "1" and the first flag is "0", in the slice header, contains The slice type slice_type of the corresponding slice.

In other words, in the slice headers illustrated in FIGS. 42-44 , when the kinds of slices within the picture are not the same, or when the kinds of slices within the picture are the same and the corresponding slice is the first slice within the picture , set the slice type.

However, in the slice headers illustrated in FIGS. 42-44 , when the types of slices within the screen are the same, but the corresponding slice is a slice other than the first slice within the screen, the slice type is not set. In this case, the slice types included in the slice header are regarded as slice types of slices other than the first slice.

Therefore, compared to the case where the slice types of all slices are set regardless of whether or not the slice types of all the slices in the screen are the same, encoding efficiency can be improved.

On the contrary, in the slice header of the conventional HEVC method illustrated in FIGS. 45-47 , as illustrated in the 11th row in FIG. 45 , regardless of whether the slice types of all slices within the screen are the same or not, it is set The slice type for all slices.

(Explanation of the processing of the encoding device)

FIG. 48 is a flowchart illustrating generation processing by the encoding apparatus 230 illustrated in FIG. 36 .

In step S281 illustrated in FIG. 48 , the setting unit 232 of the encoding device 230 performs the RPS setting process illustrated in FIG. 11 . In step S282, the encoding section 231 performs an encoding process of encoding an image in frame units input as an input signal from the outside in accordance with the HEVC method. This encoding process will be described in detail later with reference to FIGS. 49 and 50 described later.

In step S283, the setting unit 232 sets the SPS including the RPS to be given an index. In step S284, the setting unit 232 performs a PPS setting process of setting the PPS. This PPS setting process will be described in detail later with reference to FIG. 51 described later.

The processing of steps S285 and S286 is similar to the processing of steps S15 and S16 illustrated in FIG. 10 , so that the description thereof is omitted.

49 and 50 show flowcharts illustrating the encoding process of step S282 illustrated in FIG. 48 in detail.

The processing of steps S301 and S302 illustrated in FIG. 49 is similar to the processing of steps S31 and S32 illustrated in FIG. 12 , so that the description thereof is omitted.

In step S303, the motion prediction/compensation unit 251 determines whether to perform weighted prediction based on the P prediction flag or the B prediction flag included in the PPS supplied from the setting unit 232 illustrated in Fig. 36 .

More specifically, in the case where the currently encoded image is a P slice, when the P prediction flag is "1", the motion prediction/compensation unit 251 determines to perform weighted prediction. In addition, in the case where the currently encoded image is a B slice, when the B prediction flag is "1", the motion prediction/compensation unit 251 determines to perform weighted prediction. Also, in the case where the currently encoded image is an I slice, the process of step S303 is skipped, and the process then proceeds to step S304.

When it is determined in step S303 that weighted prediction is to be performed, in step S304 the intra prediction unit 46 performs intra prediction processing of all intra prediction modes that are candidates. In addition, the intra prediction unit 46 calculates cost function values of all intra prediction modes that are candidates based on the image read from the screen rearrangement buffer 32 and the predicted image generated by the intra prediction process. Subsequently, the intra prediction unit 46 determines the intra prediction mode whose cost function value is the smallest as the optimal intra prediction mode. The intra prediction unit 46 supplies the predicted image generated in accordance with the optimal intra prediction mode, and the corresponding cost function value to the predicted image selection unit 48 .

In addition, the motion prediction/compensation unit 251 performs motion prediction/compensation processing using weighted prediction of all inter prediction modes as candidates. In addition, the motion prediction/compensation unit 251 calculates the cost function values of all the inter prediction modes as candidates based on the images supplied from the screen rearrangement buffer 32 and the predicted images, and predicts the inter prediction whose cost function value is the smallest mode is determined as the best inter prediction mode. Subsequently, the motion prediction/compensation unit 251 supplies the cost function value of the optimum inter prediction mode, and the corresponding predicted image, to the predicted image selection unit 48 .

However, in the case where the currently encoded image is an I slice, motion prediction/compensation processing is not performed. After the process of step S304, the process proceeds to step S306.

On the other hand, when it is determined in step S303 that weighted prediction is not to be performed, in step S305, the intra prediction unit 46 performs the same processing as that of step S304.

In addition, the motion prediction/compensation unit 251 performs motion prediction/compensation processing of all inter prediction modes that are candidates. Further, the motion prediction/compensation unit 251 calculates the cost function values of all the inter prediction modes as candidates based on the images supplied from the screen rearrangement buffer 32, and the predicted images, and predicts the inter prediction whose cost function value is the smallest mode is determined as the best inter prediction mode. Subsequently, the motion prediction/compensation unit 251 supplies the cost function value of the optimal inter prediction mode and the corresponding predicted image to the predicted image selection unit 48 . Subsequently, the process proceeds to step S306.

The processing of steps S306 to S308 is similar to the processing of steps S34 to S36 illustrated in FIG. 12 , so that the description thereof is omitted.

After the process of step S308, in step S309, the motion prediction/compensation unit 251 determines whether or not weighted prediction is performed in the motion prediction/compensation process. When it is determined in step S309 that weighted prediction is performed in the motion prediction/compensation process, the motion prediction/compensation unit 251 supplies the weighting information of the weighted prediction to the lossless encoding unit 252 in step S310. Subsequently, the process proceeds to step S311.

The processing of steps S311-S322 is similar to the processing of steps S37-S48 illustrated in Figs. 12 and 13, so that the description thereof is omitted.

In step S323 illustrated in Fig. 50 , the lossless encoding unit 252 determines whether the unified flag contained in the PPS supplied from the setting unit 232 illustrated in Fig. 36 is "0", or the unified flag and Whether the first flag is "1".

When it is determined in step S323 that the unified flag is "0", or the unified flag and the first flag are "1", in step S324, the lossless encoding unit 252 generates the slice type of the currently encoded image. Subsequently, the process proceeds to step S325.

On the other hand, when it is determined in step S323 that the unified flag is not "0" and the unified flag and the first flag are not "1", the process proceeds to step S325.

In step S325, the lossless encoding unit 252 performs the quantization parameters, offset filter information, and filter coefficients supplied from the rate control unit 50, such as slice types, intra prediction mode information or inter prediction mode information, motion vectors, weighting information, and The index of the RPS or the lossless encoding of the RPS is used as the encoding information.

The processing of steps S326 to S329 is similar to the processing of steps S50 to S53 illustrated in FIG. 13 , so that the description thereof is omitted.

FIG. 51 is a flowchart illustrating in detail the PPS setting process of step S284 illustrated in FIG. 48 . This PPS setting process is performed on a screen-by-screen basis.

In step S331 illustrated in FIG. 51 , the setting unit 232 determines whether or not the types of all slices in the screen are the same. When it is determined in step S331 that all slices in the screen are of the same type, in step S332, the setting unit 232 sets the unified flag to "1", and includes the set unified flag in the PPS.

In step S333, the setting section 232 determines whether or not the types of all slices in the screen are I slices. When it is determined in step S333 that the types of all slices in the screen are I slices, in step S334, the setting unit 232 sets the I flag to "1", and then includes the set I flag in the PPS, and the processing Proceed to step S337.

On the other hand, when it is determined in step S333 that the types of all slices in the screen are not I slices, in step S335, the setting unit 232 sets the I flag to "0", and then includes the set I flag in the PPS , the process proceeds to step S337.

On the other hand, when it is determined in step S331 that the types of all the slices in the screen are not the same, in step S336 the setting unit 232 sets the I flag to "0", and then includes the set I flag in the PPS , the process proceeds to step S337.

In step S337, the setting unit 232 determines whether the I flag is "1". When it is determined in step S337 that the I flag is not "1", in step S338, the setting unit 232 sets the RPSL0 number and the P prediction flag contained in the PPS, and includes the set RPSL0 number and P prediction flag in the in PPS.

In step S339, setting section 232 determines whether or not B slices are included in the screen. When it is determined in step S339 that the B slice is included in the picture, in step S340 the setting unit 232 sets the B-not-present flag included in the PPS to "0", and includes the set flag in the PPS middle. In step S341, the setting unit 232 sets the RPSL1 number and the B prediction flag contained in the PPS, and includes the set RPSL1 number and the B prediction flag in the PPS. Subsequently, the process returns to step S284 illustrated in FIG. 48 and then proceeds to step S285.

On the other hand, when it is determined in step S339 that the B slice is not included in the screen, in step S342, the setting unit 232 sets the B-not-present flag to "1" and includes the set flag in the PPS middle. Subsequently, the process returns to step S284 illustrated in FIG. 48 and then proceeds to step S285.

In addition, when it is determined in step S337 that the I flag is "1", the process returns to step S284 illustrated in FIG. 48 and then proceeds to step S285.

As described above, since the encoding device 230 sets the information related to the reference image according to the type of slice in the screen, the amount of information related to the reference image is reduced, so that encoding efficiency can be improved. In addition, since the encoding device 230 sets the slice type depending on whether the types of all slices within the picture are the same, the amount of information of the slice type is reduced, so that encoding efficiency can be improved.

(Configuration example of decoding apparatus according to fourth embodiment)

FIG. 52 is a block diagram illustrating a structural example of a decoding apparatus that decodes an encoded stream transmitted from the encoding apparatus 230 illustrated in FIG. 36 to which the present technology according to the fourth embodiment is applied.

Here, the same reference numerals are assigned to each of the structures illustrated in FIG. 52 that are the same as those illustrated in FIG. 15 , and repeated explanations thereof are omitted.

The structure of the decoding apparatus 270 illustrated in FIG. 52 differs from the structure illustrated in FIG. 15 in that the decoding unit 271 is provided instead of the decoding unit 113 . When necessary, the decoding apparatus 270 performs weighted prediction when performing motion compensation processing.

More specifically, the decoding unit 271 of the decoding device 270 decodes the encoded data supplied from the extraction unit 112 according to the HEVC method based on the inter_ref_pic_set_prediction_flag of each RPS supplied from the extraction unit 112, and delta_idx_minus1 or reference picture designation information. At this time, the decoding unit 271 refers to information other than the RPS, the PPS, and the like contained in the SPS, as necessary. In addition, when necessary, the decoding unit 271 performs weighted prediction when performing motion compensation processing. The decoding unit 271 outputs the image obtained by decoding as an output signal.

(Example of structure of decoding unit)

FIG. 53 is a block diagram illustrating a structural example of the decoding unit 271 illustrated in FIG. 52 .

Here, the same reference numerals are assigned to each of the structures illustrated in FIG. 53 that are the same as those illustrated in FIG. 16 , and repeated explanations thereof are omitted.

The structure of the decoding unit 271 illustrated in FIG. 53 differs from the structure illustrated in FIG. 16 in that the lossless decoding unit 291 is provided instead of the lossless decoding unit 132, and the motion compensation unit 292 is provided additionally instead of the motion compensation unit 145.

Similar to the lossless decoding unit 132 illustrated in FIG. 16 , the lossless decoding unit 291 of the decoding unit 271 performs lossless decoding on the encoded data supplied from the accumulation buffer 131, thereby obtaining quantization coefficients and encoding information. Similar to the lossless decoding unit 132, the lossless decoding unit 291 supplies the quantized coefficients to the inverse quantization unit 133. In addition, the lossless decoding unit 291 supplies intra prediction mode information and the like as encoding information to the intra prediction unit 143 , and supplies the motion vector, inter prediction mode information, weighting information, and the like to the motion compensation unit 292 . Similar to the lossless decoding unit 132 , the lossless decoding unit 291 supplies the reference image setting unit 144 with the RPS flag and index of the RPS, or RPS, which is encoded information.

In addition, similar to the lossless decoding unit 132 , the lossless decoding unit 291 supplies intra prediction mode information or inter prediction mode information as encoding information to the switch 146 . Similar to the lossless decoding unit 132 , the lossless decoding unit 291 supplies offset filter information as encoded information to the adaptive offset filter 137 and supplies filter coefficients to the adaptive loop filter 138 .

Similar to the motion compensation unit 145 illustrated in FIG. 16 , the motion compensation unit 292 reads from the frame memory 141 specified by the reference image specifying information through the switch 142 on the basis of the reference image specifying information supplied from the reference image setting unit 144 Reference image.

In addition, similar to the motion prediction/compensation unit 251 illustrated in FIG. 37 , the motion compensation unit 292 decides whether to perform weighted prediction according to the P prediction flag or the B prediction flag contained in the PPS supplied from the extraction unit 112 .

The motion compensation unit 292 functions as a generation unit that performs motion compensation processing of weighted prediction using the optimal inter prediction mode indicated by the inter prediction mode information by using a motion vector and a reference image when it is determined that weighted prediction is to be performed. At this time, when necessary, the motion compensation unit 292 refers to the RPSL0 number when the slice of the current coded image is a P slice, and refers to the RPSL0 number and RPSL1 number when the slice of the current coded image is a B slice.

On the other hand, in a case where it is determined that weighted prediction will not be performed, similarly to the motion compensation unit 145, the motion compensation unit 292 performs motion compensation processing of the optimal inter prediction mode. The motion compensation unit 292 supplies the predicted image generated as a result thereof to the switch 146 .

(Explanation of the processing of the decoding device)

FIG. 54 is a flowchart illustrating reception processing by the decoding apparatus 270 illustrated in FIG. 52 .

The processing of steps S351 to S353 illustrated in FIG. 54 is similar to the processing of steps S111 to S113 illustrated in FIG. 17 , so that the description thereof is omitted.

In step S354, the decoding unit 271 performs decoding processing based on the RPS information and the PPS of each RPS supplied from the extraction unit 112. This decoding process will be described in detail with reference to FIG. 55 described later. Then, the processing ends.

FIG. 55 is a flowchart illustrating the decoding process of step S354 illustrated in FIG. 54 in detail.

In step S361 illustrated in FIG. 55, the accumulation buffer 131 of the decoding unit 271 receives encoded data constituted in units of frames from the extraction unit 112 illustrated in FIG. 52, and holds the received encoded data. The accumulation buffer 131 supplies the stored encoded data to the lossless decoding unit 291 .

In step S362, the lossless decoding unit 291 performs lossless decoding of the encoded data supplied from the accumulation buffer 131, thereby obtaining quantization coefficients and encoding information. The lossless decoding unit 291 supplies the quantized coefficients to the inverse quantization unit 133 . In addition, the lossless decoding unit 291 supplies intra prediction mode information and the like as encoding information to the intra prediction unit 143, and supplies a motion vector, inter prediction mode information, weighting information, an RPS flag, an index of the RPS, or the RPS, etc. to the motion compensation unit 292.

In addition, the lossless decoding unit 291 supplies intra prediction mode information or inter prediction mode information as encoded information to the switch 146 . The lossless decoding unit 291 supplies the offset filter information as encoded information to the adaptive offset filter 137 and supplies the filter coefficients to the adaptive loop filter 138 .

The processing of steps S363-S365 is similar to the processing of steps S133-S135 illustrated in FIG. 19, so that the description thereof is omitted. In step S366, similar to the motion prediction/compensation unit 251 illustrated in Fig. 37 , the motion compensation unit 292 is based on the P prediction flag or the B prediction flag contained in the PPS supplied from the extraction unit 112 illustrated in Fig. 52 , to determine whether to perform weighted prediction.

When it is determined in step S366 that weighted prediction is to be performed, the motion compensation unit 292 reads the reference image based on the reference image designation information supplied from the reference image setting unit 144, and uses the motion vector and the reference image to utilize the frame by frame Motion compensation processing for weighted prediction of the optimal inter prediction mode indicated by the inter prediction mode information.

At this time, when necessary, the motion compensation unit 292 refers to the RPSL0 number when the slice of the current coded image is a P slice, and refers to the RPSL0 number and RPSL1 number when the slice of the current coded image is a B slice. The motion compensation unit 292 supplies the predicted image generated as a result thereof to the addition unit 135 through the switch 146, and the process proceeds to step S370.

On the other hand, when it is determined in step S366 that the weighted prediction will not be performed, in step S368, the motion compensation unit 292 reads the reference image based on the reference image designation information supplied from the reference image setting unit 144, and uses motion The vector and the reference image are subjected to motion compensation processing of the optimal inter prediction mode indicated by the inter prediction mode information. The motion compensation unit 292 supplies the predicted image generated as a result thereof to the addition unit 135 through the switch 146, and the process proceeds to step S370.

The processing of steps S369-S377 is similar to the processing of steps S137-S145 illustrated in FIG. 19, so that the description thereof is omitted.

As above, the decoding apparatus 270 can decode an encoded stream with improved encoding efficiency by setting the information related to the reference image according to the kind of slice within the picture.

In addition, in the fourth embodiment, although the information related to the reference picture is described as the RPSL0 number, the RPSL1 number, the P prediction flag, and the B prediction flag, the present technology is not limited to this.

<Application to multi-view image encoding/multi-view image decoding>

The above-described series of processing can be applied to multi-view image encoding and multi-view image decoding. FIG. 56 is a diagram illustrating an example of a multi-view image encoding system.

As illustrated in FIG. 56 , the multi-viewpoint image includes images of a plurality of viewpoints, and images of predetermined viewpoints among the plurality of viewpoints are designated as images of the base view. The images of the respective viewpoints other than the images of the base view are regarded as images of the non-base view.

In the case of performing multi-view image encoding as illustrated in FIG. 56 , for each view (the same view), the difference between the quantization parameters can be obtained.

(1) Basic view:

(1-1) dQP (base view)=Current_CU_QP (base view)-LCU_QP (base view)

(1-2) dQP (basic view) = Current_CU_QP (basic view) - Previsous_CU_QP (basic view)

(1-3) dQP (base view)=Current_CU_QP (base view)-Slice_QP (base view)

(2) Non-basic view:

(2-1) dQP (non-base view)=Current_CU_QP (non-base view)-LCU_QP (non-base view)

(2-2) dQP (non-basic view)=CurrentQP (non-basic view)-PrevisousQP (non-basic view)

(2-3) dQP (non-base view)=Current_CU_QP (non-base view)-Slice_QP (non-base view)

In the case of multi-view image encoding, the difference between the quantization parameters can be obtained for each view (different view).

(3) Basic view/non-basic view:

(3-1) dQP (inter-view) = Slice_QP (base view) - Slice_QP (non-base view)

(3-2) dQP (inter-view) = LCU_QP (base view) - LCU_QP (non-base view)

(4) Non-Basic View/Non-Basic View:

(4-1) dQP (inter-view)=Slice_QP (non-base view i)-Slice_QP (non-base view j)

(4-2) dQP (inter-view) = LCU_QP (non-base view i) - LCU_QP (non-base view j)

In this case, (1)-(4) described above can be used in combination. For example, in the non-base view, it is possible to consider the technique of obtaining the difference between the quantization parameters of the base view and the non-base view (combining 3-1 and 2-3) at the slice level, and at the LCU level, obtaining the base view and Technique for the difference between quantization parameters of non-base views (combination 3-2 and 2-1). In this way, by repeatedly applying the difference, it is possible to improve encoding efficiency even in the case of multi-view encoding.

Similar to the above-described technique, for each dQP explained above, a flag for identifying whether there is a dQP having a value other than "0" can be set.

<Multi-view image encoding device>

FIG. 57 is a diagram illustrating a multi-viewpoint image encoding apparatus that performs the above-described multi-viewpoint image encoding. As illustrated in FIG. 57 , the multi-view image encoding apparatus 600 includes an encoding unit 601 , an encoding unit 602 , and a multiplexer 603 .

The encoding unit 601 encodes a base view image, thereby generating a base view image encoded stream. In addition, the encoding unit 602 encodes the non-base view image, thereby generating a non-base view image encoded stream. The multiplexer 603 multiplexes the base view image encoded stream generated by the encoding unit 601 and the non-base view image encoded stream generated by the encoding unit 602, thereby generating a multi-viewpoint image encoded stream.

The encoding device 10 ( 150 and 190 ) is applicable to the encoding unit 601 and the encoding unit 602 of this multi-viewpoint image encoding device 600 . In this case, the multi-view image encoding apparatus 600 sets the difference between the quantization parameter set by the encoding unit 601 and the quantization parameter set by the encoding unit 602, and transmits the set difference.

<Multi-view image decoding device>

FIG. 58 is a diagram illustrating a multi-view image decoding apparatus that performs the above-described multi-view image decoding. As illustrated in FIG. 58 , the multi-viewpoint image decoding apparatus 610 includes a demultiplexer 611 , a decoding unit 612 , and a decoding unit 613 .

The demultiplexer 611 demultiplexes the multi-view image coded stream obtained by multiplexing the base view image coded stream and the non-base view image coded stream, thereby extracting the base view image coded stream and the non-base view image coded stream. The decoding unit 612 decodes the base-view image encoded stream extracted by the demultiplexer 611, thereby obtaining a base-view image. The decoding unit 613 decodes the non-base view image encoded stream extracted by the demultiplexer 611, thereby obtaining a non-base view image.

The decoding device 110 ( 170 and 210 ) is applicable to the decoding unit 612 and the decoding unit 613 of this multi-viewpoint image decoding device 610 . In this case, the multi-view image decoding device 610 sets the quantization parameter based on the difference between the quantization parameter set by the encoding unit 601 and the quantization parameter set by the encoding unit 602, and performs inverse quantization.

<Application to Layered Image Encoding/Layered Image Decoding>

The above-described series of processes are applicable to layered image encoding and layered image decoding. FIG. 59 is a diagram illustrating an example of a layered image coding method.

As illustrated in FIG. 59 , the layered image includes images of a plurality of layers (resolutions), and images of a predetermined layer among the plurality of resolutions are designated as images of the base layer. The images of the respective layers other than the images of the base layer are regarded as images of the non-base layer.

In the case of performing layered image coding (spatial scalability) as illustrated in FIG. 59 , in each layer (the same layer), the difference between the quantization parameters can be obtained.

(1) Basic layer:

(1-1) dQP (base layer)=Current_CU_QP (base layer)-LCU_QP (base layer)

(1-2) dQP (base layer)=Current_CU_QP (base layer)-Previsous_CU_QP (base layer)

(1-3) dQP (base layer)=Current_CU_QP (base layer)-Slice_QP (base layer)

(2) Non-basic layer

(2-1) dQP (non-base layer)=Current_CU_QP (non-base layer)-LCU_QP (non-base layer)

(2-2) dQP (non-base layer)=CurrentQP (non-base layer)-PrevisousQP (non-base layer)

(2-3) dQP (non-base layer)=Current_CU_QP (non-base layer)-Slice_QP (non-base layer)

In the case of performing layered encoding, in each layer (different layers), the difference between the quantization parameters can be obtained.

(3) Basic layer/non-basic layer:

(3-1) dQP (inter-layer)=Slice_QP (base layer)-Slice_QP (non-base layer)

(3-2) dQP (inter-layer) = LCU_QP (base layer) - LCU_QP (non-base layer)

(4) Non-Basic Layer/Non-Basic Layer

(4-1) dQP (inter-layer)=Slice_QP (non-base layer i)-Slice_QP (non-base layer j)

(4-2) dQP (inter-layer) = LCU_QP (non-base layer i) - LCU_QP (non-base layer j)

In this case, (1)-(4) described above can be used in combination. For example, in the non-base layer, a technique of obtaining the difference between the quantization parameters of the base layer and the non-base layer at the slice level (combining 3-1 and 2-3) can be considered, and at the LCU level, obtaining the base layer and Techniques for the difference between quantization parameters of non-base layers (combination 3-2 and 2-1). In this way, by repeatedly applying the difference, the coding efficiency can be improved even in the case of performing hierarchical coding.

Similar to the above-described technique, for each dQP explained above, a flag for identifying whether there is a dQP having a value other than "0" can be set.

<Layered Image Coding Device>

FIG. 60 is a diagram illustrating a layered image encoding apparatus that performs the layered image encoding explained above. As illustrated in FIG. 60 , the layered image encoding apparatus 620 includes an encoding unit 621 , an encoding unit 622 , and a multiplexer 623 .

The encoding unit 621 encodes the base layer image, thereby generating a base layer image encoded stream. In addition, the encoding unit 622 encodes a non-base layer image, thereby generating a non-base layer image encoded stream. The multiplexer 623 multiplexes the base layer image encoded stream generated by the encoding unit 621 and the non-base layer image encoded stream generated by the encoding unit 622, thereby generating a layered image encoded stream.

The encoding device 10 ( 150 and 190 ) is applicable to the encoding unit 621 and the encoding unit 622 of such a layered image encoding device 620 . In this case, the layered image encoding apparatus 620 sets the difference between the quantization parameter set by the encoding unit 621 and the quantization parameter set by the encoding unit 622, and transmits the set difference.

<Layered Image Decoding Device>

FIG. 61 is a diagram illustrating a layered image decoding apparatus that performs the above-explained layered image decoding. As illustrated in FIG. 61 , the layered image decoding apparatus 630 includes a demultiplexer 631 , a decoding unit 632 and a decoding unit 633 .

The demultiplexer 631 demultiplexes the layered image coded stream obtained by multiplexing the base layer image coded stream and the non-base layer image coded stream, thereby extracting the base layer image coded stream and the non-base layer image coded stream. The decoding unit 632 decodes the base layer image encoded stream extracted by the demultiplexer 631, thereby obtaining a base layer image. The decoding unit 633 decodes the non-base layer image encoded stream extracted by the demultiplexer 631, thereby obtaining a non-base layer image.

The decoding device 110 ( 170 and 210 ) is applicable to the decoding unit 632 and the decoding unit 633 of such a layered image decoding device 630 . In this case, the layered image decoding device 630 sets the quantization parameter based on the difference between the quantization parameter set by the encoding unit 621 and the quantization parameter set by the encoding unit 622, and performs inverse quantization.

<Description of the computer to which this technology is applied>

The above-described series of processing can be performed by hardware or software. In the case where the series of processing is executed with software, a program constituting the software is installed on a computer. Here, the computer includes a computer built into dedicated hardware, a computer capable of executing various functions by installing various programs, such as a general-purpose computer, and the like.

FIG. 62 is a block diagram illustrating an example of a hardware configuration of a computer that executes the above-described series of processes in accordance with a program.

In the computer, a CPU (Central Processing Unit) 801 , a ROM (Read Only Memory) 802 and a RAM (Random Access Memory) 803 are interconnected by a bus 804 .

In addition, an input/output interface 805 is connected to the bus 804 . An input unit 806 , an output unit 807 , a storage unit 808 , a communication unit 809 , and a drive 810 are connected to the input/output interface 805 .

The input unit 806 is constituted by a keyboard, a mouse, a microphone, and the like. The output unit 807 is constituted by a display, a speaker, and the like. The storage unit 808 is constituted by a hard disk, a nonvolatile memory, or the like. The communication unit 809 is constituted by a network interface and the like. The drive 810 drives a removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor disk.

In the computer constructed as described above, the CPU 801 performs a series of processing described above by loading the program stored in the storage unit 808 into the RAM 803 via the input/output interface 805 and the bus 804 and executing the program. .

By recording the program on the removable medium 811 as a package medium, the program executed by the computer (CPU 801) can be provided. In addition, the program may be provided through a wired or wireless transmission medium, such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed into the storage unit 808 through the input/output interface 805 by putting the removable medium 811 in the drive 810 . In addition, the program can be received by the communication unit 809 and installed in the storage unit 808 through a wired or wireless transmission medium. Also, the program may be pre-installed to the ROM 802 or the storage unit 808 .

In addition, the program executed by the computer may be a program that is processed sequentially in the order described herein, or may be a program that is processed in parallel, or when necessary, such as when the program is called.

<Structure example of TV set>

FIG. 63 illustrates a schematic structure of a television to which the present technology is applied. The television set 900 includes: an antenna 901; a tuner 902; a demultiplexer 903; a decoder 904; a video signal processing unit 905; a display unit 906; In addition, the television set 900 includes a control unit 910, a user interface unit 911, and the like.

The tuner 902 selects a desired channel from the broadcast wave signal received through the antenna 901, performs demodulation, and then outputs the obtained encoded bit stream to the demultiplexer 903.

The demultiplexer 903 extracts packets of video or audio of the program targeted for viewing from the encoded bit stream, and outputs the data of the extracted packets to the decoder 904 . In addition, the demultiplexer 903 supplies data packets such as EPG (Electronic Program Guide) to the control unit 910 . In addition, when scrambling is performed, the scrambling is canceled by a demultiplexer or the like.

The decoder 904 performs packet decoding processing, outputs the video data generated by the decoding processing to the video signal processing unit 905 , and outputs the audio data to the audio signal processing unit 907 .

The video signal processing unit 905 performs noise removal on the video data, video processing according to user settings, and the like. The video signal processing unit 905 generates video data of a program to be displayed on the display unit 906, image data corresponding to processing based on an application provided through a network, and the like. In addition, the video signal processing unit 905 generates video data for displaying a menu screen, such as an item selection screen, etc., and superimposes the generated video data on the video data of the program. The video signal processing unit 905 generates a driving signal according to the video data generated as above, and drives the display unit 906 .

The display unit 906 drives a display device (eg, a liquid crystal display device or the like) according to the drive signal supplied from the video signal processing unit 905, thereby displaying the video of the program and the like.

The audio signal processing unit 907 performs predetermined processing such as noise cancellation on the audio data, performs D/A conversion processing of the audio data after the processing, or amplification processing of the audio data, and supplies the resultant data to the speaker 908, thereby for audio output.

The external interface unit 909 is an interface for connecting to an external device or a network, and transmits/receives data such as video data or audio data.

The user interface unit 911 is connected to the control unit 910 . The user interface 911 is composed of an operation switch, a remote control signal receiving unit, and the like, and supplies an operation signal corresponding to a user operation to the control unit 910 .

The control unit 910 is constituted by a CPU (Central Processing Unit), a memory, and the like. The memory holds programs executed by the CPU, various data necessary for processing executed by the CPU, EPG data, data obtained through a network, and the like. The programs stored in the memory are read and executed by the CPU at predetermined timings, such as when the television 900 is activated. By executing the program, the CPU controls the respective units so that the television 900 operates according to user operations.

In addition, in the television set 900, in order to connect the tuner 902, the demultiplexer 903, the video signal processing unit 905, the audio signal processing unit 907, the external interface unit 909, etc. to the control unit 910, a bus 912 is arranged.

In the television set constructed in this way, the functions of the decoding device (decoding method) according to the present application are realized in the decoder 904 . Thus, an encoded stream in which the amount of information related to information specifying a reference image is reduced can be decoded.

<Configuration example of mobile phone>

FIG. 64 illustrates a schematic structure of a mobile phone to which the present technology is applied. The mobile phone 920 includes: a communication unit 922; an audio codec 923; a camera unit 926; an image processing unit 927; These units are interconnected by bus 933 .

In addition, the antenna 921 is connected to the communication unit 922 , and the speaker 924 and the microphone 925 are connected to the audio codec 923 . Furthermore, the operation unit 932 is connected to the control unit 931 .

The mobile phone 920 performs various operations such as transmission and reception of audio signals, transmission and reception of electronic mail and image data, image capture, and data recording in various modes such as a voice call mode and a data communication mode.

In the voice call mode, the audio signal generated by the microphone 925 is converted into audio data by the audio codec 923 or compressed, and the resulting signal is provided to the communication unit 922 . The communication unit 922 performs modulation processing, frequency conversion processing, etc. on the audio data, thereby generating a transmission signal. In addition, the communication unit 922 supplies the transmission signal to the antenna 921 to be transmitted to a base station not shown in the figure. Further, the communication unit 922 performs amplification processing, frequency conversion processing, demodulation processing, etc. on the reception signal received by the antenna 921 , and supplies the obtained audio data to the audio codec 923 . The audio codec 923 performs data decompression of the audio data, converts the audio data into an analog audio signal, and outputs the resultant signal to the speaker 924 .

In addition, in the data communication mode, when a mail is transmitted, the control unit 931 receives character data input by an operation on the operation unit 932 and displays the input characters on the display unit 930 . Further, the control unit 931 generates mail data according to a user instruction from the operation unit 932 , and supplies the generated mail data to the communication unit 922 . The communication unit 922 performs modulation processing, frequency conversion processing, etc. on the mail data, and then transmits the obtained transmission signal from the antenna 921 . In addition, the communication unit 922 performs amplification processing, frequency conversion processing, demodulation processing, etc. on the reception signal received through the antenna 921, thereby restoring mail data. The mail data is supplied to the display unit 930, thereby displaying the content of the mail.

In addition, the mobile phone 920 can use the recording/reproducing unit 929 to store the received mail data in a storage medium. The storage medium may be any rewritable storage medium. For example, the storage medium is a semiconductor memory such as RAM or built-in flash memory, a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, or a removable medium such as a USB memory or a memory card.

In the data communication mode, when transferring image data, the image data generated by the camera unit 926 is supplied to the image processing unit 927 . The image processing unit 927 performs encoding processing of image data, thereby generating encoded data.

The multiplexing/demultiplexing unit 928 multiplexes the encoded data generated by the image processing unit 927 and the audio data supplied from the audio codec 923 in a predetermined manner, and supplies the multiplexed data to the communication unit 922 . The communication unit 922 performs modulation processing of the multiplexed data, frequency conversion processing, and the like, and then transmits the obtained transmission signal from the antenna 921 . In addition, the communication unit 922 performs amplification processing, frequency conversion processing, demodulation processing, etc. on the reception signal received by the antenna 921, thereby restoring the multiplexed data. The multiplexed data is supplied to the multiplexing/demultiplexing unit 928 . The multiplexing/demultiplexing unit 928 separates the multiplexed data, supplies the encoded data to the image processor 927, and supplies the audio data to the audio codec 923. The image processor 927 performs decoding processing of the encoded data, thereby generating image data. The image data is supplied to the display unit 930 so that the received image is displayed. The audio codec 923 converts the audio data into an analog audio signal, and supplies the converted analog audio signal to the speaker 924, thereby outputting the received audio.

In the mobile phone device constructed in this way, the functions of the encoding device and the decoding device (encoding method and decoding method) according to the present application are realized in the image processing unit 927 . Thus, an encoded stream in which the amount of information related to information specifying a reference image is reduced can be decoded.

<Structure example of recording and reproducing apparatus>

FIG. 65 illustrates a schematic structure of a recording and reproducing apparatus to which the present technology is applied. For example, the recording and reproducing apparatus 940 records audio data and video data of the received broadcast program on a recording medium, and provides the recorded data to the user at timing corresponding to the user's instruction. In addition, for example, the recording and reproducing apparatus 940 may obtain audio data and video data from another apparatus and record the audio data and video data on a recording medium. Furthermore, the recording and reproducing device 940 decodes and outputs audio data and video data recorded on the recording medium, so that in a monitor or the like, display of images or output of audio can be performed.

The recording and reproducing apparatus 940 includes: a tuner 941; an external interface unit 942; an encoder 943; an HDD (Hard Disk Drive) unit 944; an optical disc drive 945; a selector 946; a decoder 947; unit 949; and user interface unit 950.

The tuner 941 selects a desired channel from a broadcast signal received through an antenna (not shown). The tuner 941 outputs the encoded bit stream obtained by demodulating the reception signal of the desired channel to the selector 946 .

The external interface unit 942 is constituted by at least one of an IEEE 1394 interface, a network interface unit, a USB interface, a flash memory interface, and the like. The external interface unit 942 is an interface connected to an external device, a network, a memory card, and the like, and performs data reception of video data, audio data, and the like to be recorded.

When the video data and audio data supplied from the external interface unit 942 are not encoded, the encoder 943 encodes the video data and audio data in a predetermined manner, and then outputs the encoded bit stream to the selector 946 .

The HDD unit 944 records content data such as video and audio, various programs, and other data on the built-in hard disk, and reads the recorded data from the hard disk upon reproduction or the like.

The optical disc drive 945 performs signal recording and signal reproduction on the loaded optical disc. The optical disc is, for example, a DVD disc (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+R, DVD+RW, etc.), Blu-ray (registered trademark) disc, or the like.

When recording video or audio, the selector 946 selects the encoded bit stream supplied from the tuner 941 or the encoder 943, and outputs the selected encoded bit stream to one of the HDD unit 944 and the optical disc drive 945. In addition, the selector 946 supplies the encoded bit stream output from the HDD unit 944 or the optical disc drive 945 to the decoder 947 when reproducing video or audio.

The decoder 947 performs decoding processing of the encoded bit stream. The decoder 947 supplies the video data generated by performing the decoding process to the OSD unit 948 . In addition, the decoder 947 outputs audio data generated by performing decoding processing.

The OSD unit 948 generates video data for displaying a menu screen, such as an item selection menu, etc., and outputs the generated video data so as to overlap with the video data output from the decoder 947 .

The user interface unit 950 is connected to the control unit 949 . The user interface unit 950 is composed of an operation switch, a remote control signal receiving unit, and the like, and supplies an operation signal corresponding to a user operation to the control unit 949 .

The control unit 949 is constituted by using a CPU, a memory, and the like. The memory stores programs executed by the CPU and various data necessary for the processing performed by the CPU. The program stored in the memory is read and executed by the CPU at predetermined timing, such as when the recording and reproducing apparatus 940 is activated. The CPU executes the program, thereby performing control of the respective units, so that the recording and reproducing apparatus 940 operates according to user operations.

In the recording and reproducing apparatus constructed in this way, the functions of the decoding apparatus (decoding method) according to the present application are realized in the decoder 947 . Thus, an encoded stream in which the amount of information related to information specifying a reference image is reduced can be decoded.

<Configuration example of imaging apparatus>

FIG. 66 is a diagram illustrating an example of a schematic structure of an imaging apparatus to which the present technology is applied. The imaging device 960 images the subject and displays the image of the subject on the display unit, or records the image of the subject as image data on a recording medium.

The imaging device 960 includes: an optical component 961; an imaging unit 962; a camera signal processing unit 963; an image data processing unit 964; a display unit 965; an external interface unit 966; a storage unit 967; . In addition, the user interface unit 971 is connected to the control unit 970 . Further, the image data processing unit 964 , the external interface unit 966 , the storage unit 967 , the media drive 968 , the OSD unit 969 , the control unit 970 and the like are interconnected through the bus 972 .

The optical member 961 is constituted by using a focus lens, a diaphragm mechanism, and the like. The optical member 961 forms an optical image of the subject on the image plane of the imaging unit 962 . The imaging unit 962 is constructed by using a CCD or a CMOS image sensor, generates electrical signals corresponding to the optical image through photoelectric conversion, and supplies the generated electrical signals to the signal processing unit 963 .

The camera signal processing unit 963 performs various camera signal processing, such as knee correction, gamma correction, and color correction, on the image signal supplied from the imaging unit 962 . The camera signal processing unit 963 supplies the image data after the camera signal processing to the image data processing unit 964.

The image data processing unit 964 performs encoding processing of the image data supplied from the camera signal processing unit 963 . The image data processing unit 964 supplies the encoded data generated by performing the encoding process to the external interface unit 966 or the media drive 968 . In addition, the image data processing unit 964 performs decoding processing of the encoded data supplied from the external interface unit 966 or the media drive 968 . The image data processing unit 964 supplies the image data generated by performing the decoding process to the display unit 965 . In addition, the image data processing unit 964 performs processing of supplying image data supplied from the camera signal processing unit 963 to the display unit 965, and supplies the display data obtained from the OSD unit 969 to the display unit 965 superimposed with the image data.

The OSD unit 969 generates display data such as a menu screen or icons composed of symbols, characters, or graphics, and outputs the generated display data to the image data processing unit 964 .

The external interface unit 966 is constituted by, for example, a USB input/output terminal or the like, and in the case of printing an image, is connected to a printer. In addition, when necessary, a drive is connected to the external interface unit 966, a removable medium such as a magnetic disk or an optical disk is appropriately placed, and when necessary, a computer program read from the removable medium is installed. Further, the external interface unit 966 includes a network interface connected to a predetermined network such as a LAN or the Internet. For example, in accordance with an instruction from the user interface unit 971, the control unit 970 may read encoded data from the media drive 968, and provide the read encoded data from the external interface unit 968 to another device connected through a network. In addition, the control unit 970 may obtain encoded data or image data supplied from another device through a network through the external interface unit 966, and supply the obtained data to the image data processing unit 964.

As the recording medium driven by the media drive 968, for example, any readable/writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory is used. In addition, the kind of the recording medium as the removable medium is arbitrary, and thus may be a tape device, a magnetic disk, or a memory card. In addition, a non-contact IC (Integrated Circuit) card or the like can be used as the recording medium.

In addition, by combining the media drive 968 and the recording medium, for example, the recording medium may be constituted by a non-portable recording medium such as a built-in hard disk drive or an SSD (Solid State Drive).

The control unit 970 is constituted by using a CPU. The storage unit 967 stores programs executed by the control unit 970 , various data necessary for processing by the control unit 970 , and the like. The programs stored in the storage unit 967 are read and executed by the control unit 970 at predetermined timings, such as when the image forming apparatus 960 is activated. The control unit 970 executes a program to perform control of the respective units so that the image forming apparatus 960 operates according to user operations.

In the imaging device constructed in this way, the functions of the encoding device and the decoding device (encoding method and decoding method) according to the present application are realized in the image data processing unit 964 . Thus, the amount of information related to the information specifying the reference image can be reduced. In addition, an encoded stream in which the amount of information related to information specifying a reference image is reduced can be decoded.

<Application example of hierarchical coding>

(First System)

A specific example of use of the hierarchically encoded (scalably encoded) scalable encoded data will be described below. As in the example illustrated in Figure 67, scalable coding is used, for example, to select data to be transmitted.

In the data transmission system 1000 illustrated in FIG. 67, the distribution server 1002 reads the scalable encoded data stored in the scalable encoded data storage unit 1001, and distributes the read scalable encoded data to the via the network 1003 A terminal device such as a personal computer 1004 , an AV device 1005 , a tablet device 1006 or a mobile phone 1007 .

At this time, the distribution server 1002 selects and transmits encoded data having an appropriate quality in accordance with the capabilities of the terminal device, the communication environment, and the like. Even when the distribution server 1002 transmits data of unnecessarily high quality, high-quality images cannot be obtained in the terminal device, and there is a concern that delay or overflow may occur. In addition, there is a concern that the communication bandwidth will be unnecessarily occupied or the load of the terminal device will be unnecessarily increased. On the contrary, when the distribution server 1002 transmits data of unnecessarily low quality, there is a concern that an image with sufficient picture quality cannot be obtained in the terminal device. Thereby, the distribution server 1002 appropriately reads and transmits the scalable encoded data stored in the scalable encoded data unit 1001 as encoded data having a quality suitable for the capabilities of the terminal device, the communication environment, and the like.

For example, it is assumed that the scalable encoded data storage unit 1001 holds the scalable encoded data (BL+EL) 1011 encoded in a scalable manner. This scalable encoded data (BL+EL) 1011 is encoded data including a base layer and an enhancement layer, and is data from which an image of the base layer and an image of the enhancement layer can be obtained by decoding the scalable encoded data.

The distribution server 1002 selects an appropriate layer according to the capability of the terminal transmitting the data, the communication environment, and the like, and reads the data of the layer. For example, for a personal computer 1004 or a tablet device 1006 having high processing power, the distribution server 1002 reads high-quality scalable encoded data (BL+EL) 1011 from the scalable encoded data storage unit 1001, and transmits the scalable encoded data as it is encoded data. In contrast, for example, for the AV equipment 1005 and the mobile phone 1007 having low processing power, the distribution server 1002 extracts the data of the base layer from the scalable encoded data (BL+EL) 1011, and transmits the content and the scalable encoded data ( Scalable coded data (BL) 1012 which is the same as BL+EL) 1011 but of lower quality than scalable coded data (BL+EL).

As described above, by utilizing scalable encoded data, the amount of data can be easily adjusted. Therefore, the occurrence of delay or overflow can be suppressed, and unnecessary increase in the load of the terminal device or the communication medium can be suppressed. In addition, in the scalable encoded data (BL+EL) 1011, since the redundancy between the layers is reduced, the amount of data can be reduced compared to the case where the encoded data of the layers are arranged as separate data . Thus, the storage area of the scalable encoded data storage unit 1001 can be used more efficiently.

In addition, similar to the personal computer 1004 and the mobile phone 1007, various devices can be used as terminal devices, so that the capabilities of the hardware of the terminal devices vary from device to device. Furthermore, since there are various application programs executed by the terminal device, there are various capabilities of the software. In addition, as the network 1003 serving as a communication medium, any one of all communication line networks can be applied, including wired and/or wireless networks such as the Internet and a LAN (Local Area Network), so that data transmission capabilities differ. Furthermore, there is a concern that the data transmission capability may vary according to other communications and the like.

Thus, before starting data transmission, the distribution server 1002 can communicate with a terminal device that is a transmission destination of data to obtain information on the capabilities of the terminal device, such as the hardware capabilities of the terminal device, and the terminal device execution The capabilities of the application (software) of the network, and information about the communication environment, such as the available bandwidth of the network 1003, etc. Additionally, distribution server 1002 may be configured to select the appropriate tier based on the information obtained herein.

In addition, the extraction of layers can be performed by the terminal device. For example, the personal computer 1004 can decode the transmitted scalable encoded data (BL+EL) 1011 and display the image of the base layer or the image of the enhancement layer. Also, for example, the personal computer 1004 may extract the base layer scalable encoded data (BL) 1012 from the transmitted scalable encoded data (BL+EL) 1011, save the extracted scalable encoded data, and store the extracted scalable encoded data. The data is transmitted to another device, or the extracted scalable encoded data is decoded, thereby displaying the image of the base layer.

Here, it is obvious that the numbers of scalable encoded data storage units 1001, distribution servers 1002, networks 1003 and terminal devices are arbitrary. In the above description, although the example in which the distribution server 1002 transmits data to the terminal device is described, the use example is not limited to this. The data transmission system 1000 can be applied to any system, as long as the system selects an appropriate layer according to the capabilities of the terminal device, the communication environment, etc., and transmits the selected layer when transmitting encoded data encoded in a scalable manner to the terminal device .

(Second system)

Additionally, as in the example illustrated in Figure 68, scalable encoding is used for transmission over a variety of communication media.

In the data transmission system 1100 illustrated in FIG. 68, the broadcasting station 1101 transmits the scalable encoded data (BL) 1121 of the base layer through the terrestrial broadcasting 1111. In addition, the broadcast station 1101 transmits scalable encoded data (EL) 1122 of the enhancement layer (eg, the data is packetized and transmitted) through any network 1112 consisting of wired and/or wireless communication networks.

The terminal device 1102 has a function of receiving the terrestrial broadcast 1111 broadcast by the broadcasting station 1101 , and receives the scalable encoded data (BL) 1121 of the base layer transmitted through the terrestrial broadcast 1111 . In addition, the terminal device 1102 also has a communication function to communicate through the network 1112 and receives the scalable encoded data (EL) 1122 of the enhancement layer transmitted through the network 1112 .

The terminal device 1102 obtains the image of the base layer by decoding the scalable coded data (BL) 1121 of the base layer obtained through the terrestrial broadcast 1111, for example, according to user instructions, etc., and saves the obtained scalable coded data, or converts the obtained scalable coded data. data to another device.

In addition, the terminal device 1102 combines the scalable coded data (BL) 1121 of the base layer obtained through the terrestrial broadcast 1111 and the scalable coded data (EL) 1122 of the enhancement layer obtained through the network 1112, for example, in accordance with a user's instruction or the like, thereby obtaining scalable Scale the coded data (BL+EL), decode the scalable coded data to obtain an image of the enhancement layer, save the obtained scalable coded data, or transmit the scalable coded data to another device.

As described above, scalable encoded data may be communicated over a communication medium that is different for each layer. Therefore, the load can be distributed, and the occurrence of delay or overflow can be suppressed.

Additionally, depending on the situation, the communication medium used for transmission may be configured to be selected for each layer. For example, the scalable coded data (BL) 1121 of the base layer may be configured to transmit a relatively large amount of data over a communication medium with a larger bandwidth, and the data of an enhancement layer of a smaller amount of data may be transmitted over a communication medium with a narrower bandwidth. Scalable encoded data (EL) 1122. In addition, the communication medium over which the scalable encoded data (EL) 1122 of the enhancement layer is transmitted may be configured to switch between the network 1112 and the terrestrial broadcast 1111, for example, according to the available bandwidth of the network 1112. This applies similarly to arbitrary layers of data.

By performing such control, it is possible to further suppress an increase in the data transmission load.

Here, the number of layers is arbitrary, as is the number of communication media used for transmission. In addition, the number of terminal apparatuses 1102 as the distribution destination of data is also arbitrary. Further, in the above description, although the example of broadcasting from the broadcasting station 1101 has been exemplified, the use example is not limited to this. The data transmission system 1100 can be applied to any system as long as the system divides the coded data that is expansibly encoded into a plurality of parts in units of layers, and transmits the divided data through a plurality of lines.

(Third system)

In addition, as in the example illustrated in FIG. 69, scalable encoded data is also used to hold encoded data.

In the imaging system 1200 illustrated in FIG. 69, the imaging apparatus 1201 performs scalable encoding of image data obtained by imaging a subject 1211, and takes the resulting image data as scalable encoded data (BL+EL ) 1221, provided to the scalable encoded data storage device 1202.

The scalable encoded data storage device 1202 stores the scalable encoded data (BL+EL) 1221 supplied from the imaging device 1201 with a quality corresponding to the situation. For example, in normal operation, the scalable coded data storage device 1202 extracts the data of the base layer from the scalable coded data (BL+EL) 1221, and saves the extracted data as a scalable base layer with low quality and a small amount of data. Scaled encoded data (BL) 1222. On the contrary, for example, when paying attention, the scalable encoded data storage device 1202 stores the scalable encoded data (BL+EL) 1221 of high quality and large amount of data as it is.

In this way, the scalable encoded data storage device 1202 can save images in high quality only when necessary. Accordingly, it is possible to suppress an increase in the amount of data, and at the same time, to suppress a decrease in image value caused by a decrease in image quality, whereby the use efficiency of the storage area can be improved.

For example, assume that the imaging device 1201 is a surveillance camera. In the case where a monitoring object (such as an intruder) does not appear in the captured image (at the normal time), there is a high possibility that the content of the captured image is not important, and therefore, priority is given to reduction in the amount of data, thereby saving image data with low quality (scalable encoded data). Conversely, when the monitoring object appears in the captured image as the subject 1211 (at the time of attention), the content of the captured image is highly likely to be important, and the image quality is given priority to save the image data ( scalable encoded data).

Here, by analyzing the image using the scalable coded data storage device 1202, it can be determined whether it is a normal time or an attention time. In addition, it may be configured that the determination process is performed by the imaging device 1201 and the determination result is transmitted to the scalable encoded data storage device 1202 .

Here, the criterion for determining the normal state or the time of interest is an arbitrary criterion, and the content of the image as the criterion for determining is an arbitrary content. In addition, conditions other than the content of the image may be set as the determination criteria. For example, the judgment may be changed according to the size, waveform, etc. of the recorded voice, the judgment may be changed every predetermined time, or the judgment may be changed according to an instruction supplied from the outside (such as a user instruction).

In addition, in the above description, although the example in which the switching between the two states of the normal time and the attention time is performed is described, the number of states is arbitrary. Thus, for example, it may be configured so as to switch between three or more states including normal time, weak focus time, focus time, and strong focus time. However, the upper limit of the number of states to switch between depends on the number of layers of scalable encoded data.

Furthermore, the imaging device 1201 may be configured to determine the number of layers of scalable encoding according to the state. For example, normally, the imaging device 1201 may be configured to generate scalable encoded data (BL) 1222 of a base layer of low quality and a small amount of data, and to provide the generated scalable encoded data to the scalable encoded data storage device 1202 . In addition, for example, the imaging device 1201 may generate scalable encoded data (BL+EL) 1221 of the base layer with high quality and a large amount of data, and supply the generated scalable encoded data to the scalable encoded data storage device, for example 1202.

In the above description, although a surveillance camera is exemplified, the application of this imaging system 1200 is arbitrary and is not limited to surveillance cameras.

Here, an LCU is a CU (coding unit) having the largest size, and a CTU (coding tree unit) is a unit including a CTB (coding tree block) of the LCU and parameters when processing is performed on an LCU basis (level). In addition, a CU constituting a CTU is a unit including a CB (coding block) and parameters when processing is performed on a CU basis (level).

<Other examples>

Although examples of devices, systems, and the like to which the present technology is applicable have been described above, the present technology is not limited thereto. Thus, it is possible to install to the device, or to make up the system in all configurations of the device, for example, as a processor of a system LSI (Large Scale Integration) or the like, a module using a plurality of processors, etc., using a plurality of modules, etc. The present technology is applied in the form of a unit, or a unit or the like obtained by adding other functions to the unit (in other words, the structure of a part of the device).

(Example of the structure of the video unit)

Next, with reference to FIG. 70 , an example of a case where the present technology is applied in the form of a unit will be described. FIG. 70 illustrates an example of a schematic structure of a video set to which the present technology is applicable.

Recently, multi-functionalization of electronic equipment is progressing, and in the development or manufacture of electronic equipment, in the case of providing a part of the structure for sale, supply, etc., there is not only a case where a structure having one function is applied , and there are widespread cases of applying one unit having multiple functions obtained by combining multiple structures having related functions.

The video set 1300 illustrated in FIG. 70 has such a multifunctional structure by combining a device having functions related to image encoding or image decoding (image encoding and/or image decoding) and other devices having functions related to the functions functional device obtained.

As illustrated in FIG. 70, the video set 1300 includes a module set including a video module 1311, an external memory 1312, a power management module 1313, a front-end module 1314, etc., and a module having related functions of a connection module 1321, a camera 1322, a sensor 1323, etc. equipment.

A module is formed by arranging the functions of several components related to each other in the form of components with unified functions. Although the specific physical structure is arbitrary, for example, it can be considered by arranging a plurality of processors each having a certain function, electronic circuit components such as resistors or capacitors, and other devices, etc., on a wiring board or the like. Modules that are consistently integrated together. In addition, it is contemplated that new modules may be formed by combining modules with other modules, processors, and the like.

In the example illustrated in FIG. 70 , the video module 1311 is obtained by combining structures having functions related to image processing, including: an application processor; a video processor; a broadband modem 1333 ; and an RF module 1334 .

A processor is obtained by integrating a structure having predetermined functions on a semiconductor chip in the form of an SoC (System on Chip), for example, there is also a processor called a system LSI (Large Scale Integration) or the like. The structure having a predetermined function may be a logic circuit (hardware structure), a structure including a CPU, ROM, RAM, etc. and a program (software structure) executed using them, or a structure combining the above two structures. For example, it may be constructed such that the processor includes logic circuits, CPU, ROM, RAM, etc., some functions are realized by logic circuits (hardware structure), and other functions are realized by programs (software structure) executed by the CPU.

The application processor 1331 illustrated in FIG. 70 is a processor that executes an application related to image processing. In order to realize predetermined functions, the application executed by the application processor 1331 may not only perform calculation processing, but also control structures inside and outside the video module 1311, such as the video processor 1332, if necessary.

The video processor 1332 is a processor having functions related to image encoding and image decoding (image encoding and/or image decoding).

Broadband modem 1333 is a processor (or module) associated with wired or wireless (or wired and wireless) broadband communications over a broadband line, such as the Internet or the public telephone network. For example, the broadband modem 1333 converts data to be transmitted (digital signal) into an analog signal by digital modulation or the like, or demodulates a received analog signal so as to be converted into data (digital signal). For example, the broadband modem 1333 may perform digital modulation/demodulation of arbitrary information, such as image data processed by the video processor 1332, the code stream in which the image data is encoded, and application and setting data.

The RF module 1334 is a module that performs frequency conversion, modulation/demodulation, amplification, filtering, and the like on the RF (Radio Frequency) signal transmitted/received through the antenna. For example, the RF module 1334 generates an RF signal by performing frequency conversion or the like on the dedicated line connection system signal generated by the broadband modem 1333 . In addition, for example, the RF module 1334 generates a dedicated line connection system signal by performing frequency conversion or the like on the RF signal received via the front-end module 1314 .

In addition, as indicated by the dotted line 1341 in FIG. 70, the application processor 1331 and the video processor 1332 may be integrated so as to be constituted as one processor.

The external memory 1312 is a module arranged outside the video module 1311 and including a storage device used by the video module 1311 . The storage device of the external memory 1312 is implemented by a certain physical structure. However, in general, since memory devices are often used to store large-capacity data, such as image data constituted in units of frames, it is best to use low-cost large-capacity semiconductor memories such as DRAM (Dynamic Random Access) as the memory device. memory) implementation.

The power management module 1313 manages and controls power supply to the video module 1311 (various components within the video module 1311).

The front-end module 1314 is a module that provides front-end functions (transmission/reception side circuits on the antenna side) to the RF module 1334 . As illustrated in FIG. 70 , the front end module 1314 includes, for example, an antenna unit 1351 , a filter 1352 and an amplification unit 1353 .

The antenna unit 1351 includes an antenna that transmits/receives wireless signals and its peripheral structures. The antenna unit 1351 transmits the signal supplied from the amplifying unit 1353 in the form of a wireless signal, and supplies the received wireless signal to the filter 1352 in the form of an electric signal (RF signal). The filter 1352 performs filtering processing and the like on the RF signal received through the antenna unit 1351, and supplies the processed RF signal to the RF module 1334. The amplifying unit 1353 amplifies the RF signal supplied from the RF module 1334, and supplies the amplified RF signal to the antenna unit 1351.

The connection module 1321 is a module having a function related to connection with the outside. The physical structure of the connection module 1321 is arbitrary. For example, the connection module 1321 includes a structure having a communication function other than the communication specification to which the broadband modem 1333 corresponds, an external input/output terminal, and the like.

For example, the connection module 1321 may be configured to include devices compliant with wireless communication specifications such as Bluetooth (registered trademark), IEEE 802.11 (eg, Wi-Fi (Wireless Fidelity; registered trademark)), NFC (Near Field Communication), and IrDA ( Infrared data transfer) the communication function module, and the antenna to transmit/receive the signal conforming to the stated specification. In addition, for example, the connection module 1321 may be configured to include a module having a communication function conforming to wired communication specifications such as USB (Universal Serial Bus) and HDMI (registered trademark) (High Definition Multimedia Interface), and a terminal conforming to the specifications . Also, for example, the connection module 1321 may be configured to have additional data (signal) transmission functions of analog input/output terminals and the like, and the like.

In addition, the connection module 1321 may be configured to include a device as a transmission destination of data (signal). For example, the connection module 1321 may be configured to include a drive for data reading or data writing to a recording medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory (including not only a removable media drive, but also a hard disk, an SSD (Solid State) drives), NAS (Network Attached Storage), etc.). In addition, the connection module 1321 may be configured to include an output device (monitor, speaker, etc.) of images or audio.

The camera 1322 is a module having a function of obtaining image data of a subject by imaging the subject. Image data obtained by imaging processing by the camera 1322 is supplied to the video processor 1332 and encoded.

The sensor 1323 has any sensor, such as audio sensor, ultrasonic sensor, light sensor, illuminance sensor, infrared sensor, image sensor, rotation sensor, angle sensor, angular velocity sensor, speed sensor, acceleration sensor, tilt sensor, magnetic identification sensor, impact sensor or temperature sensor function module. The data detected by the sensor 1323 is supplied to the application processor 1331 and used by applications and the like.

In the above description, each structure described as a module can be implemented by a processor, and each structure described as a processor can be implemented by a module.

As will be described later, the present technology can be applied to the video processor 1332 of the video set 1300 having the structure as described above. Thus, the video set 1300 can be configured as a device to which the technology is applicable.

(Configuration example of video processor)

71 illustrates an example of a schematic structure of the video processor 1332 (FIG. 70) to which the present technology is applicable.

In the example illustrated in FIG. 71, the video processor 1332 has a function of receiving input of a video signal and an audio signal, and encoding the received signal in a predetermined manner, and decoding the encoded video data and the encoded audio data, And the function of reproducing and outputting video signal and audio signal.

As illustrated in FIG. 71, the video processor 1332 includes a video input processing unit 1401; a first image enlargement/reduction unit 1402; a second image enlargement/reduction unit 1403; a video output processing unit 1404; a frame memory 1405; and a storage control unit 1406. In addition, the video processor 1332 includes: an encoding/decoding engine 1407; video ES (Elementary Stream) buffers 1408A and 1408B, and audio ES buffers 1409A and 1409B. In addition, the video processor 1332 includes: an audio encoder 1410; an audio decoder 1411; a multiplexer (MUX) 1412; a demultiplexer (DMUX) 1413;

For example, the video input processing unit 1401 obtains a video signal input from the connection module 1321 (FIG. 70) or the like, and converts the obtained video signal into digital image data. The first image enlargement/reduction unit 1402 performs format conversion and image enlargement/reduction processing on the image data. For image data, the second image enlargement/reduction unit 1403 performs image enlargement/reduction processing in accordance with the format of the output destination via the video output processing unit 1404, or performs format conversion and image processing similar to the first image enlargement/reduction unit 1402 Zoom in/out processing, etc. The video output processing unit 1404 performs format conversion, conversion into an analog signal, etc. of the image data, and outputs the resulting signal as a reproduced video signal to the connection module 1321 (FIG. 70) and the like.

The frame memory 1405 is a memory for image data shared by the video input processing unit 1401 , the first image enlargement/reduction unit 1402 , the second image enlargement/reduction unit 1403 , the video output processing unit 1404 , and the encoding/decoding engine 1407 . The frame memory 1405 is implemented as a semiconductor memory such as DRAM.

The storage control unit 1406 receives the synchronization signal supplied from the encoding/decoding engine 1407, and controls access to the frame memory 1405 in accordance with the access schedule to the frame memory 1405 in the write access management table 1406A to write/ read. The storage control unit 1406 updates the access management table 1406A according to the processing performed by the encoding/decoding engine 1407, the first image enlargement/reduction unit 1402, the second image enlargement/reduction unit 1403, and the like.

The encoding/decoding engine 1407 performs encoding processing of image data, and performs decoding processing of a video stream obtained by encoding the image data. For example, the encoding/decoding engine 1407 encodes image data read from the frame memory 1405, and sequentially writes the read image data as a video stream to the video ES buffer 1408A. In addition, for example, the encoding/decoding engine 1407 sequentially reads video streams from the video ES buffer 1408B, decodes the read video streams, and sequentially writes the decoded video streams as image data to the frame memory 1405. The encoding/decoding engine 1407 utilizes the frame memory 1405 as a work area in this encoding or decoding process. In addition, the encoding/decoding engine 1407 outputs a synchronization signal to the storage control unit 1406, for example, when the processing of each macroblock is started.

The video ES buffer 1408A buffers the video stream generated by the encoding/decoding engine 1407 and supplies the video stream to the multiplexer (MUX) 1412 . The video ES buffer 1408B buffers the video stream supplied from the demultiplexer (DMUX) 1413, and supplies the video stream to the encoding/decoding engine 1407.

The audio ES buffer 1409A buffers the audio stream generated by the audio encoder 1410 and supplies the audio stream to the multiplexer (MUX) 1412 . The audio ES buffer 1409B buffers the audio stream supplied from the demultiplexer (DMUX) 1413 and supplies the audio stream to the audio decoder 1411.

The audio encoder 1410 converts, for example, an audio signal input from the connection module 1321 (FIG. 70) into, for example, a digital signal, and encodes the converted audio signal in a manner such as MPEG audio or AC3 (audio code number 3). The audio encoder 1410 sequentially writes the audio stream, which is data obtained by encoding the audio signal, into the audio ES buffer 1409A. The audio decoder 1411 decodes the audio stream supplied from the audio ES buffer 1409B, converts the decoded audio stream into an analog signal or the like, and supplies the converted signal as a reproduced audio signal to, for example, the connection module 1321 (FIG. 70) or the like.

A multiplexer (MUX) 1412 multiplexes the video stream and the audio stream. The multiplexing method (in other words, the format of the bitstream generated by multiplexing) is arbitrary. In addition, at the time of multiplexing, the multiplexer (MUX) 1412 may add predetermined header information and the like to the bit stream. In other words, the multiplexer (MUX) 1412 can convert the format of the stream through the multiplexing process. For example, by multiplexing the video stream and the audio stream, the multiplexer (MUX) 1412 converts the video stream and the audio stream into a transport stream which is a bit stream having a format for transport. In addition, for example, by multiplexing the video stream and the audio stream, the multiplexer (MUX) 1412 converts the video stream and the audio stream into data (file data) having a format for recording.

The demultiplexer (DMUX) 1413 demultiplexes the bit stream in which the video stream and the audio stream are multiplexed using a method corresponding to the multiplexing process performed by the multiplexer (MUX) 1412 . In other words, the demultiplexer (DMUX) 1413 extracts the video stream and the audio stream (the video stream and the audio stream are separated) from the bit stream read from the stream buffer 1414. In other words, the demultiplexer (DMUX) 1413 can convert the format of the stream (inverse transformation of the transformation by the multiplexer (MUX) 1412 ) through the demultiplexing process. For example, the demultiplexer (DMUX) 1413 obtains, for example, the transport stream supplied from the connection module 1321 (FIG. 70), the broadband modem 1333 (FIG. 70), etc. through the stream buffer 1414, and demultiplexes the obtained transport stream, thereby transferring the transport stream to Switch to video streaming and audio streaming. In addition, for example, the demultiplexer (DMUX) 1413 obtains file data read from various recording media by the connection module 1321 (FIG. 70) through the stream buffer 1414, and demultiplexes the obtained file data, thereby converting the file data into Video streaming and audio streaming.

Stream buffer 1414 buffers the bit stream. For example, the stream buffer 1414 buffers the transport stream supplied from the multiplexer (MUX) 1412, and supplies the transport stream to, for example, the connection module 1321 (FIG. 70), the broadband modem 1333 at predetermined timing or in accordance with a request transmitted from the outside (Fig. 70) and so on.

In addition, the stream buffer 1414 buffers the file data supplied from the multiplexer (MUX) 1412, and supplies the file data to, for example, the connection module 1321 (FIG. 70) or the like at predetermined timing or according to a request transmitted from the outside.

In addition, the stream buffer 1414 buffers, for example, the transport stream obtained by the connection module 1321 (FIG. 70), the broadband modem 1333 (FIG. 70), etc., and supplies the transport stream to the demultiplexer at predetermined timing, or in accordance with a request from the outside, or the like device (DMUX) 1413.

In addition, the stream buffer 1414 buffers the file data read from various recording media by the connection module 1321 (FIG. 70), for example, and supplies the file data to the demultiplexer ( DMUX) 1413.

Next, an example of the operation of the video processor 1332 having such a configuration will be described. For example, the video signal input to the video processor 1332 from the connection module 1321 (FIG. 70) or the like is converted into digital image data by the video input processing unit 1401 in a predetermined manner such as the 4:2:2Y/Cb/Cr method, It is then sequentially written into the frame memory 1405. The digital image data is read by the first image enlargement/reduction unit 1402 or the second image enlargement/reduction unit 1403, and the digital image data is subjected to a predetermined method such as a 4:2:0Y/Cb/Cr method. Format conversion, and enlargement/reduction processing, the processed digital image data is written into the frame memory 1405 . The image data is encoded by the encoding/decoding engine 1407, and then written into the video ES buffer 1408A as a video stream.

In addition, the audio signal input to the video processor 1332 from the connection module 1321 (FIG. 70) or the like is encoded by the audio encoder 1410, and then written into the audio ES buffer 1409A as an audio stream.

The video stream held in the video ES buffer 1408A and the audio stream held in the audio ES buffer 1409A are read by the multiplexer (MUX) 1412, multiplexed, and converted into transport streams, file data, and the like. The transport stream generated by the multiplexer (MUX) 1412 is buffered in the stream buffer 1414, and then output to an external network, for example, through the connection module 1321 (FIG. 70), the broadband modem 1333 (FIG. 70), and the like. In addition, the file data generated by the multiplexer (MUX) 1412 is buffered in the stream buffer 1414, then output to, for example, the connection module 1321 (FIG. 70) or the like, and then recorded in any one of various recording media.

In addition, through the connection module 1321 (FIG. 70), the broadband modem 1333 (FIG. 70), etc., the transport stream input to the video processor 1332 from the external network is buffered in the stream buffer 1414, and then demultiplexed by the demultiplexer (DMUX) 1413 use. In addition, using the connection module 1321 (FIG. 70) or the like, file data read from any one of various recording media and input to the video processor 1332 is buffered in the stream buffer 1414, and then demultiplexed (DMUX) 1413 points are used. In other words, the transport stream or file data input to the video processor 1332 is separated by the demultiplexer (DMUX) 1413 into a video stream and an audio stream.

The audio stream passes through the audio ES buffer 1409B, is supplied to the audio decoder 1411, and is then decoded, thereby reproducing the audio signal. In addition, the video stream is written into the video ES buffer 1408B, then sequentially read by the encoding/decoding engine 1407, decoded, and then written into the frame memory 1405. The decoded image data is enlarged or reduced by the second image enlargement/reduction unit 1403 and then written into the frame memory 1405 . Subsequently, the decoded image data is read by the video output processing unit 1404, format-converted into a predetermined format such as a 4:2:2Y/Cb/Cr format, and then further converted into an analog signal, so that the video signal is reproduced and output.

In the case where the present technology is applied to the video processor 1332 thus constituted, the present technology according to the above-described respective embodiments can be applied to the encoding/decoding engine 1407 . In other words, the encoding/decoding engine 1407 may be configured to have the functions of the encoding device 10 or the decoding device 110 . In addition, for example, the encoding/decoding engine 1407 may be configured to have the functions of the encoding device 150 and the decoding device 170 , the encoding device 190 and the decoding device 210 , or the encoding device 230 and the decoding device 270 . Also, for example, the encoding/decoding engine 1407 may be configured to have the functions of the multi-viewpoint image encoding apparatus 600 and the multi-viewpoint image decoding apparatus 610. By so constituted, the video processor 1332 can obtain the same advantages as those described above with reference to FIGS. 1-61.

In addition, in the encoding/decoding engine 1407, the present technology (in other words, the functions of the image encoding device and the image decoding device according to the above-described respective embodiments) can be implemented using hardware such as logic circuits, and can be implemented using hardware such as built-in programs. A software implementation of the class, or it can be implemented using both hardware and software.

(Another structural example of a video processor)

FIG. 72 is a diagram illustrating another example of a schematic structure of the video processor 1332 ( FIG. 70 ) to which the present technology is applied. In the case of the example illustrated in Fig. 72, the video processor 1332 has a function of encoding/decoding video data in a predetermined manner.

More specifically, as illustrated in FIG. 72 , the video processor 1332 includes: a control unit 1511; a display interface 1512; a display engine 1513; an image processing engine 1514; In addition, video processor 1332 includes: codec engine 1516; storage interface 1517; multiplexer/demultiplexer (MUX DMUX) 1518; network interface 1519;

The control unit 1511 controls operations of processing units arranged within the video processor 1332 , such as the display interface 1512 , the display engine 1513 , the image processing engine 1514 and the codec engine 1516 .

As illustrated in FIG. 72 , for example, the control unit 1511 includes a main CPU 1531 , a sub CPU 1532 , and a system controller 1533 . The main CPU 1531 executes programs for controlling the operations of the respective processing units arranged in the video processor 1332 . The main CPU 1531 generates control signals in accordance with the program and the like, and supplies the control signals to the respective processing units (in other words, controls the operations of the respective processing units). The sub CPU 1532 performs the auxiliary role of the main CPU 1531. For example, the sub CPU 1532 executes subprocessing, subroutines, and the like of the program executed by the main CPU 1531 . The system controller 1533 controls operations of the main CPU 1531 and the sub CPU 1532 , such as designation of programs to be executed by the main CPU 1531 and the sub CPU 1532 .

Under the control of the control unit 1511, the display interface 1512 outputs image data to, for example, the connection module 1321 (FIG. 70) or the like. For example, the display interface 1512 converts image data as digital data into an analog signal, and outputs the image data as a reproduced video signal or image data as digital data to a monitoring device or the like of the connection module 1321 (FIG. 70).

Under the control of the control unit 1511, the display engine 1513 performs various conversion processes such as format conversion, size conversion, and color gamut conversion on the image data so as to be adjusted to conform to hardware specifications of a monitoring device or the like that displays the image.

Under the control of the control unit 1511, the image processing engine 1514 performs predetermined image processing such as filtering processing to improve image quality on the image data.

The internal memory 1515 is a memory arranged within the video processor 1332 and shared by the display engine 1513 , the image processing engine 1514 and the codec engine 1516 . For example, the internal memory 1515 is used for data exchange between the display engine 1513 , the image processing engine 1514 and the codec engine 1516 . For example, internal memory 1515 holds data supplied from display engine 1513, image processing engine 1514, or codec engine 1516, and provides data to display engine 1513, image processing engine 1514, or codec when necessary (eg, on request). Decoder Engine 1516. Although the internal memory 1515 can be implemented with any storage device, in general, the internal memory 1515 is often used to store data with a small capacity, such as image data constituted by blocks, or parameters, so it is best to use the capacity A semiconductor memory such as SRAM (Static Random Access Memory) that is small (for example, compared to the external memory 1312) and has a high response speed is realized.

The codec engine 1516 performs processing related to encoding or decoding image data. The encoding/decoding methods corresponding to the codec engine 1516 are arbitrary, and the number may be one or two or more. For example, the codec engine 1516 may include a codec function of a plurality of encoding/decoding methods, and perform encoding or encoding of image data by using a selected one encoding/decoding method among the plurality of encoding/decoding methods decoding of the image data.

In the example illustrated in FIG. 72 , as functional blocks of codec-related processing, the codec engine 1516 includes, for example, MPEG-2 Video 1541, AVC/H.264 1542, HEVC/H.265 1543, HEVC/H.265 (scalable) 1544, HEVC/H.265 (multi-view) 1545, and MPEG-DASH 1551.

MPEG-2 Video 1541 is a functional block for encoding or decoding image data according to the MPEG-2 method. AVC/H.264 1542 is a functional block for encoding or decoding image data according to the AVC method. In addition, HEVC/H.2651543 is a functional block for encoding or decoding image data according to the HEVC method. HEVC/H.265 (Scalable) 1544 is a functional block for scalable encoding or scalable decoding of image data in the HEVC manner. HEVC/H.265 (multi-view) 1545 is a functional block for performing multi-view encoding or multi-view decoding on image data according to the HEVC method.

The MPEG-DASH 1551 is a functional block for transmitting/receiving image data according to the MPEG-DASH (MPEG-Dynamic Adaptive Streaming over HTTP) method. MPEG-DASH is a technology for streaming video by using HTTP (Hypertext Transfer Protocol), which selects and transmits one piece of coded data that differs from each other in resolutions prepared in advance. Features of encoded data. MPEG-DASH 1551 performs specification-compliant stream generation, stream transmission control, etc., and encodes/decodes image data using MPEG-2 Video 1541 or HEVC/H.265 (multi-view) 1545 explained above.

The storage interface 1517 is an interface for the external memory 1312 . Data supplied from the image processing engine 1514 or the codec engine 1516 is supplied to the external memory 1312 through the storage interface 1517 . In addition, the data read from the external memory 1312 is supplied to the video processor 1332 (the image processing engine 1514 or the codec engine 1516 ) through the storage interface 1517 .

A multiplexer/demultiplexer (MUX DMUX) 1518 multiplexes or demultiplexes various data related to images, such as bitstreams of encoded data, image data, or video signals. The multiplexing/demultiplexing method is arbitrary. For example, at the time of multiplexing processing, the multiplexer/demultiplexer (MUX DMUX) 1518 may not only arrange a plurality of data into one data, but also add predetermined header information to the data. In addition, at the time of demultiplexing, the multiplexer/demultiplexer (MUX DMUX) 1518 can not only divide one data into a plurality of parts, but also can add predetermined header information and the like to the divided data. In other words, the multiplexer/demultiplexer (MUX DMUX) 1518 can convert the format of the data through the multiplexing/demultiplexing process. For example, the multiplexer/demultiplexer (MUXDMUX) 1518 can multiplex the bit stream to convert the bit stream into a transport stream in a transport format, or into data (file data) in a file format for recording. Obviously, through the demultiplexing process, the inverse transformation can be performed.

Network interface 1519 is a dedicated interface such as broadband modem 1333 (FIG. 70) or connection module 1321 (FIG. 70). Video interface 1520 is a dedicated interface such as connection module 1321 (FIG. 70) or camera 1322 (FIG. 70).

Next, an example of the operation of such a video processor 1332 will be described. For example, when a transport stream is received from an external network through the connection module 1321 (FIG. 70), the broadband modem 1333 (FIG. 70), etc., the transport stream is provided to the multiplexer/demultiplexer (MUX) through the network interface 1519. DMUX) 1518, is demultiplexed and decoded by the codec engine 1516. For the image data obtained by the decoding process performed by the codec engine 1516, the image processing engine 1514 performs predetermined image processing, the display engine 1513 performs predetermined transformation, and the resulting image data is supplied to the connection module 1321 ( 70), etc., so that the image is displayed on the monitor. In addition, for example, image data obtained by decoding processing by the codec engine 1516 is re-encoded by the codec engine 1516, multiplexed by the multiplexer/demultiplexer (MUXDMUX) 1518, converted into file data, passed through the video The interface 1520 is output to, for example, the connection module 1321 (FIG. 70) and the like, and is recorded on any one of various recording media.

In addition, for example, encoded data obtained by encoding image data read from a recording medium not shown in the drawing using the connection module 1321 (FIG. 70) or the like is supplied to the multiplexer/demultiplexer via the video interface 1520. Demultiplexer (MUXDMUX) 1518, demultiplexed, and decoded by codec engine 1516. For image data obtained by decoding processing by the codec engine 1516, the image processing engine 1514 performs predetermined image processing, the display unit 1513 performs predetermined transformation, and the resulting image data is supplied to, for example, the connection module 1321 ( 70) etc., and then the image is displayed on the monitor. Also, for example, image data obtained by decoding processing by the codec engine 1516 is re-encoded by the codec engine 1516, multiplexed by the multiplexer/demultiplexer (MUX DMUX) 1518, converted into a transport stream, and passed through The network interface 1519 is provided, for example, to the connection module 1321 (FIG. 70), the broadband modem 1333 (FIG. 70), etc., and then to another device not shown in the figure.

In addition, using the internal memory 1515 or the external memory 1312, exchange of image data or other data between processing units arranged within the video processor 1332 is performed. In addition, the power management module 1313 controls the power supply to the control unit 1511, for example.

In the case where the present technology is applied to the video processor 1332 thus constituted, the present technology according to the above-described respective embodiments can be applied to the codec engine 1516 . In other words, the codec engine 1516 may include functional blocks that implement the encoding apparatus 10 or the decoding apparatus 110, for example. Additionally, codec engine 1516 may be configured to include functional blocks that implement encoding device 150 and decoding device 170, encoding device 190 and decoding device 210, or encoding device 230 and decoding device 270, for example. Also, for example, the codec engine 1516 may be configured to include the functions of the multi-viewpoint image encoding apparatus 600 and the multi-viewpoint image decoding apparatus 610 . By so constructed, the video processor 1332 can obtain the same advantages as those described above with reference to FIGS. 1-61.

In addition, in the codec engine 1516, the present technology (in other words, the functions of the image encoding device and the image decoding device according to the above-described respective embodiments) can be implemented using hardware such as a logic circuit, and can be implemented using a program such as a built-in program. A software implementation of the class, or it can be implemented using both hardware and software.

As above, although two structures of the video processor 1332 have been described as an example, the structure of the video processor 1332 is arbitrary and may be a structure other than the above two structures. In addition, the video processor 1332 may be composed of one semiconductor chip or a plurality of semiconductor chips. For example, the video processor 1332 may be constructed of a three-dimensional stacked LSI in which a plurality of semiconductors are stacked. In addition, the video processor 1332 may be implemented with a plurality of LSIs.

(Example of application to equipment)

The video set 1300 may be built in various devices that process image data. For example, the video set 1300 may be built in a television set 900 (FIG. 63), a mobile phone 920 (FIG. 64), a recording and reproducing apparatus 940 (FIG. 65), an imaging apparatus 960 (FIG. 66), and the like. By having the video set 1300 built into it, the device achieves the same advantages as described above with reference to Figures 1-61.

In addition, the video set 1300 may be built in, for example, terminal devices of the data transmission system 1000 illustrated in FIG. 67, such as a personal computer 1004, an AV device 1005, a tablet device 1006, and a mobile phone 1007, and the data illustrated in FIG. 68 In the broadcasting station 1101 and the terminal device 1102 of the transmission system 1100, and the imaging device 1201 and the scalable encoded data storage device 1202 of the imaging system 1200 illustrated in FIG. 69, and the like. By having the video set 1300 built into it, the device achieves the same advantages as described above with reference to Figures 1-61.

Furthermore, some of the various structures of the video group 1300 described above may be structures to which the present technology is applicable if the video processor 1332 is included therein. For example, only video processor 1332 may be configured as a video processor for which this technology is applicable. Additionally, as described above, the processor, video module 1311, etc. represented by dashed line 1341 may be configured with processors, modules, etc. to which this technology is applicable. Furthermore, for example, the video module 1311, the external memory 1312, the power management module 1313, and the front-end module 1314 may be combined so as to configure the video unit 1361 to which the technology is applicable. In any of the described configurations, the same advantages as described above with reference to Figures 1-61 can be obtained.

In other words, similar to the case of the video set 1300, any structure including the video processor 1332 may be built into various devices that process image data. For example, the video processor 1332, the processor and video module 1311 represented by the dotted line 1341, or the video unit 1361 may be built in the television set 900 (FIG. 63), the mobile phone 920 (FIG. 64), the recording and reproducing device 940 (FIG. 64) 65 ), the imaging device 960 ( FIG. 66 ), the terminal devices of the data transmission system 1000 illustrated in FIG. In the broadcasting station 1101 and the terminal device 1102 of the data transmission system 1100 shown, and the imaging device 1201 and the scalable encoded data storage device 1202 of the imaging system 1200 illustrated in FIG. 69 , and the like. By having built into it any structure to which the present technology is applicable, similar to the case of video set 1300, the device can achieve the same advantages as described above with reference to Figures 1-61.

In this specification, an example is described in which various kinds of information are multiplexed into an encoded stream, and the encoded stream is transmitted from the encoding side to the decoding side. However, the technique for transmitting information is not limited to this. For example, information may be transmitted or recorded as separate data associated with the encoded bitstream, rather than multiplexed into the encoded bitstream. Here, the term "associated with" means that a picture (slice, block, etc.; it may be a part of the picture) contained in the bitstream and the information corresponding to the picture are associated with each other at the time of the decoding process. In other words, the information may be transmitted on a transmission line different from that of the image (or bitstream). Furthermore, the information may be recorded on a recording medium (or a different storage area of the same recording medium) that is different from that of the image (or bitstream). In addition, the information and the image (or bitstream) may be correlated with each other in arbitrary units, such as multiple frames, a frame, or a portion of a frame.

The present technique is applicable when transmitting and receiving through a network medium such as satellite broadcasting, cable TV, the Internet or a mobile phone as in H.26x etc. by orthogonal transform such as discrete cosine transform and motion compensation compression A device used when processing compressed image information on storage media such as optical discs, magnetic disks or flash memory.

In addition, the present technology is applicable to, for example, HTTP streaming, such as MPEG DASH, in which appropriate encoded data is selected and used from a plurality of encoded data whose resolutions and the like differ from each other in units of segments.

Also, the encoding system according to the present technology may be an encoding system other than the HEVC system.

Embodiments of the present technology are not limited to the above-described embodiments, and various changes may be made without departing from the principles of the present technology.

In addition, the present technology may have the following structures.

(1) An encoding device, comprising:

a predicted image generation unit configured to generate a predicted image using the reference image; and

a transmission unit configured to, in the case where the currently encoded picture is a picture other than the 1st picture of a GOP (Group of Pictures), transmit reference information representing a reference to a specified reference picture of the preceding picture Whether or not the picture specifying information is used as reference picture specifying information for the current coded picture, which is the picture preceding the current coded picture in the coding order.

(2) The encoding apparatus according to (1), wherein in the case where the reference information indicates that the reference picture specifying information of the previous picture is used as the reference picture specifying information of the current encoded picture, the transmission unit transmits the previous specifying the previous picture Image specification information.

(3) The encoding device according to (2), wherein the transmission unit transmits the reference image designation of the current encoded image in a case where the reference information indicates that the reference image designation information of the previous image is not used as the reference image designation information of the current encoded image information.

(4) The encoding apparatus according to (3), further comprising a reference image information setting unit configured to set a plurality of reference image information including reference information and prior image designation information or reference image designation information,

wherein the transmission unit transmits a plurality of reference picture information set by the reference picture information setting unit, and transmits the plurality of reference pictures in the case where the currently encoded picture is a picture other than the first picture of a GOP (Group of Pictures) Among the picture information, reference picture information specifying information specifying reference picture information of the currently encoded picture.

(5) According to the encoding device described in (4),

wherein the reference image information setting unit sets the first reference image information including the reference image designation information as the reference image information, and

In a case where the currently encoded picture is the first picture of a GOP (Group of Pictures), the transmission unit transmits reference picture information specifying information specifying the first reference picture information.

(6) a kind of coding method, described coding method comprises the following steps by means of coding equipment:

a predicted image generation step that utilizes the reference image to generate a predicted image; and

A transmission step of transmitting reference information indicating reference picture designation information for designating a reference picture of a preceding picture in a case where the currently encoded picture is a picture other than the first picture of a GOP (Group of Pictures) Whether to use as reference picture designation information for the current coded picture, which is the picture preceding the current coded picture in coding order.

List of reference signs

10 encoding equipment

12 Setting unit

13 Transmission unit

33 Computing Units

47 Motion prediction/compensation unit

110 decoding equipment

111 Receiver unit

135 Addition unit

144 Reference image setting unit

145 Motion Compensation Unit

150 encoding equipment

170 decoding equipment

190 encoding equipment

210 decoding equipment

230 encoding equipment

232 Setting unit

251 Motion prediction/compensation unit

270 decoding equipment

292 Motion Compensation Unit

Claims (8)

1. A decoding device, comprising:

at least one processor;

at least one memory including a computer program, the memory and computer program configured to, working with the at least one processor, cause the decoding apparatus to perform at least the following:

decoding a syntax element received in a sequence parameter set ("SPS") of a plurality of pictures to be decoded, the syntax element indicating a plurality of short-term reference picture sets ("RPSs") included in the SPS;

setting, in a slice header of a current picture of the plurality of pictures, an index value of a short-term RPS of the current picture equal to a number of short-term RPSs included in the SPS;

setting a prediction flag value to a first value indicating that a short-term RPS included in the SPS is not predicted to a short-term RPS of the current picture if an index value of the short-term RPS of the current picture is equal to zero indicating that the prediction flag value is not received in a slice header of the current picture;

generating a predicted image of the current image; and

decoding the current picture using the predicted picture.

2. The decoding device according to claim 1, wherein performing generating a predicted image of the current image further comprises:

decoding the prediction flag value if an index value of a short-term RPS of the current picture is not equal to zero indicating that the prediction flag value is not received in a slice header of the current picture, wherein the prediction flag value is either the first value indicating that the short-term RPS of the current picture is not predicted from the short-term RPS included in the SPS or a second value indicating that the short-term RPS of the current picture is predicted from the short-term RPS included in the SPS; and

generating a short-term RPS for the current picture based on the predictive flag value; and

generating a predicted image of the current picture using the short-term RPS of the current picture.

3. The decoding device according to claim 2, wherein performing generating a predicted image of the current image further comprises:

decoding an RPS delta index received in a slice header of the current picture if the prediction flag value is equal to a second value indicating that a short-term RPS of the current picture is predicted from a short-term RPS included in the SPS; and

generating a short-term RPS for the current picture based on the RPS delta index; and

a predicted image of a current picture is generated using a short-term RPS of the current picture.

4. The decoding device according to any one of claims 1 or 2, wherein generating a predicted image of the current picture further comprises the operations of:

decoding short-term RPS specification information for a current picture received in a slice header of the current picture if the prediction flag value is equal to a first value indicating that a short-term RPS for the current picture is not predicted from a short-term RPS included in the SPS, wherein the slice header of the current picture does not include the RPS delta index; and

generating a short-term PRS for the current picture based on the short-term RPS specification information; and

generating a predicted image of the current picture using the short-term RPS of the current picture.

5. A decoding method comprising the steps of:

decoding a syntax element received in a sequence parameter set ("SPS") of a plurality of pictures to be decoded, the syntax element serving as an indicator of a plurality of short-term reference picture sets ("RPSs") included in the SPS;

setting, in a slice header of a current picture of the plurality of pictures, an index value of a short-term RPS of the current picture equal to a number of short-term RPSs included in the SPS;

setting a prediction flag value to a first value if an index value of the short-term RPS of the current picture is equal to zero, which is an indicator that the prediction flag value is not received in a slice header of the current picture, the first value being an indicator that the short-term RPS of the current picture is not predicted from the short-term RPS included in the SPS;

generating a predicted image of the current image; and

decoding the current picture using the predicted picture.

6. The decoding method according to claim 5, wherein the step of generating a predictive image of the current picture further comprises the steps of:

decoding a prediction flag value if an index value of a short-term RPS of the current picture is not equal to zero, which is an indicator that the prediction flag value is not received in a slice header of the current picture, wherein the prediction flag value is the first value indicating that the short-term RPS of the current picture is not predicted from the short-term RPS included in the SPS, or the second value indicating that the short-term RPS of the current picture is predicted from the short-term RPS included in the SPS; and

generating a short-term RPS for the current picture based on the predictive flag value; and

generating a predicted image of the current picture using the short-term RPS of the current picture.

7. The decoding method according to claim 6, wherein the step of generating a predictive image of the current image further comprises the steps of:

decoding an RPS delta index received in a slice header of the current picture if the prediction flag value is equal to a second value that is an indicator that the short-term RPS of the current picture is predicted from the short-term RPS contained in the SPS; and

generating a short-term RPS for the current picture based on the RPS delta index; and

a predicted image of a current picture is generated using a short-term RPS of the current picture.

8. The decoding method according to any one of claims 5 or 6, wherein the step of generating a predictive image of the current picture further comprises:

decoding short-term RPS specification information of a current picture received in a slice header of the current picture if the prediction flag value is equal to a first value which is an indicator that the short-term RPS of the current picture is not predicted from the short-term RPS included in the SPS, wherein the slice header of the current picture does not include the RPS delta index; and

generating a short-term PRS for the current picture based on the short-term RPS specification information; and

generating a predicted image of the current picture using the short-term RPS of the current picture.

Images