JP6614935B2 - Video encoding apparatus and program - Google Patents

Description

  The present invention relates to a video encoding device and a program.

  As a video compression technique, a method using a super-resolution technique is disclosed (for example, Patent Document 1). In the technique of Patent Document 1, the resolution of the input video is temporarily reduced on the transmission side. Then, the video with reduced resolution is encoded and compressed by an encoding apparatus using an existing video encoding method, and transmitted as a bit string. On the other hand, on the receiving side, the received bit string is decoded by a decoding device using an existing video encoding method. The video signal obtained by this decoding is restored to the original resolution by super-resolution processing.

  At this time, super-resolution processing is also attempted on the transmission side, and super-resolution parameters used when performing super-resolution processing while referring to the input video are calculated. The calculated super-resolution parameter is transmitted as auxiliary information from the transmission side to the reception side, and is used for the super-resolution processing on the reception side. As described above, in the super-resolution processing on the receiving side, the super-resolution parameter calculated with reference to the input video whose resolution is not reduced is used, so that high-quality video transmission can be realized. That is, in the method of Patent Document 1, a high-resolution video is input to the transmission side, is transmitted with a low resolution, and a process for high resolution is performed on the reception side, thereby improving the image quality. It has gained.

Japanese Patent No. 5419795

  However, the technique described in Patent Document 1 has a problem that, when the super-resolution parameter is calculated on the transmission side, an input video requires a higher resolution video than the video to be transmitted.

  The present invention has been made in view of such circumstances, and a video encoding device and a program that do not require a higher-resolution video than a video to be transmitted when calculating a super-resolution parameter on the transmission side. I will provide a.

(1) The present invention has been made to solve the above-described problems, and one aspect of the present invention includes a first super-resolution unit that super-resolutions an input video by a first super-resolution method, The input video is super-resolved by a second super-resolution unit for super-resolution by a plurality of second super-resolution methods different from the first super-resolution method, and the first super-resolution unit. A selection unit that selects one of the plurality of second super-resolution methods by comparing the image and the images super-resolved by each of the plurality of second super-resolution methods; And an information output unit that outputs auxiliary information representing the second super-resolution method selected by the selection unit.

(2) According to another aspect of the present invention, there is provided the video encoding device according to (1), wherein the first super-resolution method is any of the plurality of second super-resolution methods. However, the calculation amount is large.

(3) According to another aspect of the present invention, there is provided the video encoding device according to (1), wherein the input video is irreversibly compressed by a video encoding unit that performs irreversible compression on the input video. A video decoding unit that decodes the compressed video, and the input video that the second super-resolution unit performs super-resolution using the plurality of second super-resolution methods is a video that is decoded by the video decoding unit. It is.

(4) According to another aspect of the present invention, there is provided the video encoding apparatus according to (1), wherein the information output unit compresses and encodes the auxiliary information.

(5) According to another aspect of the present invention, there is provided a computer, a first super-resolution unit that super-resolutions an input video by a first super-resolution method, and the input video that is the first super-resolution. A second super-resolution unit that performs super-resolution using a plurality of second super-resolution methods different from the imaging method, a video that is super-resolved by the first super-resolution unit, and the plurality of second super-resolutions. A selection unit that selects one of the plurality of second super-resolution methods by comparing the images super-resolved by each of the image methods, and the second super-resolution selected by the selection unit It is a program for making it function as an information output part which outputs the information which shows a method.

  According to the present invention, when calculating the super-resolution parameter, an image having a higher resolution than the image to be transmitted is not required.

It is a schematic block diagram which shows the structure of the video transmission system by one Embodiment of this invention.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a video transmission system according to an embodiment of the present invention. The video transmission system 100 includes a video encoding device 1, a video decoding device 2, a first transmission unit 9, and a second transmission unit 10. The video encoding device 1 and the video decoding device 2 are connected through first transmission means 9 and second transmission means 10.

  The video encoding device 1 generates encoded data by compressing and encoding the input low-resolution video L (input video). Furthermore, the video encoding device 1 generates auxiliary information that is a parameter related to super-resolution processing in the video decoding device 2. The first transmission unit 9 transmits the encoded data generated by the video encoding device 1 to the video decoding device 2. The second transmission means 10 transmits the auxiliary information generated by the video encoding device 1 to the video decoding device 2. The video decoding device 2 decodes the encoded data transmitted by the first transmission unit 9 and further super-resolutions the video obtained by this decoding using the auxiliary information transmitted by the second transmission unit 10. By doing so, the super-resolution video H is generated. The super-resolution video H is an output video of the video decoding device 2, and the low-resolution video L is an input image of the video encoding device 1. The super-resolution video H is a video with a higher resolution than the low-resolution video L.

  The video encoding device 1 includes a video encoding unit 3, a first super-resolution unit 4, a video decoding unit 5, a second super-resolution unit 6, a selection unit 7, and an auxiliary information encoding unit 8. The second super resolving means 6 includes candidate super resolving means 6-1 to 6-N. N is an integer of 2 or more. The video encoding means 3 (video encoding unit) compresses the input low resolution video L and converts it into a bit string. For the conversion to the bit string by the video encoding means 3, any method for encoding and compressing the video can be used. The encoding compression method may be lossless compression or lossy compression. For example, H.264, which is a known video encoding method. 261, MPEG-2, MPEG-4, MPEG-4 AVC / H. H.264, MPEG-H HEVC / H. H.265, Motion JPEG, Motion JPEG2000, VC-1, or the like can be used. In the case of lossless compression, the decoding result by the video decoding unit 5 is the same as that of the low resolution video L, so the video decoding unit 5 is unnecessary.

  The first super-resolution means 4 (first super-resolution unit) increases the resolution of each frame of the low-resolution video L by super-resolution of the low-resolution video L by the first super-resolution method. Let The super-resolution method (first super-resolution method) in the first super-resolution means 4 is the second super-resolution means 6 (second super-resolution part), that is, candidate super-resolution means 6-1 to 6-6. It is different from a plurality of super-resolution methods by -N (a plurality of second super-resolution methods). As the super-resolution method in the first super-resolution means 4, it is preferable to use a method having a higher calculation cost (computation amount) than any of the second super-resolution means 6. This is because a method having a higher calculation cost may obtain a video closer to the original image or a more beautiful video, that is, higher performance. For example, a single-frame super-resolution technique such as wavelet super-resolution is used for the candidate super-resolution means 6-1 to 6-N, whereas the first super-resolution means 4 has a reconfigurable multiple frame. Use super-resolution technology. Alternatively, as the first super-resolution means 4, a plurality of resolution conversion methods (for example, self-congruent super-resolution, multi-frame super-resolution, interpolation interpolation, etc.) are used. It is also possible to use a method of generating video to be output by switching in units of frames or partial areas according to whether it is a flat part, whether it is a moving area, whether it is a human face area, etc. It is.

  Specifically, the technique of Japanese Patent No. 5405389 is used for the first super-resolution means 4, and the technique of Japanese Patent Application Laid-Open No. 2014-119949 is used for the candidate super-resolution means 6-1 to 6-N. You may do it. Alternatively, the first super-resolution means 4 and the candidate super-resolution means 6-1 to 6-N are the same algorithm, but the number of repetitions of any one of the iterative processes in the super-resolution is the first super-resolution. The number of image means 4 may be increased.

The video decoding unit 5 (video decoding unit) obtains decoded video data by decoding the bit string encoded by the video encoding unit 3. The video decoding means 5 uses the same video decoding system as the video encoding means 3 and a pair.
If the video encoding means 3 includes a local decoding means (local decoder) and can output the result of local decoding (local decoding), the video decoding means 5 is omitted and the video decoding is performed. Instead of the output of the means 5 (decoded video data), the local decoding output (local decoded video data) of the video encoding means 3 may be used.

  Each of the candidate super-resolution means 6-1 to 6-N super-resolves the decoded video represented by the decoded video data decoded by the video decoding means 5, so that each frame (or video encoding means) of the decoded video is displayed. 3), the resolution of the locally decoded output video frame from 3 (hereinafter the same) is increased. Candidate super-resolution means 6-1 to 6-N use different super-resolution methods or use parameters having different values. Thereby, a different super-resolution result is obtained for each candidate super-resolution means 6-1 to 6-N for the same frame.

  In the present embodiment, the second super resolving unit 6 includes the candidate super resolving units 6-1 to 6-N to perform super resolving processing using a plurality of different super resolving methods. Alternatively, super-resolution processing using a plurality of parameters having different values can be performed. Not only this but one execution part which the 2nd super-resolution means 6 has switches the super-resolution system, or switches the parameter to be used, so that multiple super-resolution systems using different super-resolution systems can be used. Resolution processing may be performed, or a plurality of super-resolution processing using parameters having different values may be performed.

  The selection unit 7 (selection unit) receives images of super-resolution results from the first super-resolution unit 4 and the second super-resolution unit 6. The selection means 7 refers to the super-resolution result by the first super-resolution means 4 and selects at least one from the super-resolution results by the second super-resolution means 6. The selection means 7 uses the auxiliary information encoding means as information indicating which of the candidate super-resolution means 6-1 to 6-N is the selected super-resolution result as auxiliary information. 8 Note that the auxiliary information may be information indicating the super-resolution method used for generating the super-resolution result selected by the selection unit 7, or the parameter value used for generating the super-resolution result. It may be information indicating.

  The selection unit 7 may select the super-resolution result in units of video frames, may be performed in units of partial areas (for example, blocks) obtained by dividing the video frame, or an image in which a plurality of frames are combined. The processing may be performed in units of columns (for example, Group of Pictures (GOP) in the video encoding means 3).

As shown in FIG. 1, the second super resolving means 6 has a plurality of candidate super resolving means (in FIG. 1, candidate super resolving means 6-1 to candidate super resolving means 6-N). Then, the selection unit 7 determines which one of the super-resolution results of the candidate super-resolution unit 6-1 to the candidate super-resolution unit 6-N is closest to the output image of the first super-resolution unit 4. . For example, when the candidate super-resolution means 6-n (n is an integer of 1 or more and N or less) is closest, the selection means 7 outputs an identifier n indicating the candidate super-resolution means 6-n as auxiliary information. .
Note that the selection unit 7 may output information for enabling the video decoding apparatus 2 to execute the super-resolution processing for obtaining the selected super-resolution result as auxiliary information, and the selected super-resolution is selected. It may be information indicating a super-resolution method used for the result or a parameter.

  The selection means 7 uses, for example, an error, a structural similarity, an image feature amount, or a linear combination or non-linear combination thereof as a determination criterion when determining the closeness between images (or image partial areas). Can be used. As the error, for example, a square error sum or an absolute value error sum between the outputs of the candidate super-resolution means 6-1 to 6-N and the output of the first super-resolution means 4 is used. As the structural similarity, for example, a structural similarity index (Structural Similarity; SSIM) between the outputs of each of the candidate super-resolution means 6-1 to 6-N and the output of the first super-resolution means 4 is used. Use. Further, as the image feature amount, for example, the total variation (total variation) value of each output of the candidate super-resolution means 6-1 to 6-N is used.

  The auxiliary information encoding unit 8 (information output unit) encodes and compresses the determination result output from the selection unit 7 and outputs a bit string obtained by the encoding compression as auxiliary information. Although the compression method of the auxiliary information encoding means 8 is arbitrary, reversible compression is preferable. For example, Huffman coding or arithmetic coding may be applied to the determination result expressed as a bit string, or the Lempel-Ziv-Welch method may be applied.

  The first transmission unit 9 is a transmission path for transmitting the bit string output from the video encoding unit 3. The second transmission means 10 is a transmission path for transmitting the bit string output from the auxiliary information encoding means 8. The first transmission means 9 and the second transmission means 10 may be separate transmission paths or the same transmission path. Even if the first transmission unit 9 and the second transmission unit 10 share the same transmission path, the bit sequence output from the video encoding unit 3 and the bit sequence output from the auxiliary information encoding unit 8 are used. It may be transmitted in a separate stream, or these may be multiplexed and transmitted.

  As the first transmission means 9 and the second transmission means 10, for example, any of wired and wireless such as digital broadcasting and the Internet may be used, and protocols such as IP (Internet Protocol) and ISDB (Integrated Services Digital Broadcasting) Any system is acceptable. The first transmission unit 9 and the second transmission unit 10 may be transmission units using a portable medium such as Blu-ray (registered trademark).

  Even when the bit sequence output from the video encoding unit 3 and the bit sequence output from the auxiliary information encoding unit 8 are multiplexed, MPEG-2 TS (Transport Stream), MMT (MPEG Multimedia Transport), etc. The method is arbitrary. However, it is assumed that the timing of each bit string sent by the first transmission means 9 and the second transmission means 10 can ensure synchronization. The synchronization can be realized, for example, by adding time information to both bit strings sent by the first transmission means 9 and the second transmission means 10, and a hash value as disclosed in Japanese Patent Application Laid-Open No. 2014-132730. It can also be realized by the method used. Alternatively, the synchronization can be realized by using a multiplexing technique such as MMT.

  Next, the configuration of the video decoding device 2 will be described. The video decoding device 2 illustrated in FIG. 1 includes a video decoding unit 11, a third super-resolution unit 12, an auxiliary information decoding unit 13, and a switching unit 14. The third super resolving means includes candidate super resolving means 12-1 to 12-N. The video decoding unit 11 is a decoding unit that is paired with the video encoding unit 3, decodes the bit string output from the video encoding unit 3 and received through the first transmission unit 9, and outputs a video signal. The operation of the video decoding unit 11 is the same as that of the video decoding unit 5.

  Each of the candidate super-resolution means 12-1 to 12-N increases the resolution of each frame of the decoded video by super-resolution of the decoded video represented by the decoded video data decoded by the video decoding means 11. Candidate super-resolution means 12-1 to 12-N perform the same super-resolution processing as candidate super-resolution means 6-1 to 6-N, respectively. Therefore, different super-resolution results are obtained for each candidate super-resolution means 12-1 to 12-N for the same frame. Note that the third super resolving unit 12 also has the same function as the second super resolving unit 6 in which one execution unit of the third super resolving unit 12 switches the super resolving method or switches the parameters to be used. Thus, a plurality of super-resolution processes using different super-resolution methods may be performed, or a plurality of super-resolution processes using parameters having different values may be performed. The third super resolving means 12 preferably performs the same super resolving process as each of the candidate super resolving means 6-1 to 6-N, but it is sufficient that a similar result is obtained, and they are not exactly the same. May be.

  The auxiliary information decoding unit 13 performs a decoding process that makes a pair with the auxiliary information encoding unit 8. The auxiliary information decoding unit 13 decodes the bit string related to the auxiliary information generated by the auxiliary information encoding unit 8 and transmitted via the second transmission unit 10, and outputs the auxiliary information decoded from the bit string. For example, when the auxiliary information encoding unit 8 performs information compression by Huffman encoding, the auxiliary information decoding unit 13 performs a decoding process using the code tree (Huffman tree) used in the auxiliary information encoding unit 8. Do.

  The switching unit 14 switches the outputs of the candidate super-resolution units 12-1 to 12-N according to the auxiliary information decoded by the auxiliary information decoding unit 13 (determination result in the selection unit 7), and outputs super-resolution video for output. Configure H and output. The switching unit of the switching unit 14 is matched with the control state determination unit in the selection unit 7. For example, when the selection unit 7 determines the optimal control state in units of blocks, the switching unit 14 also performs switching in units of the same blocks.

  In the present embodiment, the switching unit 14 is connected to the subsequent stage of the candidate super resolving units 12-1 to 12-N, and the super resolving result of the candidate super resolving units 12-1 to 12-N is displayed. One of them is output, but it is not limited to this. For example, the switching unit 14 is connected to the preceding stage of the candidate super-resolution units 12-1 to 12-N, and the candidate super-resolution unit that inputs the decoding result of the video decoding unit 11 is switched according to the auxiliary information. May be. By doing so, it is not necessary to perform processing by a plurality of candidate super-resolution means for the same frame, so that the amount of calculation can be further reduced.

  In this embodiment, the low-resolution video L input to the video encoding device 1 is not compression-encoded, but may be compression-encoded. In that case, the video encoding device 1 may not have the video encoding unit 3, and the video decoding unit 5 decodes the input low resolution video L, and the decoding result of the video decoding unit 5 is In addition to the second super resolving means 6, the first super resolving means 4 may also be input. The first transmission unit 9 may transmit the low resolution video L input to the video encoding device 1.

As described above, the selection unit 7 compares the video super-resolved by the first super-resolution unit 4 with the video super-resolved by each of the plurality of second super-resolution methods, One of the second super-resolution methods is selected. Accordingly, even if a video having a higher resolution than the video to be transmitted is not input to the video encoding device 1, the receiver side (video decoding device 2 side) is super close to a higher-performance super-resolution technique. The resolution result can be obtained.
Further, the amount of calculation of each of the candidate super-resolution means 6-1 to 6-N is smaller than the amount of calculation by the first super-resolution means 4. As a result, a higher-performance super-resolution result can be obtained with a smaller amount of computation.

  1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, thereby executing the video encoding apparatus 1. The video decoding device 2 may be realized. Here, the “computer system” includes an OS and hardware such as peripheral devices.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

Each functional block of the video encoding device 1 and the video decoding device 2 in FIG. 1 described above may be individually chipped, or a part or all of them may be integrated into a chip. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Either hybrid or monolithic may be used. Some of the functions may be realized by hardware and some by software.
In addition, when a technology such as an integrated circuit that replaces an LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology can be used.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design changes and the like within a scope not departing from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 1 ... Video encoding apparatus, 2 ... Video decoding apparatus, 3 ... Video encoding means, 4 ... 1st super-resolution means, 5 ... Video decoding means, 6 ... 2nd super-resolution means, 7 ... Selection means, 8 ... Auxiliary information encoding means, 9 ... first transmission means, 10 ... second transmission means, 11 ... video decoding means, 12 ... third super-resolution means, 13 ... auxiliary information decoding means, 14 ... switching means

Claims (4)

A first super-resolution unit that super-resolutions the input video by the first super-resolution method;
A video encoding unit for irreversibly compressing the input video;
A video decoding unit for decoding the video that has been irreversibly compressed by the video encoding unit;
A second super-resolution unit that super-resolutions the video decoded by the video decoding unit using a plurality of second super-resolution methods different from the first super-resolution method;
The plurality of second super-resolution methods by comparing the image super-resolved by the first super-resolution unit with the images super-resolved by each of the plurality of second super-resolution methods. A selection unit for selecting one of them,
A video encoding device comprising: an information output unit that outputs auxiliary information representing the second super-resolution method selected by the selection unit.   The video encoding apparatus according to claim 1, wherein the first super-resolution method has a larger calculation amount than any of the plurality of second super-resolution methods.   The video encoding apparatus according to claim 1, wherein the information output unit compresses and outputs the auxiliary information. Computer
A first super-resolution unit that super-resolutions the input video by the first super-resolution method,
A video encoding unit for irreversibly compressing the input video;
A video decoding unit for decoding the video that has been irreversibly compressed by the video encoding unit;
A second super-resolution unit that super-resolutions the video decoded by the video decoding unit using a plurality of second super-resolution methods different from the first super-resolution method;
The plurality of second super-resolution methods by comparing the image super-resolved by the first super-resolution unit with the images super-resolved by each of the plurality of second super-resolution methods. A selection part for selecting one of them,
A program for functioning as an information output unit that outputs information indicating the second super-resolution method selected by the selection unit.

Images