JP2020150441A - Image padding method, image padding device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a rectangular image as a coding target by performing a padding process on an arbitrary shape image at a higher speed while suppressing an increase in a code amount.
An image padding device that converts an arbitrary shape image into a rectangular image by padding and outputs an image signal indicating the rectangular image is a shape that indicates the shape of the arbitrary shape image for each image block in which the arbitrary shape image is divided. It exists in the image block and the necessity determination unit 205 that determines whether or not padding needs to be performed based on the information and the coded target block information indicating the position and size of the image block to be encoded. An acquisition unit 207 that derives a padded image block based on a sparse matrix indicating the pixel position of the pixel, an in-region pixel information obtained by multiplying the image block by the sparse matrix, and an in-screen predicted value. If it is determined that it is not necessary to perform padding, it has an output unit 209 that outputs an image block, and if not, outputs a padded image block.
[Selection diagram] Fig. 5

Description

The present invention relates to an image padding method, an image padding device, and a program.

Conventionally, as a method of encoding an image of an arbitrary shape that is not a rectangle (hereinafter referred to as "arbitrary shape image"), an international video coding standard defined by ISO / IEC (International Organization for Standardization and International Electrotechnical Commission). There is a coding method using SA-DCT (Shape-Adaptive Discrete Cosine Transform) of MPEG-4 (ISO / IEC 14496-2). Further, for example, by extrapolating the pixels so as to surround the arbitrary shape image as shown in FIG. 7 (hereinafter, referred to as “padding”), the arbitrary shape image can be made into a rectangular image as shown in FIG. , "Rectangular image"), and there is a method of encoding using a general coding method capable of encoding a rectangular image. According to this method, a coding method such as a JPEG (ISO / IEC JTC 1 / SC 29/WG 1, Joint Photographic Experts Group) method, which cannot directly encode an arbitrary shape image, can be used.

As the simplest padding method, for example, when the image to be encoded is an 8-bit monochrome image, it is conceivable to perform padding with pixels having a predetermined pixel value (for example, 128). However, in this case, in the boundary region of the arbitrary shape image (for example, the region of the boundary between the region inside and the extrapolation region (padding target region) shown in FIG. 9), the pixel values of the pixels inside the region and the extrapolation region A large gap may occur between the pixel value and the pixel value. As a result, the coding efficiency (data compression rate) may decrease. In order to improve such a decrease in coding efficiency, a padding method called LPE (Low Pass Extrapolation) can be used in the above-mentioned MPEG-4. LPE is a method of padding an extrapolated region with a kind of LPF (Low Pass filter) using the pixel values of the pixels inside the region.

As another padding method, there are the methods described in Non-Patent Document 1 and Non-Patent Document 2. These padding methods are methods that are often used in lossy image coding to generate a padding signal such that the high frequency coefficient after the conversion process becomes 0.

Yoshimitsu Kuroki, Yoshifumi Kamishige, "New Padding Method in Arbitrary Shape DCT Coding", 2002 Image Coding Symposium (PCSJ2002), P-2.12, p. 31-32, November 11-13, 2002 Yoshimitsu Kuroki and 3 others, "Padding method in orthogonal conversion of H.264 / AVC", IPSJ Research Report, 2005-AVM-50 (4), p. 17-22, October 7, 2005 N.Parikh and S.Boyd, "Proximal Algorithms," [online], the essence of knowledge, Foundations and Trends in Optimization, Vol.1, No.3, 2013, p.123-231, 2013, [2019 Search on March 11], Internet <URL: http://web.stanford.edu/~boyd/papers/pdf/prox_algs.pdf> S.Boyd and L.Vandenberghe, "Convex Optimization," Cambridge University Press, 2004. [Searched March 11, 2019], Internet <URL: https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook .pdf ＞

In the padding method described in Non-Patent Document 1 and Non-Patent Document 2, the amount of calculation for padding is nC (n / 2) at the maximum when the number of pixels of the image block to be encoded is n. The amount of calculation is proportional to. For example, in the case of an image block of 8 pixels × 8 pixels used in JPEG and the like (i.e., n = 64), the amount of calculation for performing padding up to a 64C32 ≒ 1.83 × ^{10 18,} enormous It becomes the amount of calculation. Further, for example, H. In an image block of 32 pixels × 32 pixels (that is, n = 1024), which is a coding unit used in 265 / HEVC, the maximum amount of calculation is 4.48 × 10 ³⁰⁶ . Therefore, the image block to be encoded is, for example, H.I. Unless the size is about 4 pixels × 4 pixels (that is, n = 16), which is the smallest coding unit used in 264 / AVC, padding processing within a practical time is difficult (note that this). The maximum amount of calculation in this case is 12,870). As described above, the conventional padding method has a problem that when the number of pixels of the image block to be encoded increases, the coding efficiency decreases or the padding process within a practical time becomes difficult.

The present invention has been made in view of the above technical background, and provides a technique capable of performing padding processing on an arbitrary shape image at a higher speed while suppressing an increase in the code amount. The purpose.

One aspect of the present invention is an image padding method in which an arbitrary shape image is converted into a rectangular image by padding and an image signal indicating the rectangular image is output to an image coding device, wherein the arbitrary shape image is divided. For each block, padding is performed based on the shape information indicating the shape of the arbitrary shape image and the coded target block information indicating the position and size of the coded image block output from the image coding device. Within the region obtained by multiplying the image block by the padding necessity determination step for determining whether or not it is necessary, the sparse matrix indicating the pixel positions of the pixels existing in the image block, and the sparse matrix for the image block. A solution step for deriving a padded image block based on the pixel information and the in-screen predicted value output from the image encoding device, and when it is determined that the padding is not necessary, the image block is used. When it is determined that it is necessary to output the indicated image signal to the image coding device and perform the padding, an image signal output step of outputting the image signal indicating the padded image block to the image coding device and It is an image padding method having.

Further, one aspect of the present invention is an image padding device that converts an arbitrary shape image into a rectangular image by padding and outputs an image signal indicating the rectangular image to an image coding device, and the arbitrary shape image is divided. For each image block, the shape information indicating the shape of the arbitrary shape image and the coded target block information indicating the position and size of the coded image block output from the image coding device are used. It was obtained by multiplying the image block by a padding necessity determination unit for determining whether or not padding is necessary, a sparse matrix indicating the pixel positions of pixels existing in the image block, and a sparse matrix for the image block. A solution unit that derives a padded image block based on the in-region pixel information and the in-screen predicted value output from the image coding device, and an image signal output that outputs the image signal to the image coding device. When it is determined that the padding is not necessary, the image signal output unit outputs an image signal indicating the image block, and when it is determined that the padding needs to be performed. This is an image padding device that outputs an image signal indicating the padded image block.

Further, one aspect of the present invention is the image padding device, and when it is determined that the padding needs to be performed, all the pixels in the image block are padded based on the predicted value in the screen. The image signal output unit further includes an out-of-area determination unit for determining whether or not the pixel indicates a target area, and the image signal output unit determines that all the pixels in the image block are pixels indicating a padding target area. In this case, an image signal based on the predicted value in the screen is output.

Another embodiment of the present invention is an image padding apparatus described above, the solution finding unit, in minimizing the objective function and constraints represented by the following formula, already the padding to minimize the value of 1 T ^u The value of x indicating the image block is derived.

Here, u is an n-dimensional vector, n is the number of pixels of the image block, z is the predicted value in the screen, D is the sparse matrix, y is the pixel information in the region, and Φ is the predicted residual in the screen (x). transform coefficients -z), and 1 ^{T u} represents the sum of pixel values of all pixels in the image block.

Further, one aspect of the present invention is a program for operating a computer as the above-mentioned image padding device.

According to the present invention, it is possible to perform padding processing on an arbitrary shape image at a higher speed while suppressing an increase in the amount of code.

It is a block diagram which shows the functional structure of a general image coding apparatus 10. It is a schematic diagram which shows an example of the outline of the general in-screen prediction processing. It is a flowchart which shows the operation of the general image coding apparatus 10. It is an overall block diagram of the image coding system 1 which concerns on one Embodiment of this invention. It is a block diagram which shows the functional structure of the image padding apparatus 20 which concerns on one Embodiment of this invention. It is a flowchart which shows the operation of the image padding apparatus 20 which concerns on one Embodiment of this invention. It is a figure which shows an example of an arbitrary shape image. It is a figure which shows an example of the image which padded an arbitrary shape image. It is a figure for demonstrating the boundary area between the area inside and the extrapolation area.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

The image coding system 1 described below is a system that encodes in units of image blocks that are rectangular images. The image coding system 1 divides an arbitrary shape image to be coded into image blocks and performs padding for each image block. Then, the image coding system 1 encodes each padded image block. Any method can be used as the padding method.

Hereinafter, it is assumed that the number of pixels of the image block is n. Further, let x be a vector in which the pixel values of all the pixels in the padded image block are arranged in one dimension. Therefore, x is an n-dimensional vector. Further, it is assumed that the number of pixels in the region of the arbitrary shape image in the image block is m (that is, m <n). Further, let y be a vector in which the pixel values of all the pixels in the region are arranged in one dimension. Therefore, y is an m-dimensional vector.

Here, let the matrix of m × n be the matrix D, and consider the constraint that Dx = y. Further, it is assumed that the matrix D is a sparse matrix in which only one value of 1 exists in each row and all other values are 0. By multiplying x consisting of the pixel values of all the pixels in the image block by the matrix D, y consisting of all the pixel values inside the region of the arbitrary shape image is generated. That is, this restriction means that the pixel value inside the region does not change regardless of the pixel value of the extrapolated region (padding target region).

By the way, H. 264 / AVC and H. In 265 / HEVC, in order to improve the coding efficiency, a process called "in-screen prediction" that predicts the pixel values of the pixels in the image block from the pixel values of the decoded pixels around the image block to be encoded. Is done.

Hereinafter, let z be a vector in which the pixel values of all the pixels in the image block predicted on the screen are arranged in one dimension. Further, let Φ be the operator that performs the conversion process on the image block. The conversion process referred to here is a conversion process that is usually used in lossy image coding as described above. Then, the derivation of the padded image block (that is, the derivation of the value of x) is performed to obtain the value of x that minimizes the L0 norm under the constraint of Dx = y, as shown in the following equation (1). Formulate as an optimization problem.

The term || Φ (x-z) || ₀ , which is the objective function, represents the number of non-zero coefficients of the conversion coefficient of the predicted residual (x-z) in the screen. , It is supposed that the amount of code actually generated is accurately approximated. That is, by obtaining the value of x that satisfies the above equation (1), the padding process can be performed while suppressing the increase in the code amount.

However, since the L0 norm has a mathematical property of being non-convex, it is generally considered difficult to solve the minimization problem from the viewpoint of computational complexity. Therefore, it is conceivable to approximate with the L1 norm instead of the L0 norm and minimize the L1 norm. That is, as shown in the following equation (2), it is conceivable to formulate it as an optimization problem for finding the value of x that minimizes the L1 norm under the constraint of Dx = y.

As a method for solving the optimization problem formulated by the above equation (2), generally, for example, the Proximal Gradient Method described in Non-Patent Document 3 and the ADMM Method (Alternating Direction Method of) There are methods such as Multipliers Method). Although it is possible to find the solution of the optimization problem shown in Eq. (2) by such a method, in the present embodiment, the minimization objective function and the constraint are set so that the solution can be found at a higher speed. , Each is formulated as the following equation (3).

Here, u is an n-dimensional vector. Further, as shown in Eq. (3), the constraint that the absolute value of each element of Φ (x−z) (that is, an n-dimensional vector in which the conversion coefficients of the predicted residuals of the image block are arranged) is u or less. And there is a constraint that the element of u is non-zero. Here, 1 T ^u represents the sum of all the elements of u, equation (3) means determining the value of x that minimizes the u. This optimization problem belongs to a problem called a linear programming problem, and many solutions are known. For example, this optimization problem can be efficiently solved by the simplex method described in Non-Patent Document 4.

Hereinafter, the configuration of the image coding system 1 according to the present embodiment will be described, but for the sake of clarity, the functional configuration of a general image coding device will be described first.

[Functional configuration of image encoding device]
FIG. 1 is a block diagram showing a functional configuration of a general image coding device 10. The image to be encoded by the image coding system 1 according to the present embodiment is not a moving image but a still image. Therefore, in FIG. 1, the description of the functional unit and the like related to inter-screen prediction and distortion removal, which are not used in the coding of still images, is omitted.

As shown in FIG. 1, the image coding device 10 includes an image signal input unit 101, a subtraction unit 102, a conversion unit 103, a quantization unit 104, an entropy coding unit 105, and a coded data output unit 106. The inverse quantization unit 107, the inverse conversion unit 108, the addition unit 109, the frame memory 110, and the in-screen prediction unit 111 are included.

The image signal input unit 101 receives the input of the image signal to be encoded. The image signal input unit 101 divides the input image signal and outputs the image signal to the subtraction unit 102 in units of predetermined image blocks. The predetermined image block unit is, for example, H. In the case of 265 / HEVC, it is a processing unit called a prediction unit.

The subtraction unit 102 acquires the image signal output from the image signal input unit 101. Further, the subtraction unit 102 acquires a separately generated prediction signal output from the in-screen prediction unit 111. The prediction signal is an image signal indicating a predicted later image block. The subtraction unit 102 obtains a prediction residual signal by subtracting the prediction signal from the acquired image signal. The subtraction unit 102 outputs the predicted residual signal to the conversion unit 103.

The conversion unit 103 acquires the predicted residual signal output from the subtraction unit 102. The conversion unit 103 performs a discrete cosine transform (DCT) process on the acquired predicted residual signal. The conversion unit 103 outputs the discrete cosine-converted predicted residual signal to the quantization unit 104.

The quantization unit 104 acquires the predicted residual signal output from the conversion unit 103. The quantization unit 104 quantizes the acquired predicted residual signal. The quantization unit 104 outputs the quantized predicted residual signal to the entropy coding unit 105 and the inverse quantization unit 107, respectively.

The entropy coding unit 105 acquires the predicted residual signal output from the quantization unit 104. The entropy coding unit 105 obtains coded data by performing entropy coding on the acquired predicted residual signal. The entropy coding unit 105 outputs the coded data to the coded data output unit 106.

The coded data output unit 106 acquires the coded data output from the entropy coding unit 105. The coded data output unit 106 outputs the acquired coded data.

The inverse quantization unit 107 acquires the predicted residual signal output from the quantization unit 104. The inverse quantization unit 107 performs inverse quantization on the acquired predicted residual signal. The inverse quantization unit 107 outputs the inverse quantization predicted residual signal to the inverse conversion unit 108.

The inverse conversion unit 108 acquires the predicted residual signal output from the inverse quantization unit 107. The inverse transform unit 108 performs an inverse discrete cosine transform (inverse DCT) process on the acquired predicted residual signal. The inverse transform unit 108 outputs the predicted residual signal subjected to the inverse discrete cosine transform to the adder unit 109.

The addition unit 109 acquires the predicted residual signal output from the inverse conversion unit 108. Further, the addition unit 109 acquires a separately generated prediction signal output from the in-screen prediction unit 111. The adding unit 109 obtains an image signal (hereinafter, referred to as “decoded image signal”) indicating a decoded image by adding a predicted signal to the acquired predicted residual signal. The addition unit 109 stores the decoded image signal in the frame memory 110. The decoded image signal stored in the frame memory 110 is the same signal as the decoded image signal restored by the decoding device corresponding to the image coding device 10.

The frame memory 110 stores the decoded image signal output from the addition unit 109. The frame memory 110 is, for example, a storage medium such as a RAM (Random Access Memory; read / write memory), a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), and an HDD (Hard Disk Drive), or a storage medium thereof. Consists of any combination.

The in-screen prediction unit 111 performs in-screen prediction processing using the decoded image signal stored in the frame memory 110 to generate a prediction signal. As described above, the prediction signal is an image signal indicating a predicted later image block. The in-screen prediction unit 111 outputs the generated prediction signal to the subtraction unit 102 and the addition unit 109, respectively.

A frame in which motion compensation prediction is not performed and all prediction units in the frame are predicted and encoded in the screen is called an I frame (Intra-coded Frame).

[In-screen prediction processing]
Hereinafter, an example of the in-screen prediction process performed by the in-screen prediction unit 111 will be described. FIG. 2 is a schematic diagram showing an outline of a general in-screen prediction process. In FIG. 2, the shaded circles are pixels that have already been decoded in the screen (hereinafter, referred to as “reference pixels”). Further, the unshaded circles are pixels to be predicted and encoded in the screen (hereinafter, referred to as “predicted pixels”), and are collectively present for each image block. In the case of the example shown in FIG. 2, the predicted pixels are collectively present for each image block of 4 pixels × 4 pixels (that is, n = 16).

The in-screen prediction process is a process of associating some value with the predicted pixel by using the information of the reference pixel. For example, in the case of the example shown in FIG. 2, the pixel value of the reference pixel in the upper left direction is associated with (copied) the predicted pixel in the lower right direction. The copying method shown in FIG. 2 is an example, and other methods may be used. For example, H. In 265 / HEVC, 33 methods of copying are provided, including the method of copying from the lower left direction to the upper right direction as described above. In addition, H. In 265 / HEVC, a method of using the average value of the pixel values of the reference pixels as the pixel value of the predicted pixel can also be used.

[Operation of image encoding device]
Hereinafter, an example of the operation of the image coding device 10 will be described. FIG. 3 is a flowchart showing the operation of a general image coding device 10. This flowchart starts when the image signal input unit 101 receives the input of the image signal to be encoded and outputs the input image signal to the subtraction unit 102 in units of predetermined image blocks.

The subtraction unit 102 acquires the image signal output from the image signal input unit 101. Further, the subtraction unit 102 acquires the prediction signal output from the in-screen prediction unit 111. The subtraction unit 102 generates a prediction residual signal by subtracting the prediction signal from the acquired image signal (step S001). The subtraction unit 102 outputs the predicted residual signal to the conversion unit 103.

The conversion unit 103 performs a discrete cosine transform (DCT) on the acquired predicted residual signal. The conversion unit 103 outputs the predicted residual signal subjected to the discrete cosine transform to the quantization unit 104. The quantization unit 104 performs quantization on the acquired predicted residual signal (step S002). The quantization unit 104 outputs the quantized predicted residual signal to the entropy coding unit 105 and the inverse quantization unit 107, respectively.

The entropy coding unit 105 obtains coded data by performing entropy coding on the acquired predicted residual signal (step S003). The entropy coding unit 105 outputs the coded data to the coded data output unit 106. The coded data output unit 106 outputs the acquired coded data.

The dequantization unit 107 dequantizes the acquired predicted residual signal. The inverse quantization unit 107 outputs the inverse quantization predicted residual signal to the inverse conversion unit 108. The inverse transform unit 108 performs inverse discrete cosine transform (inverse DCT) on the acquired predicted residual signal. The inverse transform unit 108 outputs the predicted residual signal subjected to the inverse discrete cosine transform to the adder unit 109 (step S004).

The addition unit 109 generates a decoded image signal by adding a prediction signal to the acquired prediction residual signal (step S005). The addition unit 109 stores the decoded image signal in the frame memory 110 (step S006).

The in-screen prediction unit 111 performs in-screen prediction processing using the decoded image signal stored in the frame memory 110, and generates a prediction signal indicating a later image block (step S007). The in-screen prediction unit 111 outputs the generated prediction signal to the subtraction unit 102 and the addition unit 109, respectively. This completes the operation of the image coding device 10 shown in the flowchart of FIG.

[Overall configuration of image coding system]
Hereinafter, the overall configuration of the image coding system 1 will be described. FIG. 4 is an overall configuration diagram of the image coding system 1 according to the embodiment of the present invention. As shown in FIG. 4, the image coding system 1 includes an image coding device 10 and an image padding device 20. The functional configuration of the image coding device 10 shown in FIG. 4 is the same as the functional configuration of the general image coding device 10 described above.

First, the image padding device 20 acquires an image signal indicating an arbitrary shape image to be encoded from an external device. Further, the image padding device 20 acquires information indicating the shape of an arbitrary shape image to be encoded (hereinafter, referred to as “shape information”) from an external device. Further, the image padding device 20 acquires the in-screen prediction value indicating the result of the in-screen prediction processing and the coding target block information from the image coding device 10. The coded target block information is information including information indicating the position and size of the coded image block.

The image padding device 20 determines whether or not the image block to be encoded is an image block that needs to be padded, based on the acquired information. When it is determined that it is not necessary to perform padding, the image padding device 20 outputs the image signal indicating the image block as it is to the image coding device 10. On the other hand, when it is determined that it is not necessary to perform padding, the image padding device 20 performs padding on the image block to be encoded, and sends an image signal indicating the padded image block to the image coding device 10. Output.

[Functional configuration of image padding device]
Hereinafter, the functional configuration of the image padding device 20 will be described. FIG. 5 is a block diagram showing a functional configuration of the image padding device 20 according to the embodiment of the present invention. As shown in FIG. 5, the image padding device 20 includes an image signal input unit 201, a shape information input unit 202, a coded block information input unit 203, an in-screen predicted value input unit 204, and a padding necessity determination. A unit 205, an out-of-area determination unit 206, an in-area pixel acquisition unit 207, a linear problem solving unit 208, and an image signal output unit 209 are included.

The image signal input unit 201 receives an input of an image signal indicating an arbitrary shape image to be encoded. The image signal input unit 201 divides the input image signal and outputs it to the padding necessity determination unit 205 in units of predetermined image blocks.

The shape information input unit 202 accepts input of shape information. The shape information input unit 202 outputs the input shape information to the padding necessity determination unit 205.

The coding target block information input unit 203 acquires the coding target block information from the image coding device 10. The coded target block information input unit 203 outputs the acquired coded target block information to the padding necessity determination unit 205.

The in-screen predicted value input unit 204 acquires the in-screen predicted value from the image coding device 10. The in-screen predicted value input unit 204 outputs the acquired in-screen predicted value to the out-of-area determination unit 206.

The padding necessity determination unit 205 acquires the image signal output from the image signal input unit 201. Further, the padding necessity determination unit 205 acquires the shape information output from the shape information input unit 202. Further, the padding necessity determination unit 205 acquires the coding target block information output from the coding target block information input unit 203. The padding necessity determination unit 205 determines whether or not it is necessary to perform padding on the image signal based on the shape information and the coding target block information. Specifically, the padding necessity determination unit 205 determines whether or not all the pixels in the image block based on the image signal are pixels indicating the inside of the region of the arbitrary shape image (that is, whether or not m = n). ) Is determined to determine whether or not it is necessary to perform padding on the image signal.

When it is determined that it is not necessary to perform padding (that is, when m = n and all the pixels in the image block exist inside the region), the padding necessity determination unit 205 determines the acquired image signal. Is output to the image signal output unit 209. On the other hand, when it is determined that padding is necessary (that is, when m <n and at least one of the pixels in the image block is a pixel indicating an extrapolation region), the padding necessity determination unit 205 , The acquired image signal is output to the extrapolation determination unit 206.

The out-of-region determination unit 206 acquires the image signal output from the padding necessity determination unit 205. Further, the out-of-area determination unit 206 acquires the in-screen prediction value (that is, the above-mentioned value of z; hereinafter referred to as “in-screen prediction value z”) output from the in-screen prediction value input unit 204. Based on the acquired in-screen predicted value z, the out-of-region determination unit 206 determines whether or not all the pixels in the image block based on the image signal are pixels indicating the extrapolation region (that is, m = 0). Whether or not) is determined.

When it is determined that all the pixels in the image block are pixels indicating the extrapolation region (that is, when m = 0), the out-of-region determination unit 206 determines the in-screen predicted value z as the image signal output unit 209. Output to. In this case, the predicted residuals (x-z) in the screen are all 0. On the other hand, when it is determined that at least one of the pixels in the image block is a pixel indicating the inside of the region of the arbitrary shape image (that is, when m> 0), the out-of-region determination unit 206 uses the image signal. And the in-screen predicted value z is output to the in-region pixel acquisition unit 207.

The in-region pixel acquisition unit 207 acquires the image signal output from the out-of-region determination unit 206 and the in-screen predicted value z. Further, the in-region pixel acquisition unit 207 generates the above-mentioned sparse matrix D indicating the pixel positions of the pixels existing in the image block. The intra-regional pixel acquisition unit 207 obtains intra-regional pixel information (that is, the value of y described above, hereinafter referred to as “intra-regional pixel y”) by multiplying the acquired image signal by a sparse matrix D. The in-region pixel acquisition unit 207 outputs the sparse matrix D, the in-region pixels y), and the in-screen predicted value z to the linear problem solving unit 208.

The linear problem solving unit 208 acquires the sparse matrix D, the area pixel y, and the in-screen predicted value z output from the area pixel acquisition unit 207. The linear problem solving unit 208 derives a value of x that satisfies the above equation (3) based on the acquired sparse matrix D, the pixel y in the region, and the predicted value z in the screen. For example, the linear problem solving unit 208 derives the value of x by the Simplex method or the like. The linear problem solving unit 208 outputs the value of x to the image signal output unit 209.

When the image signal output unit 209 acquires the image signal output from the padding necessity determination unit 205, the image signal output unit 209 outputs the image signal to the image coding device 10. Further, when the image signal output unit 209 acquires the in-screen predicted value z output from the out-of-area determination unit 206, the image signal output unit 209 outputs the in-screen predicted value z to the image coding device 10. Further, when the image signal output unit 209 acquires the value of x output from the linear problem solving unit 208, the image signal output unit 209 outputs the value of x to the image coding device 10.

[Operation of image padding device]
Hereinafter, an example of the operation of the image padding device 20 will be described. FIG. 6 is a flowchart showing the operation of the image padding device 20 according to the embodiment of the present invention. This flowchart starts when an image signal is input to the image padding device 20.

The image signal input unit 201 receives the input of the image signal to be encoded (step S101). The image signal input unit 201 outputs the input image signal to the padding necessity determination unit 205 in units of predetermined image blocks.

The padding necessity determination unit 205 is based on the shape information acquired from the shape information input unit 202 and the coded object block information acquired from the image coding device 10 via the coded object block information input unit 203. It is determined whether or not it is necessary to perform padding on the image signal (step S102).

When it is determined that it is not necessary to perform padding (step S102 / No), the padding necessity determination unit 205 outputs the acquired image signal to the image coding device 10 via the image signal output unit 209 (step S103). ).

On the other hand, when it is determined that padding is necessary (step S102 · Yes), the padding necessity determination unit 205 outputs the acquired image signal to the out-of-region determination unit 206. The out-of-area determination unit 206 acquires the in-screen predicted value z output from the in-screen predicted value input unit 204 (step S104). The out-of-region determination unit 206 determines whether or not all the pixels in the image block based on the image signal are pixels indicating the extrapolation region based on the acquired in-screen prediction value z (step S105).

When all the pixels in the image block are determined to be pixels indicating the extrapolation region (step S105 · Yes), the out-of-region determination unit 206 determines the in-screen predicted value z via the image signal output unit 209. Is output to the image coding apparatus 10 (step S106).

On the other hand, when it is determined that the pixels exist inside the region of at least one arbitrary shape image among the pixels in the image block (step S105 / No), the out-of-region determination unit 206 determines the image signal and the in-screen predicted value. z is output to the in-region pixel acquisition unit 207. Next, the in-region pixel acquisition unit 207 generates a sparse matrix D indicating the pixel positions of the pixels existing in the image block (step S107). The in-region pixel acquisition unit 207 obtains the in-region pixel y by multiplying the acquired image signal by a sparse matrix D. The in-region pixel acquisition unit 207 outputs the sparse matrix D, the in-region pixel y, and the in-screen predicted value z to the linear problem solving unit 208.

The linear problem solving unit 208 solves the linear programming problem by deriving the value of x that satisfies the above equation (3) based on the acquired sparse matrix D, the pixels y in the region, and the predicted value z in the screen. Obtain (step S108). The linear problem solving unit 208 outputs the value of x to the image coding device 10 via the image signal output unit 209 (step S109). This completes the operation of the image padding device 20 shown in the flowchart of FIG.

As described above, the image padding device 20 according to the embodiment of the present invention described above converts an arbitrary shape image into a rectangular image by padding, and outputs an image signal indicating the rectangular image to the image coding device 10. .. The image padding device 20 includes a padding necessity determination unit 205, a linear problem solving unit 208 (solving unit), and an image signal output unit 209. The padding necessity determination unit 205 includes shape information indicating the shape of the arbitrary shape image and the image block to be encoded output from the image coding device 10 for each image block in which the arbitrary shape image is divided. It is determined whether or not padding needs to be performed based on the coded block information indicating the position and size of the image. The linear problem solving unit 208 includes a sparse matrix D indicating the pixel positions of pixels existing in the image block, an in-region pixel y obtained by multiplying the image block by a sparse matrix D, and the image coding. Based on the in-screen predicted value z output from the device 10, the value of x indicating the padded image block is derived. The image signal output unit 209 outputs an image signal indicating the image block when it is determined that the padding does not need to be performed, and when it is determined that the padding needs to be performed, the value of x is calculated. Output.

By having the above-described configuration, the image padding device 20 according to the embodiment of the present invention can efficiently perform padding processing on an arbitrary shape image while suppressing the amount of generated code.

As described above, the conventional padding method has a problem that the coding efficiency (data compression rate) is not good or it takes a lot of time for image coding. On the other hand, the image padding device 20 according to the embodiment of the present invention reduces the padding process to a linear programming problem by providing, for example, an objective function and constraints as shown in the above equation (3). As a result, the image padding device 20 can generate a padded image at a higher speed while suppressing a decrease in coding efficiency.

Further, since the image padding device 20 according to the embodiment of the present invention rectangularizes the arbitrary shape image by padding the arbitrary shape image, it is general that only the rectangular image is to be encoded. It is possible to encode an arbitrary shape image by using a simple image coding method. Further, the image padding device 20 can suppress an increase in the code amount, can obtain a decoded image of higher quality with a smaller code amount, and compress the image more efficiently than the conventional coding method. Can be done.

As described above, the image padding device 20 according to the embodiment of the present invention can be widely applied to the case of coding and decoding an arbitrary shape image for the purpose of increasing the data compression rate in image coding. is there.

In the above-described embodiment, the image coding device 10 and the image padding device 20 are separate devices, but the present invention is not limited to this. One information processing device (for example, a general-purpose computer or the like) may be configured to include the functional unit of the image coding device 10 and the functional unit of the image padding device described above.

A part or all of the image coding system 1 in the above-described embodiment may be realized by a computer. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system. It may be realized by using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

Although the embodiments of the present invention have been described above with reference to the drawings, it is clear that the above embodiments are merely examples of the present invention, and the present invention is not limited to the above embodiments. Therefore, components may be added, omitted, replaced, and other changes may be made without departing from the technical idea and gist of the present invention.

1 ... Image coding system, 10 ... Image coding device, 20 ... Image padding device, 101 ... Image signal input unit, 102 ... Subtraction unit, 103 ... Conversion unit, 104 ... Quantization unit, 105 ... Entropy coding unit, 106 ... Encoded data output unit, 107 ... Inverse quantization unit, 108 ... Inverse conversion unit, 109 ... Addition unit, 110 ... Frame memory, 111 ... In-screen prediction unit, 201 ... Image signal input unit, 202 ... Shape information input Unit, 203 ... Coded block information input unit, 204 ... In-screen prediction value input unit, 205 ... Padding necessity determination unit, 206 ... Out-of-area determination unit, 207 ... In-area pixel acquisition unit, 208 ... Linear problem solving unit , 209 ... Image signal output unit

Claims (5)

An image padding method that converts an arbitrary shape image into a rectangular image by padding and outputs an image signal indicating the rectangular image to an image encoding device.
For each image block in which the arbitrary shape image is divided, shape information indicating the shape of the arbitrary shape image and a coding target indicating the position and size of the image block to be encoded output from the image coding device. A padding necessity determination step for determining whether or not padding needs to be performed based on the block information, and
A sparse matrix indicating the pixel positions of pixels existing in the image block, intra-regional pixel information obtained by multiplying the image block by a sparse matrix, and an in-screen predicted value output from the image encoding device. And the solution step to derive the padded image block based on
When it is determined that the padding is not necessary, an image signal indicating the image block is output to the image encoding device, and when it is determined that the padding is necessary, the padded image block is indicated. An image signal output step for outputting an image signal to the image encoding device, and
Image padding method with. An image padding device that converts an arbitrary shape image into a rectangular image by padding and outputs an image signal indicating the rectangular image to an image coding device.
For each image block in which the arbitrary shape image is divided, shape information indicating the shape of the arbitrary shape image and a coding target indicating the position and size of the image block to be encoded output from the image coding device. A padding necessity determination unit that determines whether or not padding needs to be performed based on the block information, and
A sparse matrix indicating the pixel positions of pixels existing in the image block, intra-regional pixel information obtained by multiplying the image block by a sparse matrix, and an in-screen predicted value output from the image encoding device. And the solution part that derives the padded image block based on
An image signal output unit that outputs the image signal to the image encoding device,
With
The image signal output unit
When it is determined that the padding does not need to be performed, an image signal indicating the image block is output.
An image padding device that outputs an image signal indicating the padded image block when it is determined that the padding needs to be performed. When it is determined that it is necessary to perform the padding, an out-of-area determination unit that determines whether or not all the pixels in the image block are pixels indicating a padding target region based on the in-screen prediction value.
With more
The image signal output unit
The image padding device according to claim 2, wherein when it is determined that all the pixels in the image block are pixels indicating a padding target area, an image signal based on the predicted value in the screen is output. The solution unit
In minimizing the objective function and constraints represented by the following equation, the image padding device according to claim 2 or claim 3 derives a value of x indicating the padded image block that minimizes the value of 1 T ^u ..
Here, u is an n-dimensional vector, n is the number of pixels of the image block, z is the predicted value in the screen, D is the sparse matrix, y is the pixel information in the region, and Φ is the predicted residual in the screen (x). transform coefficients -z), and 1 ^{T u} represents the sum of pixel values of all pixels in the image block. A program for operating a computer as the image padding device according to any one of claims 2 to 4.