JP7549463B2 - Encoding device and method, program, and storage medium - Google Patents

Description

The present invention relates to a technology for encoding RAW images of moving images.

In conventional encoding devices, raw image information (RAW images) captured by an imaging sensor is de-Bayered (demosaiced) and converted into signals consisting of luminance and color difference, and each signal is then subjected to so-called development processing, such as noise removal, optical distortion correction, and image optimization. The developed luminance and color difference signals are then typically compression-encoded and recorded on a recording medium.

On the other hand, there are also imaging devices that can record RAW images. Although the amount of data required to record RAW images is enormous, they have the advantage that correction and degradation to the original image is kept to a minimum, and they can be edited after shooting, so they are preferred by advanced users.

However, because RAW images contain a huge amount of data, it is desirable to reduce the amount of data so that as many images as possible can be recorded on a limited recording medium. For this reason, the amount of data is reduced by compressing the RAW images. However, depending on the shooting conditions, compression can lead to degradation of image quality. Patent Document 1 discloses a configuration for encoding using adaptive quantization, which changes the quantization coefficient according to visual characteristics.

JP 2017-126859 A

However, in the above conventional configuration, the quantization coefficient for dark areas is reduced and a large amount of code is allocated so that information in dark areas is not lost. However, at low bit rates, the quantization coefficient is generally larger than at high bit rates in order to suppress the amount of code generated. In order to maintain the specified bit rate, there are some images for which a quantization coefficient large enough to prevent the loss of information in dark areas cannot be allocated. In such cases, the quantization coefficient becomes large, information in dark areas is lost, and noise may occur. In particular, vertical and horizontal streaky noise occurs. As a result, there is a problem of degradation in image quality.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an encoding device that can suppress image quality degradation when encoding RAW images at a low bit rate.

The encoding device of the present invention comprises a conversion means for frequency-converting data based on raw data of an image, a quantization means for quantizing the conversion coefficients of multiple frequency components output by the conversion means based on a quantization coefficient, and a control means for determining, for each block, the quantization coefficients of the multiple frequency components contained in the block, wherein, when a predetermined condition is satisfied that the brightness of the block is a dark area lower than a predetermined threshold value, the control means determines the quantization coefficient of the block so that the quantization coefficient of the low frequency components is smaller and the quantization coefficient of the high frequency components is larger than when the predetermined condition is not satisfied.

The present invention makes it possible to suppress degradation of image quality when encoding RAW images at a low bit rate.

1 is a block diagram showing the configuration of an encoding device according to a first embodiment of the present invention. FIG. FIG. 2 is a block diagram showing the configuration of a RAW compression encoding unit. A diagram of subband formation at decomposition level 2 of the discrete wavelet transform (DWT). 11 is a flowchart showing an initial setting process of a quantization control unit. 13 is a diagram showing an image in which only one decoded DWT coefficient is other than 0 in the Lv1 subband. FIG. 13 is a conceptual diagram of code amount allocation in a dark area. FIG. 11 is a block diagram showing the configuration of a RAW compression encoding unit according to the second embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

(First embodiment)
FIG. 1 is a block diagram showing the configuration of an encoding device 100 according to a first embodiment of the present invention.

In FIG. 1, when an instruction to start a shooting operation is given, an optical image of a subject to be imaged is input via an imaging optical system 101 and formed on an imaging sensor unit 102. The imaging sensor unit 102 converts light that has passed through red, green, and blue (RGB) color filters arranged for each pixel into an electrical signal.

Figure 2 shows an example of color filters arranged in the image sensor unit 102, and shows the pixel arrangement of the image handled by the encoding device 100. As shown in Figure 2, red (R), green (G), and blue (B) color filters are arranged in a mosaic pattern for each pixel. The pixels are arranged regularly so that for every 2 x 2 set of four pixels, there is one red pixel, one blue pixel, and two green pixels. This type of pixel arrangement is generally called a Bayer array.

The electrical signal photoelectrically converted by the imaging sensor unit 102 is subjected to pixel restoration processing by the sensor signal processing unit 103. The restoration processing includes processing for missing pixels or low reliability pixels in the imaging sensor unit 102, such as interpolating using surrounding pixel values or subtracting a predetermined offset value. In this embodiment, the image information output from the sensor signal processing unit 103 is called a RAW image, which means a raw (undeveloped) image.

The RAW image is input to the RAW compression encoding unit 104. The RAW compression encoding unit 104 generates subband data, which is a frequency band (frequency component), from the RAW image data output by the sensor signal processing unit 103 using a wavelet transform (frequency transform), quantizes the output subband data, and encodes it on a subband basis. The encoded data is stored in the buffer 105. The processing of the RAW compression encoding unit 104 will be explained in detail later.

The encoded data in subband units is stored in a buffer 105, then input to a recording data control unit 106 and output to an output terminal 107 for recording on a recording medium, an external storage device, etc.

The processing of the RAW compression encoding unit 104 will be described in detail below. FIG. 3 is a block diagram showing the configuration of the RAW compression encoding unit 104. The processing of the RAW compression encoding unit 104 will be described using FIG. 3.

The Bayer RAW data input from the input terminal 301 is input to the plane division unit 302. The plane division unit 302 divides the Bayer RAW data into color components. In this case, the green (G) component is twice as large as the red (R) and blue (B) components. For this reason, the green adjacent to red (R) is designated as G1, and the green adjacent to blue (B) is designated as G2, and the data is divided into four planes and treated as data with the same number of pixels.

Each plane data split by the plane splitting unit 302 is input to the discrete wavelet transform unit 303. The discrete wavelet transform unit 303 generates transform coefficients (DWT coefficients, wavelet coefficients) by converting the input plane data into frequency domain signals.

Figure 4 shows the subband formation diagram for decomposition level 2, where plain data is decomposed twice by applying vertical and horizontal filtering of discrete wavelet transform (DWT) to each frequency band. As shown in Figure 4, in this embodiment, the decomposition level of DWT is set to 2. However, the present invention is not limited to decomposition level 2.

In DWT, the frequency band is decomposed into multiple bands by applying filters vertically and horizontally. Then, by recursively applying DWT to the low-frequency subband generated by the above transformation, the decomposition level is increased, and the granularity of the frequency decomposition can be made finer as shown in Figure 4. In the description of Figure 4, Lv indicates the decomposition level including the number after it, and "L" and "H" indicate low frequency (low frequency components) and high frequency (high frequency components), respectively. The order is as follows: the front side indicates the band resulting from horizontal filtering, and the rear side indicates the band resulting from vertical filtering. Also, LL indicates the lowest frequency subband. In this way, discrete wavelet transform processing is performed to obtain data on a subband basis.

The above subband-based data is input to the quantization unit 304. Meanwhile, the quantization coefficient determination unit 307 determines the quantization ratio between the subbands, calculates the Q value, which is a quantization coefficient, from this ratio, and inputs it to the quantization unit 304. The quantization unit 304 quantizes the subband-based data for each wavelet coefficient using the Q value, which is the quantization coefficient input from the quantization coefficient determination unit 307. Note that the larger the Q value is, the more the amount of code is reduced, but the more noticeable the degradation in image quality becomes. The operation of the quantization coefficient determination unit 307 will be described in detail later.

The image data quantized by the quantization unit 304 is entropy coded by the entropy coding unit 305 and output from the output terminal 306. The output of the entropy coding unit 305 is input to the code amount calculation unit 309, which calculates the code amount for each subband. The calculated code amount data is input to the quantization control unit 308.

The quantization control unit 308 performs quantization control in picture units and rectangular blocks, which are quantization units. In picture units, a quantization coefficient (initQ value) that is optimal for the next picture is generated from the picture target code amount. In rectangular block units, a ratio calculation is performed from the Q value of the basic subband to calculate the Q value of each subband corresponding to the rectangular block. In this embodiment, the raw data is divided into plain data, and then the quantization coefficients of the transform coefficients of each subband generated by wavelet transform are determined in rectangular block units obtained by dividing the raw data (picture) by a predetermined rectangular size, and quantization is performed for each subband using the determined quantization coefficients. Therefore, one rectangular block includes transform coefficients of multiple subbands, and the number of transform coefficients (rectangle size for each subband) of the subbands corresponding to one rectangular block differs for each decomposition level.

In this embodiment, the quantization coefficient determination unit 307 and the quantization control unit 308 are configured separately, but the quantization coefficient determination unit 307 may not be provided separately and the quantization control unit 308 may perform the functions of the quantization coefficient determination unit 307.

In addition, for each quantization unit, the amount of encoded code is subtracted from the picture target code amount to calculate the remaining target code amount, and the number of remaining rectangular blocks is calculated from the total number of rectangular blocks and the number of encoded rectangular blocks. The remaining target code amount is divided by the number of remaining rectangular blocks to calculate the remaining rectangular block target code amount, which is compared with the rectangular block target code amount calculated by dividing the target code amount before picture encoding by the number of rectangular blocks, and a quantization coefficient (varQ value) that is optimal for maintaining the target code amount is generated. However, in the first quantization of a picture, the varQ value = initQ value. In this embodiment, the Q value of Lv1HL is set to the varQ value and is output from the quantization control unit 308.

Here, the quantization ratio determination process of the quantization coefficient determination unit 307 will be described in detail. FIG. 5 is a flowchart showing the quantization ratio determination process of the quantization coefficient determination unit 307.

In FIG. 5, in step S501, the process of the quantization coefficient determination unit 307 is started, and in step S502, it is determined whether the set bit rate is a low bit rate or not. If it is determined that the set bit rate is not a low bit rate, in step S506, all quantization processes for that picture are set to setting 1, and in step S511, the process ends.

On the other hand, if it is determined that the set bit rate is a low bit rate, in step S503, the address (Badr) of the unit rectangular block to be controlled is set to the initial value of 0.

Next, in step S504, the DC evaluation value (DCV) of the rectangular block in Badr is obtained. In this embodiment, the average value of R, G1, G2, and B at the same position in each plane is used as the DC evaluation value.

In step S505, it is determined whether the DCV is smaller than an arbitrarily set threshold (DC_Thr). If the DCV is equal to or greater than DC_Thr, in step S507, the quantization setting of Badr is set to setting 1. If the DCV is smaller than DC_Thr, in step S508, the quantization setting of Badr is set to setting 2.

After steps S507 and S508, in step S509, it is determined whether Badr is the last block of the picture. If Badr is not the last block of the picture, in step S510, 1 is added to Badr, and the process returns to step S504. If Badr is the last block of the picture, the process ends in step S511.

In this embodiment, the average value of R, G1, G2, and B at the same position in each plane is used as the DC evaluation value, but this is not limited to this value.

Next, we will explain the above settings 1 and 2. In these settings, the relationship between the Q values of each subband is determined.

The Q values of each subband of Lv1 (decomposition level 1: low-order decomposition level) in the DWT of Figure 4 are QLv1LH, QLv1HL, and QLv1HH, and the Q values of each subband of Lv2 (decomposition level 2: high-order decomposition level) are QLv2LL, QLv2LH, QLv2HL, and QLv2HH, and will be used for the explanation.

As a premise, the minimum value of each Q value is 1.

Min(QLv2LH) = 1
Min(QLv2HL)=1
Min(QLv2HH)=1 ... Equation (1)
Min(QLv1LH) = 1
Min(QLv1HL)=1
Min(QLv1HH)=1
First, a description will be given of setting 1. Normally, the ratio of the Q value according to the decomposition level is expressed by the following formula.

QLv2LH:QLv1LH=1:2
QLv2HL:QLv1HL=1:2 … Equation (2)
QLv2HH:QLv1HH=1:2
however,
QLv2LH=QLv2HL=QLv2HH/2... Formula (3)
In setting 1, the parameters shown in the above formulas (2) and (3) are used.

In setting 2, parameters to be used for dark areas at low bit rates are set. At low bit rates, the quantization coefficient is large to reduce the amount of generated code. Also, in dark areas, the image signal is small, so the DWT coefficient is small. If the DWT coefficient is small and the quantization coefficient is large, there is a high possibility that the decoded DWT coefficient obtained by inverse quantization after quantization will be 0.

For example, if the decoded DWT coefficient of only HL of Lv1 is n (n ≠ 0) in a location where the decoded DWT coefficients of HL/LH/HH of Lv1 are all 0, vertical lines will appear in the image. Even if the vertical lines are not very strong in the original image, after quantization they will only appear as vertical lines, making them emphasized and noticeable.

Figure 6 shows an image in which only one decoded DWT coefficient in the Lv1 subband is non-zero. The short lines in the figure look like degradation of image quality.

To prevent this degradation in image quality, the quantization coefficient is increased and the decoded DWT coefficient of Lv1 is set to 0. However, if the decoded DWT coefficient of Lv2 also becomes 0, the result will be a state of only DC components, which will result in poor resolution of dark areas and degradation in image quality. Therefore, it is necessary to ensure that the coefficient of Lv2 does not become 0.

Figure 7 shows the concept of code amount allocation in dark areas. As shown in Figure 7, the code amount is allocated so that the quantization coefficient of Lv2 is reduced by increasing the quantization coefficient of Lv1 at the same position so that the coefficient of Lv2 does not become 0. By allocating in this way, it is possible to prevent a large impact on other positions.

From the above, the ratio of Q values according to the decomposition level of setting 2 is
QLv2LH:QLv1LH=1:8
QLv2HL:QLv1HL=1:8 ... Equation (4)
QLv2HH:QLv1HH=1:8
However, the denominator can be changed arbitrarily, and a ratio that makes the amount of generated code at the same position equal may be obtained by feedback or the like.

In setting 2, the parameters shown in the above formula (4) and formula (3) are used. Since the above is the ratio of the quantization coefficients, the amount of code between decomposition levels at the same position is not necessarily distributed exactly one-to-one, but degradation of image quality can be suppressed.

In this way, the quantization ratio of each subband is determined. The Q value of the quantization coefficient of each subband is determined based on the optimal Lv1HL quantization coefficient (varQ value) output from the quantization control unit 308.

In this embodiment, not only the quantization ratio but also the rounding process can be switched between settings 1 and 2. In addition, in this embodiment, the ratio of the quantization coefficients of each subband is used as the quantization ratio, but the quantization coefficient of each subband may be calculated using the code amount ratio in each subband.

When quantization is performed, rounding is performed. In rounding, the value changes depending on whether the fraction is rounded up or down when dividing by the quantization coefficient. Rounding reduces quantization error. For this reason, setting 1 performs rounding or a similar rounding process.

However, when the wavelet coefficient becomes larger than the input value, more high frequency components are generated, resulting in some parts of the input image being emphasized. The phenomenon shown in Figure 6 can occur even in such cases. For this reason, in setting 2, truncation or a similar rounding process is performed to suppress deterioration of image quality in the high frequency bands of dark areas. This concludes the explanation of settings 1 and 2.

In this way, by suppressing noise that occurs in dark areas at low bit rates, it is possible to record RAW images without degrading image quality.

Second Embodiment
In the second embodiment, the block configuration of the encoding device is similar to that of the first embodiment shown in Fig. 1. However, in this embodiment, the processing of the RAW compression encoding unit 104 in Fig. 1 is different from that of the first embodiment, and this part will be described in detail.

Figure 8 is a block diagram showing the configuration of the RAW compression encoding unit 104 in this embodiment. The same functional parts as those in Figure 3 used in the first embodiment are given the same reference numerals. The processing of the RAW compression encoding unit 104 in this embodiment will be described using Figure 8. Since the overall processing is similar to that in the first embodiment, the plane conversion unit 810 and the quantization coefficient determination unit 307, which are parts where the processing is different, will be described below.

The plane conversion unit 810 performs color conversion processing on the plane data of each color component output from the plane division unit 302, and separates it into multiple planes consisting of a luminance plane (luminance component) and a chrominance plane (chrominance component). In this embodiment, four planes consisting of luminance Y and chrominances C1, C2, and C3 are generated. For example, the conversion formula is as follows:

Y=(R+B+G1+G2)/4... Formula (5)
C1=RG1... Formula (6)
C2=B-G2... Formula (7)
C3=(R+G1)/2-(B+G2)/2... Formula (8)
Each plane data that has been subjected to the above conversion is input to a discrete wavelet transform unit 303 .

The quantization coefficient determination unit 307 performs the same processing as described in the first embodiment using FIG. 5, but the settings are different, so this will be explained.

We will now explain settings 1 and 2 in this embodiment. In these settings, the relationship between the Q values of each subband is determined, but the settings are different for the luminance plane and the chrominance plane.

The Q values of each subband of Lv1 in the DWT of FIG. 4 are QLv1LH, QLv1HL, and QLv1HH, and the Q values of each subband of Lv2 are QLv2LL, QLv2LH, QLv2HL, and QLv2HH, and will be used in the following explanation. In addition, when the Q values of the luminance plane Y and the chrominance planes C1, C2, and C3 are different, the respective Q values will be explained as YQ, C1Q, C2Q, and C3Q. The premise is the same as in the first embodiment, and the minimum value of each Q value is 1 (Equation (1)).

First, setting 1 will be described. Normally, the ratio of Q values according to the decomposition level is as shown in formulas (2) and (3). However, the allocation by plane is different,
YQLv1LH: C1QLv1LH: C2QLv1LH: C3QLv1LH
= αY: αC1: αC2: αC3... Formula (9)
It becomes.
At this time,
αY≦αC1=αC2
αY ≦ αC3…Formula (10)
It becomes.

Each parameter αY, αC1, αC2, and αC3 is given arbitrarily, and αY is set to be equal to or less than the quantization coefficients of the other planes. In setting 1, the parameters shown in the above formulas (9), (10), and (3) are used.

In setting 2, parameters to be used for dark areas at low bit rates are set as in the first embodiment. As in the first embodiment, it is necessary that the coefficient of Lv2 is not 0. The ratio of Q values according to the decomposition levels in setting 2 is as shown in equation (4). In setting 2, the parameters shown in equations (4), (9), (10), and (3) above are used. By increasing the quantization coefficient of Lv1, the amount of generated code can be reduced, and the quantization coefficient of Lv2 can be reduced accordingly.

Moreover, in the parameter of the formula (10), when αY is made small,
YQLv2LH:YQLv1LH=1:8
YQLv2HL:YQLv1HL=1:8... Formula (11)
YQLv2HH:YQLv1HH=1:8
and the chrominance plane may be expressed as equation (2).

In this way, the quantization ratio of each subband is determined. The Q value of the quantization coefficient of each subband is determined based on the optimal Lv1HL quantization coefficient (varQ value) output from the quantization control unit 308. The rounding process is the same as in the first embodiment. This concludes the explanation of settings 1 and 2.

In this way, even for images that have undergone plane conversion, noise that occurs in dark areas at low bit rates can be suppressed, making it possible to record RAW images without degrading image quality.

In addition, in the present embodiment, the plane conversion unit 810 has been described using equations (5) to (8), but the YCbCr format such as BT.601, BT.709, or BT.2020 or other color space conversion formats may be used.
In that case,
C1=Cb
C2=Cr
C3=(G1-G2)/2
It is also possible to use the following.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

101: Imaging optical system, 102: Imaging sensor unit, 103: Sensor signal processing unit, 104: RAW compression encoding unit, 105: Buffer, 106: Recording data control unit, 107: Output terminal, 301: Input terminal, 302: Plane division unit, 303: Discrete wavelet transform unit, 304: Quantization unit, 305: Entropy encoding unit, 306: Output terminal, 307: Quantization coefficient determination unit, 308: Quantization control unit, 309: Code amount calculation unit, 810: Plane conversion unit

Claims (14)

A conversion means for converting frequency of data based on raw data of an image;
a quantization means for quantizing the transform coefficients of the plurality of frequency components output by the transform means based on a quantization coefficient;
A control means for determining, for each block, a quantization coefficient of a plurality of frequency components included in the block;
the control means, when a predetermined condition that the brightness of the block is a dark area lower than a predetermined threshold is satisfied, determines the quantization coefficient of the block such that the quantization coefficient of the low frequency component is smaller and the quantization coefficient of the high frequency component is larger than when the predetermined condition is not satisfied.
13. An encoding device comprising: The encoding device according to claim 1, characterized in that the control means determines the quantization coefficients so that, in the case of a low bit rate, when the brightness of a block satisfies the predetermined condition, the quantization coefficient of the low-frequency components of the block is smaller and the quantization coefficient of the high-frequency components of the block is larger than when the brightness of the block does not satisfy the predetermined condition. The control means
determining a quantization coefficient for each of the plurality of frequency components based on a quantization ratio corresponding to the plurality of frequency components;
The encoding device according to claim 1 or 2, characterized in that, when the brightness of a block satisfies the specified condition, the quantization ratio of the block is changed so that the quantization coefficient of the low-frequency components of the block is smaller and the quantization coefficient of the high-frequency components of the block is larger than when the specified condition is not satisfied. The transforming means performs a wavelet transform on data based on raw data of an image to generate transform coefficients of a plurality of subbands;
The encoding device according to any one of claims 1 to 3, characterized in that, when the brightness of a block satisfies the specified condition, the control means determines the quantization coefficients so that the quantization coefficients of the subbands of high-frequency components of a higher-order decomposition level of the block are smaller and the quantization coefficients of the subbands of high-frequency components of a lower-order decomposition level of the block are larger when the brightness of the block satisfies the specified condition, compared to when the specified condition is not satisfied. The encoding device according to claim 4, characterized in that the control means determines the quantization coefficients so that, when the brightness of a block satisfies the predetermined condition, the quantization coefficients of the subbands of the high frequency components of the block at decomposition level 2 are smaller and the quantization coefficients of the subbands of the high frequency components of the block at decomposition level 1 are larger than when the brightness of the block does not satisfy the predetermined condition. The transforming means performs a wavelet transform on data based on raw data of an image to generate transform coefficients of a plurality of subbands;
The control means determines quantization coefficients for a plurality of subbands included in a block based on quantization ratios of the plurality of subbands;
the control means determines a quantization coefficient of each subband of the block using a first quantization ratio when the brightness of the block satisfies the predetermined condition, and determines a quantization coefficient of each subband of the block using a second quantization ratio when the brightness of the block does not satisfy the predetermined condition;
3. The encoding device according to claim 1, wherein the first quantization ratio is a ratio in which the quantization coefficients of high frequency components at decomposition level 2 are smaller than the second quantization ratio, and the quantization coefficients of high frequency components at decomposition level 1 are larger than the second quantization ratio. The encoding device according to any one of claims 1 to 6, characterized in that the data based on the RAW data is RAW data. The encoding device according to any one of claims 1 to 6, characterized in that the data based on the RAW data is data obtained by converting the RAW data into a luminance component and a color difference component. 9. The encoding device according to claim 1, wherein the quantization means performs rounding in the quantization of the dark areas by truncation or a similar rounding process in a high frequency band. 10. The encoding device according to claim 1, wherein whether or not the brightness satisfies the predetermined condition is determined using an average value of each color component. 10. The encoding device according to claim 1, wherein whether or not the brightness satisfies the predetermined condition is determined using a luminance component. A conversion step of frequency converting data based on raw data of an image;
a quantization step of quantizing the transformation coefficients of the plurality of frequency components output in the transformation step based on a quantization coefficient;
A control step of determining, for each block, a quantization coefficient of a plurality of frequency components included in the block;
In the control step, when a predetermined condition that the brightness of the block is a dark area lower than a predetermined threshold is satisfied, the quantization coefficient of the block is determined so that the quantization coefficient of the low frequency component is smaller and the quantization coefficient of the high frequency component is larger than when the predetermined condition is not satisfied.
13. An encoding method comprising: A program for causing a computer to function as each of the means of the encoding device according to any one of claims 1 to 11 . A computer-readable storage medium storing a program for causing a computer to function as each of the means of the encoding device according to any one of claims 1 to 11 .

Images