KR102195669B1 - Apparatus for transmitting image - Google Patents

Abstract

An image transmission apparatus according to an embodiment includes a down-sampling unit for down-sampling an original high-resolution image and converting it into a low-resolution image, and a first coding technique that can be decoded by each of a small user and a large user. Based on the first coding unit for coding the low-resolution image, and when receiving the low-resolution image converted by the down-sampling unit, a pre-learned super-resolution unit to output a processed high-resolution image of lower quality than the original high-resolution image And, a residual map providing unit that calculates a difference between the original high-resolution image and the processed high-resolution image to provide a residual map in the form of a residual map, and a second coding technique that can be decoded by the high-end terminal, the residual It includes a second coding unit for coding the map.

Description

Image sending device {APPARATUS FOR TRANSMITTING IMAGE}

The present invention relates to an image transmission apparatus.

In general data communication, distortion is not allowed when performing compression, whereas multimedia signals such as video and audio allow a certain amount of distortion to further increase compression efficiency.

In this case, there is a demand for a technique of allowing a certain amount of distortion and compressing accordingly, taking into account not only the case where the terminal at the receiving end is a small user but also a large user.

Republic of Korea Patent Publication 10-2017-0047489, Publication date May 08, 2017

The problem to be solved of the present invention is to provide a technology related to effectively transmitting images to terminals having different number of antennas and having different screen resolutions.

However, the problem to be solved of the present invention is not limited to the ones mentioned above, and another problem to be solved that is not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

An image transmission apparatus according to an embodiment includes a down-sampling unit for down-sampling an original high-resolution image and converting it into a low-resolution image, and a first coding technique that can be decoded by each of a small user and a high-end terminal. Based on the first coding unit for coding the low-resolution image, and when receiving the low-resolution image converted by the down-sampling unit, a pre-learned super-resolution unit to output a processed high-resolution image of lower quality than the original high-resolution image And, a residual map providing unit that calculates a difference between the original high-resolution image and the processed high-resolution image to provide a residual map in the form of a residual map, and a second coding technique that can be decoded by the high-end terminal, the residual It includes a second coding unit for coding the map.

In addition, the first coding technique may be an Alamouti technique.

In addition, the second coding technique may be a golden coding technique.

In addition, the super-resolution unit may include a plurality of residual block units configured to include a convolution layer and a ReLU layer but not include a batch normalization layer.

In addition, the residual map forming unit may provide the residual map based on a convex optimization technique.

According to an embodiment, optimal performance can be exhibited from an end-to-end point of view, and in particular, terminals having different number of antennas and having different screen resolutions, such as low-end and high-end terminals, in the same area Even if it is inside, the image can be effectively transmitted to each of these terminals.

1 is a block diagram schematically illustrating a configuration of an image transmission apparatus according to an embodiment.
2 is a block diagram schematically showing the configuration of a high-end terminal (big user) according to an embodiment.
3 is a block diagram schematically showing the configuration of a low-end terminal (small user) according to an embodiment.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, and only these embodiments make the disclosure of the present invention complete, and are common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have, and the invention is only defined by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout this specification.

1 is a block diagram schematically illustrating a configuration of an image transmission apparatus 100 according to an exemplary embodiment.

Referring to FIG. 1, the image transmission apparatus 100 includes a down-sampling unit 110, an image source coding unit 120, a first preprocessing unit 130, a first coding unit 140, and a first mux unit 150. ), a super-resolution unit 160, a residual map providing unit 170, a second preprocessing unit 180, a second coding unit 190, and a second mux unit 191. However, the configuration of the image transmission apparatus 100 shown in FIG. 1 is only exemplary. For example, according to an embodiment, the image transmission apparatus 100 may be implemented or configured to be different from that illustrated in FIG. 1.

First, the image transmission device 100, and the components included in each of these image transmission devices 100, are a memory that stores instructions implemented to perform functions to be described below, and a microcomputer that executes instructions stored in these memories. It can be implemented by a processor.

First, the image transmission device 100 receives an original high resolution (HR) image 10. Then, the original high-resolution image 10 received in this way is converted into a low resolution (LR) image 11 and then coded based on the Alamouti technique or the like. Here, a coding technique for a low-resolution image is not limited thereto.

In addition, the above-described original high-resolution image 10 received by the image transmission device 100 is converted into a processed high-resolution image of low quality, and thereafter, indicating the difference between the processed high-resolution image and the original high-resolution image 10. A residual map is created. The generated residual map is coded based on a golden code technique or the like. Here, the coding technique for the residual map is not limited thereto.

Then, the low-resolution image 11 coded based on the Alamouti technique described above and the residual map coded based on the golden code technique are transmitted to an image receiving device not shown in FIG. 1, for example, a low-end terminal or a high-end terminal. .

Here, a low-end terminal (small user) refers to a terminal having one antenna and a screen supporting low resolution. In contrast, a high-end terminal (big user) refers to a terminal having a plurality of (for example, two) antennas and a screen that supports high resolution.

In a high-end terminal, an image as close as possible to the original high-resolution image, that is, a reconstructed high-resolution image, is generated using the received low-resolution image 11 and the residual map. Here, in the process of generating the reconstructed high-resolution image, the Alamouti technique or the golden code technique described above may be used.

On the other hand, in the low-end terminal, the image is restored and generated using only the low-resolution image 11 thus transmitted. In this restoration process, the above-described Alamouti technique or the like may be used.

Hereinafter, the above-described image transmission apparatus 100 itself will be described in more detail.

The down sampling unit 110 is configured to down-sample the original high-resolution image 10. That is, the down-sampling unit 110 converts the original high-resolution image 10 into a low-resolution image 11.

Here, although not shown in FIG. 1, the down-sampling unit 110 may be implemented to additionally include an anti-aliasing low-pass filter. In this case, the down-sampling unit 110 includes an original high-resolution image 10 ), as well as anti-aliasing low-pass filtering function may be performed.

The image source coding unit 120 is configured to perform source coding on the low-resolution image 11. Here, since the source coding work on the image is a known technique, a detailed description thereof will be omitted. In addition, the image source coding unit 120 may not be included in the image transmission apparatus 100 according to embodiments. In this case, the low-resolution image 11 from the down-sampling unit 110 may be transferred to the super-resolution unit 160 and the first preprocessor 130 to be described later. However, hereinafter, the description will be made on the premise that the image source coding unit 120 is included in the image transmission apparatus 100.

The first preprocessor 130 is configured to receive the low-resolution image 11 on which the source coding has been performed from the image source coding unit 120 and perform a predetermined preprocessing. The preprocessing here may include, for example, error detection and correction coding, and may also include constellation symbol mapping. However, the first preprocessor 130 may not be included in the image transmission apparatus 100 according to an exemplary embodiment. In this case, the low-resolution image 11 from the image source coding unit 120 may be transmitted to the first coding unit 140 to be described later. However, hereinafter, the description will be made on the premise that the first preprocessor 130 is included in the image transmission apparatus 100.

The first coding unit 140 is configured to receive the low-resolution image 11 on which the above-described pre-processing has been performed from the first pre-processing unit 130 and code the low-resolution image 11 based on the first coding technique. Here, the first coding technique may be, for example, an Alamouti technique, but is not limited thereto.

Meanwhile, the super-resolution unit 160 may be configured to receive the low-resolution image 11 on which the source coding has been performed from the image source coding unit 120, and to increase super-resolution, that is, the resolution. . The result of the super-resolution process will be referred to as a'processed high-resolution image' hereinafter.

Here, the super-resolution unit 160 may be pre-trained to output a processed high-resolution image when a low-resolution image is input. Hereinafter, the process of learning the super-resolution unit 160 will be described in more detail.

First, learning data for learning the super-resolution unit 160 is provided. The training data may be, for example, a well-known DIV2K dataset. The DIV2K dataset contains a number of original high-resolution images.

Then, although not shown in FIG. 1, each of a plurality of original high-resolution images included in the DIV2K dataset is down-sampled by a separate down-sampling module. In this case, according to an embodiment, each of the plurality of original high-resolution images may pass through an anti-aliasing low-pass filter. Thereafter, each of the plurality of original high-resolution images down-sampled in this way may be compressed by a separate compression module according to various different compression rates. For example, the compression rate at this time may be 0.05, 0.1, 0.15, ... 2.0 bpp, but is not limited thereto. As a result, a plurality of low-resolution images compressed at different compression rates may be provided.

Thereafter, by a separate learning module, a plurality of original high-resolution images prepared as described above are used as correct answers, and a plurality of low-resolution images compressed with different compression rates are input as inputs, so that the super-resolution unit 160 can be learned. have. Here,'learning' may be machine learning, such as deep learning, but is not limited thereto.

In this case, the low-resolution images used for learning may be compressed at different compression rates as described above. For example, when there are 100 original high-resolution images, 100 low-resolution images compressed with 2.0 bpp, 100 low-resolution images compressed with 1.95 bpp, ... 100 low-resolution images compressed with 0.05 bpp can be prepared. have.

In addition, in the learning process, learning can be performed on the super-resolution unit 160 by 300 epochs with a low-resolution image compressed to 2.0 bpp and an original high-resolution image corresponding thereto, and when the learning is finished, it is compressed to another bpp of another value. Learning may be performed on the super-resolution unit 160 by 150 epochs with the low-resolution image and the high-resolution image corresponding thereto. Here, the compression ratio for the low-resolution image used when learning is performed and the value of the epoch mentioned in each are only examples.

Meanwhile, the above-described super-resolution unit 160 may include a plurality of residual block units. In this case, each of the plurality of residual block units may be configured to include a convolution layer and a ReLU layer, but may be configured not to include a batch normalization layer. Since each of the plurality of residual block units is configured in this way, the memory footprint may be smaller and the performance may be further improved.

The residual map providing unit 170 is configured to provide a residual map. The residual map refers to a map in the form of a matrix representing the difference between the above-described processed high-resolution image and the above-described original high-resolution image 10. Such a residual map may be provided based on a convex optimization technique, but is not limited thereto. Here, since the convex optimization itself is a known technique, a description thereof will be omitted. In one embodiment, limitations on the residual map may be generated so that a reconstructed high-resolution image closer to the original high-resolution image can be generated by the residual map. (constraint) is introduced.

To examine these limitations, first, the original high-resolution image is called U ₀ , the processed high-resolution image is called U ₁ , and the residual map is called R. Each of these U ₀ , U ₁ and R is called a matrix of N _HR x N _HR . Let's assume. Here, N _HR x N _HR may mean the resolution for each of U ₀ , U ₁ and R. In this case, if the total number of elements included in R is N _HR ² , the maximum number of elements having a non-zero value among these elements may be L (L is a natural number less than N _HR ² ). In this case, the value of L may have a value based on a compression rate or a bit budget.

The second preprocessor 180 is configured to receive the residual map from the residual map providing unit 170 and perform a predetermined preprocessing. The preprocessing here may include, for example, error detection and correction coding, and may also include constellation symbol mapping. However, the second preprocessor 180 may not be included in the image transmission apparatus 100 according to an exemplary embodiment. However, hereinafter, the description will be made on the premise that the first preprocessor 130 is included in the image transmission apparatus 100.

The second coding unit 190 is configured to receive the residual map on which the above-described pre-processing has been performed from the second pre-processing unit 180 and code the residual map based on the second coding technique. Here, the second coding technique may be, for example, a golden coding technique, but is not limited thereto.

The first mux unit 150 receives the low-resolution image 11 coded based on the first coding technique from the first coding unit 140, and also receives the low-resolution image 11 from the second coding unit 190 based on the second coding technique. After receiving the coded residual map, it is configured to perform a multiplexing operation on the received ones. The result of the multiplexing operation may be transmitted to the image receiving apparatus through the antenna shown in FIG. 1.

Similarly, the second mux unit 191 receives the low-resolution image 11 coded based on the first coding technique from the first coding unit 140, and also receives the low-resolution image 11 from the second coding unit 190 to the second coding technique. It is configured to perform a multiplexing operation on the received residual maps after receiving the coded residual map. The result of the multiplexing operation may be transmitted to the image receiving apparatus through the antenna shown in FIG. 1.

That is, according to an embodiment, in the image transmission apparatus 100, a low-resolution image is generated based on the original high-resolution image, and a processed high-resolution image of low quality is generated based on the generated low-resolution image, and a processed high-resolution image of low quality and A residual map representing the difference between the above-described original high-resolution images is generated. In addition, the generated residual map and the above-described low-resolution image are each coded based on a predetermined coding technique and transmitted to the image receiving apparatus.

Hereinafter, such an image receiving device will be described. As described above, the image receiving apparatus may include a high-end terminal and a low-spec terminal. Hereinafter, a high-end terminal will be first described.

2 is a diagram schematically showing a configuration of a high-end terminal 200 according to an embodiment.

Referring to FIG. 2, the high-end terminal 200 includes a first demux unit 210, a first decoding unit 220, a first post-processing unit 230, an image source decoding unit 240, and a super-resolution unit 250. ), a second demux unit 211, a second decoding unit 260, and a second post-processing unit 270. However, the configuration of the high-end terminal 200 shown in FIG. 2 is only exemplary. For example, according to an embodiment, the high-end terminal 200 may be implemented or configured to be different from that shown in FIG. 2.

First, the high-end terminal 200, and the configurations included in each of these high-end terminals 200, are stored in a memory that stores instructions implemented to perform functions to be described below, and a microprocessor that executes instructions stored in these memories. Can be implemented by

First, the high-end terminal 200 receives the above-described two pieces of data from the image transmission apparatus 100 through the two antennas shown in FIG. 2. One is the residual map and the other is a low-resolution image. Here, as described above, the residual map is coded by the first coding technique, and the low-resolution image is coded by the second coding technique.

The first demux unit 210 and the second demux unit 211 provide a residual map coded by a first coding technique to the first decoding unit 220 and the second decoding unit 260, respectively, and a second coding. It provides a low-resolution image coded by the technique.

Then, the first decoding unit 220 performs decoding based on the received items. During decoding, an Alamouti decoding technique may be employed, but is not limited thereto.

The result decoded by the first decoding unit 220 is transmitted to the first post-processing unit 230. The first post-processing unit 230 may be configured to perform constellation symbol mapping, and may also be configured to detect and correct errors.

The result processed by the first post-processing unit 230 is transmitted to the image source decoding unit 240. The image source decoding unit 240 may be configured to output a low-resolution image by performing, for example, source decoding on the result transmitted from the first post-processing unit 230.

Meanwhile, the second decoding unit 260 decodes the items transmitted from the first demux unit 210 and the second demux unit 211. During decoding, a golden coding technique may be employed, but is not limited thereto.

The result decoded by the second decoding unit 260 is transmitted to the second post-processing unit 270. The second post-processing unit 270 may be configured to perform constellation symbol mapping, and may also be configured to detect and correct errors.

The super-resolution unit 250 generates a reconstructed high-resolution image 21 based on a low-resolution image 20 and a residual map that is a result of the second post-processing unit 270 as a result from the image source decoding unit 240. . Here, the process of generating the reconstructed high-resolution image by the super-resolution unit 250 is shown in Equation 1 below.

[Equation 1]

U ₂ = U ₁ +GR

Here, U2 denotes a reconstructed high-resolution image generated by the super-resolution unit 250, U ₁ denotes a processed high-resolution image processed from a low-resolution image, R denotes a residual map, and G denotes a predefined two-dimensional convolutional kernel. . Here, G may be a Gaussian convolution filter as shown in Equation 2 below, but is not limited thereto.

Hereinafter, with reference to FIGS. 1 and 2, after the original high-resolution image is input to the image transmission device 100, a process of outputting the restored high-resolution image from the high-end terminal 200 will be described.

The original high-resolution image 10 is down-sampled by the down sampling unit 110. In this case, as described above, the original high-resolution image 10 may pass through the anti-aliasing low-pass filter according to the embodiment.

The down-sampling unit 110 outputs a low-resolution image 11 in response to the original high-resolution image 10 being input. Then, the image source coding unit 120 performs a predetermined source coding on the result output from the down-sampling unit 110.

Then, the first pre-processing unit 130 performs a predetermined pre-processing on the result output by the image source coding unit 120, and the first coding unit 140 generates the result output by the first pre-processing unit 130. 1 Coding based on coding technique.

Meanwhile, the result output by the image source coding unit 120 is also transmitted to the super-resolution unit 160. Then, the super-resolution unit 160 outputs a low-quality high-resolution image in response thereto. Then, the residual map providing unit 170 provides a residual map representing this difference in matrix form based on the difference between the original high-resolution image 10 and the super-resolution image 10 and the result of the super-resolution image.

The second preprocessor 180 performs a predetermined preprocessing on the residual map, and the second coding unit 190 preprocesses the result of the second preprocessor 180 based on a second coding technique.

Then, the first mux unit 150 receives the result of the first coding unit 140 and the result of the second coding unit 190 and performs a multiplexing operation, and the second mux unit 191 is also a first coding unit. A multiplexing operation is performed by receiving the result of 140 and the result of the second coding unit 190. Each of the multiplexed results is transmitted to the high-end terminal 200 through the antenna shown in FIG. 1.

Next, we will look at the low-end terminal. 3 is a diagram schematically showing the configuration of a low-end terminal 300 according to an embodiment.

Referring to FIG. 3, the low-end terminal 300 includes a first demux unit 310, a first decoding unit 320, a first post-processing unit 330, and an image source decoding unit 340. However, the configuration of the low-end terminal 300 shown in FIG. 3 is only exemplary. For example, according to an embodiment, the low-end terminal 300 may be implemented or configured to be different from that shown in FIG. 3.

First, the low-end terminal 300, and the configurations included in each of these low-end terminals 300, are stored in a memory that stores instructions implemented to perform functions to be described below, and a microprocessor that executes instructions stored in these memories. Can be implemented by

First, the low-end terminal 300 receives a low-resolution image from the image transmission device 100 through one antenna shown in FIG. 3. Here, the low-resolution image is in a state of being coded by the second coding technique described above.

The first demux unit 310 provides the first decoding unit 320 with a coded low-resolution image coded by a second coding technique.

Then, the first decoding unit 320 performs decoding based on the received items. During decoding, an Alamouti decoding technique may be employed, but is not limited thereto.

The result decoded by the first decoding unit 320 is transmitted to the first post-processing unit 330. The first post-processing unit 330 may be configured to perform constellation symbol mapping, and may also be configured to detect and correct errors.

The result post-processed by the first post-processing unit 330 is transmitted to the image source decoding unit 340. The image source decoding unit 340 may be configured to output a low-resolution image by performing, for example, source decoding on the result transmitted from the first post-processing unit 330.

As described above, according to an embodiment, optimal performance can be exhibited from an end-to-end perspective, and in particular, terminals having different number of antennas and having different screen resolutions, such as low-end terminals Even if the and high-end terminals are in the same area, images can be effectively transmitted to each of these terminals.

Meanwhile, according to an embodiment, the image transmission method may be performed by the image transmission apparatus 100 according to an embodiment, and the steps that may be included in each of these methods are instructions programmed to perform these steps. It may be stored in a computer-readable recording medium storing the.

Combinations of each block in the block diagram attached to the present invention and each step in the flowchart may be performed by computer program instructions. Since these computer program instructions can be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, the instructions executed by the processor of the computer or other programmable data processing equipment are shown in each block or flow chart of the block diagram. Each step creates a means to perform the functions described. These computer program instructions can also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular way, so that the computer-usable or computer-readable memory It is also possible to produce an article of manufacture in which the instructions stored in the block diagram contain instruction means for performing the functions described in each block or flow chart. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operating steps are performed on a computer or other programmable data processing equipment to create a computer-executable process to create a computer or other programmable data processing equipment. It is also possible for the instructions to perform the processing equipment to provide steps for performing the functions described in each block of the block diagram and each step of the flowchart.

In addition, each block or each step may represent a module, segment, or part of code comprising one or more executable instructions for executing the specified logical function(s). In addition, it should be noted that in some alternative embodiments, functions mentioned in blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, or the blocks or steps may sometimes be performed in the reverse order depending on the corresponding function.

The above description is merely illustrative of the technical idea, and those of ordinary skill in the technical field to which the present invention belongs will be able to make various modifications and variations without departing from the essential characteristics. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea, but to describe it, and the scope of the technical idea is not limited by these embodiments. The scope of protection should be interpreted by the claims below, and all technical thoughts within the scope equivalent thereto should be construed as being included in the scope of the rights.

Claims (5)

A downsampling unit that downsamples the original high-resolution image and converts it into a low-resolution image,
A first coding unit that codes the low-resolution image based on a first coding technique that can be decoded by each of a small user and a large user,
It includes a plurality of residual block units configured to include a convolutional layer and a ReLU layer, and when the low-resolution image converted by the down-sampling unit is input, it is previously learned to output a processed high-resolution image of lower quality than the original high-resolution image. With a super resolution unit,
A residual map providing unit that calculates a difference between the original high-resolution image and the processed high-resolution image and provides it in the form of a residual map,
Including a second coding unit for coding the residual map based on a second coding technique that can be decoded by the high-end terminal
Image sending device. The method of claim 1,
The first coding technique,
Alamouti technique
Image sending device. The method of claim 1,
The second coding technique,
Golden coding technique
Image sending device. The method of claim 1,
The residual block unit,
Configured to include the convolutional layer and the ReLU layer, but not include a batch normalization layer,
Image sending device. The method of claim 1,
The residual map providing unit,
Providing the residual map based on a convex optimization technique
Image sending device.

Images