JP5308322B2 - Image encoding apparatus, image decoding apparatus, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently improve the resolution of an image including a subject by using a reinforcement signal. <P>SOLUTION: An object detection means 2 of an image encoder 100 detects an object from a reduced image L of an image H to generate object area information D. An image encoding means 1 generates a reinforcement signal E for improving the resolution of an area indicated by the object area information D and encodes the reduced image L and the reinforcement signal E. An object detection means 4 of an image decoder 200 detects a prescribed object from the reduced image L to generate object area information D. An image decoding means 3 decodes a code string inputted from the image encoder 100 to generate the reduced image L and the reinforcement signal E and generates a sharpened image H from the reinforcement signal E disposed in an area indicated by the object area information D and the reduced image L. Thus, the data amount of the code string is small to allow the enhancement in efficiency of encoding because it is unnecessary to generate a code string of the object area information D. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image encoding device, an image decoding device, an image encoding program, and an image decoding program, and more particularly to a technique for realizing high definition of an image using a reinforcement signal.

  Conventionally, a second generation EDTV (Enhanced Definition Television) system, a hierarchical coding system, an object coding system, and the like are known as a system for realizing high definition of an image. The second generation EDTV system is an encoding and modulation system that uses a reinforcement signal. A horizontal resolution reinforcement signal is superimposed on an open hole in the spectrum of an NTSC signal, and a transmitted image having an aspect ratio of 16: 9 is aspect ratio 4 :. When transmitting in a 3-format letterbox, the vertical resolution reinforcement signal and the vertical time resolution reinforcement signal are transmitted using the non-image portions generated at the top and bottom of the screen.

  The hierarchical encoding method divides an image into a plurality of layers (mainly images having different resolutions), selects an encoding method for each layer, and performs encoding (for example, Patent Document 1, Patent Document 1). 2). The object encoding method divides an image into regions of partial images including a subject and the like, and performs encoding by applying a method and parameters for each region (for example, see Non-Patent Document 1). .

JP-A-63-73786 JP 2000-41149 A

“Information Technology-Coding of audio-visual objects-, Part 2: Visual, Amenity 1: Visual extensions,” ISO / IEC 14496-2

  However, in the above-mentioned second generation EDTV system, a reinforcement signal such as horizontal and vertical applied to the entire image is transmitted, so it cannot be applied only to a partial image including a local subject. For this reason, the reinforcement signal is assigned not only to a partial image including a local subject but also to other images. In other words, since the reinforcement signal is assigned not only to a region including a local subject that requires high definition but also to a region that does not require high definition, the efficiency of information transmission is impaired. .

  Further, in the above-described hierarchical encoding method disclosed in Patent Document 1, an image is divided (hierarchized) into a portion having a large influence on reproduction image quality and a portion having a small influence. However, this hierarchization is performed according to signal characteristics that are not directly related to the subject. For this reason, since the hierarchization is performed regardless of the presence or absence of a partial image including a predetermined important subject, there is a problem in that the region that is noticed by the user is not necessarily highly refined.

  Further, in the above-described hierarchical coding method of Patent Document 2, it is possible to realize the efficiency of information transmission by dividing an image and transmitting a reinforcement signal of only a region noted by a user as difference information. However, it is necessary to feed back a request operation for a region that is noticed by the user from the image decoding apparatus to the image encoding apparatus, and bidirectional communication is required. For this reason, this method has a problem that it cannot be applied to a unidirectional communication channel such as broadcasting. If the request operation is not fed back, the image decoding apparatus cannot recognize where the difference information received from the image encoding apparatus is an image of which area in the entire image. In addition to the entire image and the code string of the difference information, it is necessary to transmit the code string of the information related to the region to the image decoding apparatus. For this reason, there is a problem in that the amount of codes increases, the efficiency of encoding cannot be achieved, and the image of the area that is noticed by the user cannot be efficiently refined.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to perform image coding that can efficiently improve the resolution of an image including a subject that is an object using a reinforcement signal. An apparatus, an image decoding device, and a program are provided.

In order to achieve the above object, an invention according to claim 1 is directed to an image encoding device that inputs an image, reduces the resolution of the image, encodes the reduced-resolution image, and outputs a code string. An object detection means for detecting a predetermined object included in the resolution-reduced image, and an object detected by the object detection means by lowering the resolution of the input image and encoding it into a first code string An image that generates a reinforcement signal for increasing the resolution of the image based on the image, encodes the reinforcement signal into a second code string, and outputs the first code string and the second code string comprising a coding means, wherein the object detecting means, wherein detecting a predetermined object contained in the low resolution has been the image, space territory including the object in the image Is generated as object region information, and the image encoding means generates a reduced image with reduced resolution by thinning out pixels from the input image, and detects the predetermined object with the reduced image. Image reduction means for outputting to the object detection means as an object detection image, reduced image encoding means for encoding a reduced image generated by the image reduction means and generating a code string, and generated by the image reduction means An image enlargement unit that inserts pixels into the reduced image and generates an enlarged image, and an image of a spatial region indicated by the object region information generated by the object detection unit is cut out from the enlarged image generated by the image enlargement unit Generated by the object detection means from the first image cutout means and the input image. Difference between the second image cutout unit that cuts out the image of the spatial area indicated by the object region information, the image cut out by the first image cutout unit, and the image cut out by the second image cutout unit Are generated by the reduced image encoding means, and a difference calculation means for generating a reinforcement signal and a reinforcement signal encoding means for encoding the reinforcement signal generated by the difference calculation means and generating a code string. The encoded code sequence and the code sequence generated by the reinforcement signal encoding unit, and the image encoding unit further decodes the code sequence generated by the reduced image encoding unit, A decoded image is generated, and the local decoded image is output to the object detection means as an object detection image for detecting the predetermined object. Local image decoding means, wherein the image enlarging means enlarges the local decoded image generated by the local image decoding means to generate an enlarged image, and the object detection means is output by the local image decoding means. A predetermined object is detected based on the object detection image, and object area information is generated .

According to the first aspect of the present invention, the code string of the low resolution image and the reinforcement signal for increasing the resolution of the image is output, and the code string related to the region including the predetermined object is not output. Further, since the reinforcement signal is generated based on the image including the object, the image not including the object is not reflected in the reinforcement signal. Therefore, the data amount of the code string is small, and the burden of the encoding process can be reduced. Further, the decoding side can generate the same object region information by the same processing as the object detection means provided in the image encoding device, using the reduced image in the code string output by the image encoding device. By using the reinforcement signal in the code string output by the image encoding device, it is possible to increase the resolution of the image in the spatial area indicated by the object area information. Therefore, the image encoding device does not need to output the code sequence of the object area information to the decoding side, and the reinforcement signal is generated based on the image including the object. It is not reflected in. The reinforcement signal is generated by calculating the difference between the images. That is, the data amount of the code string is small, and the burden of the encoding process can be reduced. Further, since the image encoding device generates a reinforcement signal for a subject that is an object that is spatially uneven in the image, the decoding side uses the reinforcement signal to generate an image of the spatial region indicated by the object region information. Can be reproduced with high resolution, and a smooth contour and detailed texture of the subject can be reproduced. Further, since the reduced image obtained by decoding the code sequence on the decoding side is closer to the local decoded image than the reduced image in the image encoding device, the decoding side uses the reduced image obtained by decoding the code sequence to generate an image. By performing the same processing as the object detection means provided in the encoding apparatus, the same object area information as the object area information generated by the object detection means of the image encoding apparatus can be generated. Therefore, even when the reduced image encoding means uses an irreversible encoding method, there is almost no difference in the object detection result between the image encoding device and the decoding side, so that the image quality deterioration due to encoding is not caused. Even if it occurs, there is no effect on object detection and no failure occurs. In addition, since the reinforcement signal is generated by the enlarged image generated using the local decoded image, even when the reduced image encoding means uses an irreversible encoding method, image quality degradation due to encoding is reduced. The signal takes distortion into consideration. By using such a reinforcement signal, the decoding side can realize high resolution of an image in consideration of distortion of image quality degradation due to encoding.

According to a second aspect of the present invention, there is provided an image encoding apparatus for inputting an image, reducing the resolution of the image, encoding the reduced-resolution image, and outputting a code string, wherein the resolution is reduced. Based on an object detection unit that detects a predetermined object included in the image, the input image is reduced in resolution and encoded into a first code string, and the image includes the object detected by the object detection unit Image encoding means for generating a reinforcement signal for increasing the resolution of an image, encoding the reinforcement signal into a second code string, and outputting the first code string and the second code string; The object detection means detects a predetermined object included in the low-resolution image, and a space area including the object in the image is defined as an object area. Generated as information, and the image encoding means thins out pixels from the input image to generate a reduced image with reduced resolution, and the reduced image is used for object detection to detect the predetermined object. Image reduction means for outputting to the object detection means as an image, reduced image encoding means for encoding a reduced image generated by the image reduction means and generating a code string, and reduced image generated by the image reduction means An image enlarging unit that inserts a pixel into the image and generating an enlarged image; and a first image that cuts out an image of a spatial area indicated by the object area information generated by the object detecting unit from the enlarged image generated by the image enlarging unit An object segment information generated by the object detection means from the image cutout means and the input image. The difference between the second image cutout unit that cuts out the image of the spatial region indicated by the report, the image cut out by the first image cutout unit, and the image cut out by the second image cutout unit is calculated. And a difference calculation means for generating a reinforcement signal and a reinforcement signal encoding means for encoding the reinforcement signal generated by the difference calculation means and generating a code string, and generated by the reduced image encoding means. From the image encoding device that outputs the code sequence generated and the code sequence generated by the reinforcement signal encoding means, a code sequence of a low resolution image and a code sequence of a reinforcement signal for obtaining a high resolution image are obtained. In an image decoding device that inputs and decodes the code string and outputs an image, a low-resolution image generated by decoding the code string is used to generate a predetermined object included in the low-resolution image. Object detection means for detecting an object, and decoding the input code string, generating a low-resolution image and a reinforcement signal, arranging the reinforcement signal in a region where the object is detected, and generating a reinforcement image, Image decoding means for adding the low-resolution image and the reinforcement image and outputting the addition result image is provided.

According to the invention of claim 2 , a low-resolution image and a code string of a reinforcement signal for obtaining a high-resolution image are input, an object is detected from the low-resolution image, and the object is reinforced in the detected region. Since the reinforcement image is generated by arranging the signals, a sharpened image can be generated by adding these images. As a result, there is no need to input a code string related to the area including the object, and the data amount of the code string can be reduced.

The predetermined invention of claim 3, the image decoding apparatus according to claim 2, from the image encoding device, enter the code sequence, said object detecting means is included in said low-resolution image Reduced image decoding means for detecting the object of the image, generating a spatial area including the object in the image as object area information, and wherein the image decoding means decodes the code string of the low resolution image and generates a reduced image A reinforcement signal decoding unit that decodes the code string of the reinforcement signal and generates a reinforcement signal; an image enlargement unit that inserts a pixel into the reduced image generated by the reduced image decoding unit and generates an enlarged image; The reinforcement signal generated by the reinforcement signal decoding means is arranged in the space area indicated by the object area information generated by the object detection means, and reinforcement is performed. An arrangement means for generating an image, an addition means for adding the enlarged image generated by the image enlargement means and the reinforcing image generated by the arrangement means, and outputting an image after addition is provided. And

According to the invention of claim 3 , object region information indicating a spatial region of a spatially localized object is generated from the reduced image, and a reinforcement signal is arranged in the spatial region indicated by the object region information to generate a reinforcement image. Since the image is generated, a sharpened image can be generated by adding the reduced image and the reinforcing image. As a result, it is possible to reproduce the smooth outline and detailed texture of the subject that is the object. Further, it is not necessary to input a code string of the object area information, and the data amount of the code string can be small.

According to a fourth aspect of the present invention, in the image decoding device according to the third aspect , in place of the arrangement unit and the addition unit, an arrangement addition unit is provided, and the arrangement addition unit is generated by the image enlargement unit. Adding the reinforcement signal generated by the reinforcement signal decoding means to the image of the spatial area indicated by the object area information generated by the object detection means in the enlarged image, and outputting the enlarged image after the addition; To do.

According to the fourth aspect of the present invention, since the reinforcement signal is directly added to the enlarged image without generating the reinforcement image, and the sharpened image is generated, the burden of the decoding process can be reduced.

Further, the invention of claim 5 is the image decoding device according to claim 3 , wherein the object detection means includes F (F is an integer of 0 or more) regions Di (i = 0,...) Each including the object. .., F-1) are generated as object area information, and the reinforcement signal decoding means decodes the code string of the reinforcement signal to generate F reinforcement signals Ei, and the arrangement means detects the object detection The F reinforcement signals Ei generated by the reinforcement signal decoding means are arranged in F areas Di indicated by the object area information generated by the means, and a reinforcement image is generated.

According to the invention of claim 5 , since the object region information Di and the reinforcement signal Ei are generated according to the number of objects included in the image, even if the objects are scattered in the image Decryption processing is performed.

The invention of claim 6 is the image decoding apparatus according to claim 3 , further comprising display means for displaying a reduced image generated by the reduced image decoding means, the display means being operated by a user. The image of the area indicated by the object area information generated by the object detection means is cut out from the added image output by the addition means and displayed on the screen.

According to invention of Claim 6 , according to a user's operation, among the reduced images displayed on the screen, a high-resolution, easy-to-see partial image including an object can be presented to the user.

The invention of claim 7 is a computer, the image encoding program for functioning as an image encoding apparatus according to claim 1.

The invention of claim 8 resides in an image decoding program for causing a computer to function as the image decoding apparatus according to any one of claims 2 to 6 .

  As described above, according to the present invention, it is possible to efficiently improve the resolution of an image including a subject that is an object using a reinforcement signal.

It is a block diagram which shows the structure of the image coding apparatus by embodiment of this invention. It is a block diagram which shows the structure of the image coding means by Example 1. It is a figure explaining the example of an image processed by the image coding means of Example 1 in the spatial direction. It is a block diagram which shows the structure of the image coding means by the modification of Example 1. It is a figure explaining the example of an image processed by the image coding means of the modification of Example 1 in the space direction. It is a figure explaining the example of an image processed by the image coding means of Example 2 in a time direction. It is a figure explaining the example of an image processed by the image coding means of the modification of Example 2 in a time direction. It is a block diagram which shows the structure of the image decoding apparatus by embodiment of this invention. It is a block diagram which shows the structure of an image decoding means. It is a figure explaining the example of an image processed by the image decoding means corresponding to the modification of Example 1 and Example 1 in a spatial direction. It is a figure explaining the example of an image processed in the time direction by the image decoding means corresponding to the modification of Example 2 and Example 2. FIG.

The best mode for carrying out the present invention will be described below with reference to the drawings.
The present invention includes an image encoding apparatus that inputs and encodes an image and outputs a code string, and an image decoding apparatus that receives the code string from the image encoding means, decodes the code string, and outputs an image. In this system, the image encoding device detects a predetermined object based on the low-resolution image, generates object region information, and generates a reinforcement signal for increasing the resolution of the region indicated by the object region information. The low-resolution image and the reinforcement signal are encoded, and a code string is output. Further, according to the present invention, an image decoding apparatus inputs a code string from an image encoding apparatus, decodes the code string to generate a low resolution image and a reinforcement signal, and generates a predetermined object based on the low resolution image. Is detected to generate object region information, and a sharpened image is generated from a reinforcement signal arranged in the region indicated by the object region information and a low-resolution image. Thus, the data output from the image encoding device to the image decoding device is a code sequence of a low-resolution image and a code sequence of a reinforcement signal arranged in the area indicated by the object area information. Since the region information code string is not included, the data amount of the code string is small. Therefore, it is possible to efficiently improve the resolution of an image including a subject that is an object using the reinforcement signal.

[Image coding device]
First, an image encoding device according to an embodiment of the present invention will be described. FIG. 1 is a block diagram illustrating a configuration of an image encoding device. The image encoding apparatus 100 includes an image encoding unit 1 and an object detection unit 2. The image encoding apparatus 100 receives an image H, and performs image encoding processing and object detection processing to increase the resolution of the spatial region or time region reinforcement signal E _i indicated by the object region information D, A code string μ including a low-resolution image that is the reduced image L is generated and output to the image decoding apparatus.

  The image encoding means 1 receives the image H, receives the object area information D from the object detection means 2, performs image encoding based on the image H and the object area information D, and generates a code string μ. Output. The image encoding unit 1 generates an image (object detection image) used by the object detection unit 2 to detect an object based on the image H, and outputs the object detection image to the object detection unit 2. .

  The object detection means 2 receives an object detection image from the image encoding means 1, detects a predetermined object from the object detection image, and determines a spatial region or time determined to require transmission of a high-definition image. A region (region including the object) is generated as object region information D and output to the image encoding means 1.

Hereinafter, the case where the image is a monochrome image will be described as an example. However, in the case of a color image having a plurality of components (color components) according to color, wavelength, or spectral characteristics, each pixel is represented as a color vector having each color component as an element. It is assumed that the pixel value of is handled. Therefore, the image is not limited to a monochrome image but can be applied to a color image or the like. Also, the size of the image H input to the image encoding means 1 is defined as the width S _x and the height S _y , the coordinates of the horizontal position x and the vertical position y of the image are (x, y), and the coordinates (x, The pixel value of the image H in y) is assumed to be H (x, y).

  Hereinafter, the image encoding unit 1 and the object detection unit 2 of the image encoding apparatus 100 will be described separately in the first embodiment, the modified example of the first example, the modified example of the second example, and the second example. The first embodiment is an example in which image processing in the spatial direction is performed in order to improve the spatial resolution of the image, and the second embodiment is an example in which image processing in the temporal direction is performed in order to improve the temporal resolution of the image. is there. The first and second embodiments are examples in which the reduced image L of the image H is used as an object detection image and the object region information D is generated based on the object detection image. In the example, the code sequence of the reduced image L of the image H is decoded to generate a local decoded image L ′, the local decoded image L ′ is used as an object detection image, and the object region information D is generated based on the object detection image This is an example.

[Example 1]
First, Example 1 will be described. In Example 1, in order to improve the spatial resolution of the image H, the image encoding device 100 uses the reduced image L of the image as an object detection image, generates object region information D based on the object detection image, Perform spatial image processing.

  The object detection means 2 of the image encoding device 100 shown in FIG. 1 receives a reduced image L that is an object detection image from the image encoding means 1, detects a predetermined object from the reduced image L, and detects the object. Is output to the image encoding means 1 as object region information D. Specifically, the object detection unit 2 detects a region that matches a template such as a preset image pattern or color histogram from the reduced image L as a predetermined object. Further, the object detection means 2 may detect a predetermined object from the reduced image L by using an optical character recognition (OCR) method based on a preset character group template. A face object may be detected from the reduced image L by using an existing face detection method such as a boosting method based on the Haar-like feature, or a region having a high local contrast in the image may be locally detected. Use a variety of methods to calculate the importance of an image, such as distinct colors (such as colors that deviate from the average color), specific color areas (areas with skin color), etc. May be. Here, the object area information D output by the object detection means 2 is a coordinate information group indicating an area including the detected object, and is information on the same spatial scale as the image H.

The object area information D includes F (F is an integer greater than or equal to 0) subset area, and the i-th area (i∈ {0,..., F−1}) Let _Di. The area D _i may have any shape, for example, a binary image in which the pixel value outside the area is 0, the pixel value in the area is 1, and the number of vertices and the image coordinate group of the vertices. You may make it make it the outline which expressed the square, the bounding box represented by two image coordinates of upper left and lower right, or the boundary of an area | region by the chain code. Formulas for the region D _i will be described later (see formulas (19), (20), and (21)).

  FIG. 2 is a block diagram illustrating a configuration of the image encoding unit 1 according to the first embodiment. The image encoding unit 1 includes an image reduction unit 10, a reduced image encoding unit 11, an image enlargement unit 12, image clipping units 13 and 14, a difference calculation unit 15, a reinforcement signal encoding unit 16, and a multiplexing unit 17. Configured.

(Image reduction means)
The image reduction means 10 receives the image H, thins out a predetermined pixel in the image H, reduces it, generates a reduced image L with low resolution (low spatial resolution), and reduces the image encoding means 11 and image enlargement means. 12 and output to the object detection means 2 as an object detection image. Specifically, the image reducing means 10 assumes that the width of the reduced image L is T _x , the height is T _y, and the pixel value of the reduced image L at the coordinates (x, y) is L (x, y). Then, the pixels of the image H are sampled by sampling at intervals of (S _x / T _x , S _y / T _y ), and the reduced image L is generated by the following equation. As a result, the reduced image L has a smaller number of pixels than the image H, and becomes a spatially low resolution image.

Q _x and Q _y are functions for scaling and quantizing the coordinate values. For example, Q _x is expressed by the following expression (2) or expression (3), and Q _y is expressed by the following expression (4) or expression ( 5).

Here, O _x and O _y are constants for shifting the sampling position (phase), and preferably 0 ≦ O _x <S _x / T _x and 0 ≦ O _y <S _y / T _y . Equations (2) and (4) are obtained by rounding down when the sampling position obtained by (S _x / T _x ) x + O _x or (S _y / T _y ) y + O _y becomes a non-integer. Used when performing integerization. Further, when the sampling position obtained by (S _x / T _x ) x−O _x or (S _y / T _y ) y−O _y becomes a non-integer, Expression (3) and Expression (5) In addition, it is used when rounding up to the nearest whole number.

ここで、Ｍ_ｘ＝Ｓ_ｘ／Ｔ_ｘ、Ｍ_ｙ＝Ｓ_ｙ／Ｔ_ｙとし、Ｍ_ｘ，Ｍ_ｙをスケールという。Ａは、フィルタの直流ゲインが１となるように定めた定数である。ｓｉｎｃ関数は、以下の式により定義される。

The image reducing means 10 applies an image filter to the image H and then samples the image H by thinning out the pixels of the image H at intervals of (S _x / T _x , S _y / T _y ) to generate a reduced image L. You may make it do. For example, a low-pass filter (LPF: Low-Pass Filter) having an impulse response f (x, y) characteristic represented by the following equation can be used as the image filter. The impulse response f (x, y) has a characteristic that approximates the product of each sinc function with respect to x and y.

_{Here, M x = S x / T} x, and _{M y = S y / T y} , M x, the _{M y} of scale. A is a constant determined so that the DC gain of the filter is 1. The sinc function is defined by the following equation.

ここで、Ｂ_Ｍは、フィルタの直流ゲインが１となるように定めた定数である。 In this case, the image reducing means 10 may apply the LPF to the image H in an independent and cascade manner in the horizontal direction and the vertical direction when applying the image filter to the image H. Assuming that the impulse response by the LPF of the scale M with respect to one axis in the horizontal direction or the vertical direction is f _M (t), the impulse response f _M (t) can be expressed by the following equation.

Here, B _M is a constant determined so that the DC gain of the filter is 1.

直流ゲインが１であるから、

となる。 For example, when an LPF with a scale M = 2 is configured by a 7-tap filter, the impulse response f _M (t) can be expressed by the following equation.

Since the DC gain is 1,

It becomes.

これは、整数の掛け算及び右８ビットシフト（２５６の割り算）により、近似的に以下の式で表すことができる。

That is, it is represented by the following formula.

This can be approximately expressed by the following equation by integer multiplication and right 8-bit shift (256 division).

  For example, when the image H is reduced to ½ each in the horizontal direction and the vertical direction, the image reduction means 10 first applies the LPF having the tap coefficient of Expression (7) or Expression (8) to the image H. Apply one-dimensionally in the horizontal direction (or vertical direction), and then apply the resulting image one-dimensionally in the vertical direction (or horizontal direction). Sub-sampling in the direction with a period of two pixels. As a result, a reduced image L obtained by reducing the image H by 1/2 in the horizontal direction and the vertical direction is generated.

(Reduced image encoding means)
The reduced image encoding means 11 receives the low resolution reduced image L from the image reducing means 10, encodes the reduced image L and converts it into a code string λ, and sends the code string λ of the low resolution image to the multiplexing means 17. Output. Specifically, the reduced image encoding means 11 expresses the pixel value of each pixel of the reduced image L as a finite code string, and scans the reduced image L to convert the code string of each pixel in a one-dimensional manner. To generate a code string λ and output it. This encoding method is called pulse code modulation (PCM). That is, the reduced image encoding means 11 changes x from 0 to T _x −1 for each y while changing _y from 0 to T _y −1 in the image coordinates (x, y) on the reduced image L. By doing so, the pixel value L (x, y) at the pixel position (x, y) is quantized and converted into a finite code string, and the code string is sequentially combined one-dimensionally to generate a code string λ. And output

  Note that the reduced image encoding unit 11 may use an existing image encoding method or a moving image encoding method. For example, JPEG, JPEG 2000, MPEG-1, MPEG-2, MPEG-4, MPEG-4 AVC / H. H.264 or the like is used.

(Image enlargement means)
The image enlarging means 12 receives the reduced image L having a low resolution from the image reducing means 10, and inserts and enlarges the reduced image L until the same size as the image H (the number of pixels in the horizontal and vertical directions). Then, an enlarged image K is generated and output to the image cutout means 13. Here, the enlarged image K is a blurred image because there is no broad spectrum component. Specifically, the image enlarging means 12 enlarges the reduced image L by the nearest neighbor interpolation method shown in the following equation to generate an enlarged image K.

R _x and R _y are functions for scaling and quantizing coordinate values. For example, R _x is expressed by the following formula (13) or formula (14), and R _y is expressed by the following formula (15) or formula ( 16).

Here, P _x and P _y are constants that determine the method of rounding decimals, and preferably 0 ≦ P _x <1 and 0 ≦ P _y <1. In particular, the expression (13) at P _x = 1/2 and the expression (15) at P _y = 1/2 are rounded off by rounding off.

ここで、

である。 The image enlarging means 12 may enlarge the reduced image L by an interpolation method other than the nearest neighbor method, and generate an enlarged image K. For example, the image enlarging means 12 generates an enlarged image K as shown in the following equation by a bilinear interpolation method.

here,

It is.

ここで、

である。 Further, for example, the image enlarging means 12 generates an enlarged image K as shown in the following equation by a bicubic interpolation method.

here,

It is.

(Image clipping means)
The image cutout means 13 inputs the enlarged image K from the image enlargement means 12, receives the object area information D from the object detection means 2, and from the enlarged image K, the coordinates of the spatial area indicated by the object area information D are displayed. The partial image B _i is cut out and the partial image B _i is output to the difference calculation means 15. The object area information D may be a plurality of aggregate strings (F aggregate strings). F is an integer of 0 or more. In this case, the partial image B _i output by the image cutout unit 13 corresponds to each area D _i (i∈ {0,..., F−1}) included in the object area information D. When F = 0, the partial image B _i is not output. That is, the image cutout unit 13 cuts out the partial image B _i for all indexes i = 0,..., F−1, and the partial image sequence (B _i ) _{i = 0,. -1} is output.

ここで、φは空集合である。領域Ｄ_ｉは、任意の形状であってもよいが、計算の簡易化のため、例えば、各辺が画像の水平軸または垂直軸と平行な長方形または正方形の領域内及びその境界とするのが望ましい。 The object area information D is defined by the following formula.

Here, φ is an empty set. The area D _i may have any shape, but for simplification of calculation, for example, each area may be in a rectangular or square area parallel to the horizontal or vertical axis of the image and its boundary. desirable.

ここで、

とする。この場合、部分画像Ｂ_ｉは、以下の式により表される。

For example, if the region D _i is a rectangular or square region and its boundary, the left side x coordinate is p _i , the right side x coordinate is q _i , the upper side y coordinate is r _i , and the lower side y coordinate is s. _{Assuming i} , the region D _i is defined by the following equation.

here,

And In this case, the partial image B _i is represented by the following equation.

On the other hand, the image cutout means 14 inputs the image H, inputs the object area information D from the object detection means 2, and cuts out the partial image C _i of the coordinates of the spatial area indicated by the object area information D from the image H. The partial image C _i is output to the difference calculation means 15. The object area information D is the same as that input by the image cutout means 13, and the partial image C _i is an area D _i (i∈ {0,..., F-1} included in the object area information D. ). When F = 0, the partial image C _i is not output. The image cutout unit 14 performs the same process as the image cutout unit 13, inputs the image H instead of the enlarged image K, and generates and outputs a partial image C _i instead of the partial image B _i . That is, the image cutout unit 14 generates a partial image C _i for all indexes i = 0,..., F−1, and the partial image sequence (C _i ) _{i = 0,. F-1} is output.

For example, when the object area information D is in a rectangular or square area and its boundary as shown in Expression (20), the partial image C _i is expressed by the following expression.

(Difference calculation means)
The difference calculation means 15 inputs the partial image B _i from the image cut-out means 13, inputs the partial image C _i from the image cut-out means 14, and obtains the difference image E _i between the partial image B _i and the partial image C _i. The difference image E _i is calculated and output to the reinforcement signal encoding means 16 as a reinforcement signal. Here, the difference calculation means 15 calculates the difference image E _i for all the indexes i = 0,..., F−1, and the difference image image sequence (E _i ) _{i = 0,. , F-1} is output. When F = 0, the difference calculation means 15 does not output the difference image E _i because the partial images B _i and C _i are not input from the image cutout means 13 and 14.

Specifically, the difference calculation means 15 generates the difference image E _i by the normal subtraction shown in the following equation.

部分画像Ｂ_ｉ及び部分画像Ｃ_ｉがカラー画像の場合、差分演算手段１５は、色成分毎に式（２５）の演算を行い、差分画像Ｅ_ｉを求める。 Note that the difference calculation means 15 uses the partial image B _i and the partial image C _i modulo the number of quantization steps G of the luminance of the image (eg, G = 2 ^V when the luminance value is quantized with V bits). The difference image E _i may be obtained by subtraction between the two as shown in the following equation.

When the partial image B _i and the partial image C _i are color images, the difference calculation means 15 calculates the equation (25) for each color component to obtain the difference image E _i .

ここで、オフセットＵは、好ましくは量子化ステップ数Ｇの１／２（小数点以下の端数を生じる場合は整数に丸める）とする。例えば、Ｖビットで輝度値を量子化した場合、Ｕ＝２^Ｖ−１とする。部分画像Ｂ_ｉ及び部分画像Ｃ_ｉがカラー画像の場合、差分演算手段１５は、色成分毎に式（２６）の演算を行い、差分画像Ｅ_ｉを求める。また、差分演算手段１５は、部分画像Ｂ_ｉ及び部分画像Ｃ_ｉの階調数を法とする剰余の差分を演算し、この演算結果を差分画像Ｅ_ｉとして生成するようにしてもよい。 Further, the difference calculation means 15 performs subtraction between the partial image B _i and the partial image C _i modulo the quantization step number G of the luminance of the image, adds the offset U to the subtraction result, and As in the equation, the difference image E _i may be obtained.

Here, the offset U is preferably ½ of the number of quantization steps G (rounded to an integer when a fractional part is generated). For example, when the luminance value is quantized with V bits, U = ^2V-1 . When the partial image B _i and the partial image C _i are color images, the difference calculation means 15 calculates the equation (26) for each color component to obtain the difference image E _i . Further, the difference calculation means 15 may calculate a residual difference modulo the number of gradations of the partial image B _i and the partial image C _i and generate the calculation result as a difference image E _i .

(Reinforcement signal encoding means)
The reinforcement signal encoding means 16 receives the difference image E _i (reinforcement signal E _i ) from the difference calculation means 15, encodes the difference image E _i to convert it into a code string ε, and multiplexes the code string ε. Output to. In the case of F = 0, the reinforcement signal encoding unit 16 does not output the code sequence ε or outputs a predetermined code sequence ε for identifying that the difference image E _i does not exist.

Specifically, the reinforcement signal encoding means 16 expresses the pixel value of each pixel of the difference image E _i with a finite code sequence for all indexes i = 0,. By scanning the image E _i , the code sequence of each pixel is one-dimensionally combined to generate and output a code sequence ε. This encoding method is called pulse code modulation. That is, the reinforcement signal encoding unit 16 changes y from 0 to s _i −r in the image coordinates (x, y) on the difference image E _i for all indexes i = 0,. while changing to _i, by changing x from 0 to _q i -p _i for each y, finite code sequence that pixel position (x, y) pixel values in the _E i a (x, y) is quantized Is converted to, and the code sequence is sequentially combined one-dimensionally to generate and output a code sequence ε.

  The reinforcement signal encoding unit 16 may use an existing image encoding method or moving image encoding method. For example, JPEG, JPEG 2000, MPEG-1, MPEG-2, MPEG-4, MPEG-4 AVC / H. H.264 or the like is used.

(Multiplexing means)
The multiplexing unit 17 receives the code sequence λ of the reduced image L from the reduced image encoding unit 11 and the code sequence ε of the difference image E _i (reinforcement signal E _i ) from the reinforcement signal encoding unit 16. The sequence λ and the code sequence ε are multiplexed to generate a code sequence μ and output it to the image decoding apparatus 200. Specifically, the multiplexing unit 17 combines and multiplexes the code sequence λ and the code sequence ε in a preset order to generate a code sequence μ.

  The multiplexing means 17 may generate the code string μ including additional information such as header information or footer information when generating the code string μ. For example, the additional information is information for specifying a position in the code string μ in which the code string λ and the code string ε are described.

(Example of image processed by image encoding means of embodiment 1)
FIG. 3 is a diagram illustrating an example of an image processed in the spatial direction by the image encoding unit 1 according to the first embodiment. The reduced image L is generated from the image H by the image reducing unit 10, the object area information D is generated from the object detection image that is the reduced image L by the object detecting unit 2, and the enlarged image K is converted by the image expanding unit 12. It is generated from the reduced image L. The partial image C _i is generated from the image H and the object region information D by the image cutout unit 14, and the partial image B _i is generated from the enlarged image K and the object region information D by the image cutout unit 13. The reinforcing signal _{E i} is a differential image _{E i} is the differential operation circuit 15, obtained by subtracting the partial image _{B i} from the partial images _{C i.}

The code sequence λ of the low resolution image is generated from the reduced image L by the reduced image encoding unit 11, and the code sequence ε for generating the high resolution image is generated by the reinforcement signal encoding unit 16 using the difference image E. a _i is generated from the reinforcing signal _{E i.}

As described above, according to the image encoding device 100 including the image encoding unit 1 of the first embodiment, the object detection unit 2 performs high definition based on the object detection image that is the reduced resolution L of the low resolution. Object area information D (information of an area including a predetermined object) determined to be necessary to transmit a high-resolution image is generated, and the difference calculation means 15 of the image encoding means 1 indicates the object area information D The reinforcement signal E _i for increasing the resolution of the spatial region is generated by calculating the difference between the image H and the enlarged image K obtained by enlarging the reduced image L, and the multiplexing unit 17 performs the reduction image. the coding unit 11 is generated from the reduced image L, a code sequence λ of the low-resolution image, which is generated from the reinforcing signal E _i by the reinforcing signal encoding unit 16, and a code sequence for the higher resolution ε multi However, and it outputs a code string μ to the image decoding apparatus 200. Accordingly, the image decoding apparatus 200 can generate the object region information D from the reduced image L obtained by decoding the code string λ by using the same means as the object detection means 2 provided in the image encoding apparatus 100. it can. In addition, for the spatial region indicated by the object region information D, a high-definition high-resolution image that reproduces the smooth contour and detailed texture of the subject that is the object is generated from the reinforcement signal E _{i obtained} by decoding the code sequence ε. Can do.

The signal encoded by the reinforcement signal encoding unit 16 in the image encoding unit 1 of the image encoding apparatus 100 is a spatial region of the object region information D that is determined to require transmission of a high-definition high-resolution image. needed only reinforcing signal E _i for reinforcing signal E _i is the partial image B _i in the difference calculating unit _15, since is generated by a difference calculation of C _i, it is possible to reduce the code amount. The code sequence μ output from the image encoding device 100 to the image decoding device 200 includes the code sequence λ of the low resolution image and the reinforcement signal E _i for increasing the resolution in the spatial region indicated by the object region information D. Since the code string ε does not include the object area information D related to the arrangement of the objects, the code amount can be reduced. By reducing the amount of codes, it is possible to improve the encoding efficiency. That is, it is possible to efficiently improve the spatial resolution of the partial image including the subject that is the object by using the reinforcement signal E _i .

In the first embodiment, the image enlarging unit 12 and the image clipping unit 13 included in the image encoding unit 1 have been described as components of separate processing units. However, these are components of one processing unit. You may do it. Specifically, the image encoding unit 1 includes an image enlargement and cutout unit 18 having these functions as shown in FIG. 2 instead of the image enlargement unit 12 and the image cutout unit 13. The image enlargement / cutout means 18 inputs the reduced image L from the image reduction means 10 and the object area information D from the object detection means 2, and each area D _i (i = 0,...) Included in the object area information D. .., F-1), the reduction ratio (scale M _x = S _x / T _x , M _y = S _y / T) when generating a reduced image L by thinning predetermined pixels from the image H in the image reduction means 10. Considering _y ), a predetermined area is thinned out from each area D _i at the same reduction ratio and reduced. Then, the image enlargement / cutout unit 18 cuts out a partial image from the reduced image L based on the reduced area information, and considers the enlargement ratio when the image enlargement unit 12 enlarges the image by increasing the pixels, and the same enlargement ratio. It expanded by inserting a pixel in the partial image at, for generating a partial image B _i. As a result, the image encoding unit 1 enlarges the clipped partial image without generating the enlarged image K, so that the calculation amount and the memory amount are small, and the reinforcement signal E _i is used to Thus, the spatial resolution of the partial image including the subject can be improved more efficiently.

[Modification of Example 1]
Next, a modification of the first embodiment will be described. In the modification of the first embodiment, the image encoding device 100 generates a locally decoded image L ′ by decoding the code sequence of the reduced image L of the image H in order to improve the spatial resolution of the image, and generates the locally decoded image. L ′ is an object detection image, object region information D is generated based on the object detection image, and image processing in the spatial direction is performed.

  In the modification of the first embodiment, the object detection unit 2 of the image encoding device 100 inputs the local decoded image L ′ instead of the reduced image L as the object detection image, and the object detection unit 2 of the first embodiment Similar processing is performed.

  FIG. 4 is a block diagram illustrating a configuration of the image encoding unit 1 according to a modification of the first embodiment. The image encoding means 1 includes an image reduction means 10, a reduced image encoding means 11, an image enlargement means 12, an image clipping means 13, 14, a difference calculation means 15, a reinforcement signal encoding means 16, a multiplexing means 17 and a local means. An image decoding means 19 is provided. 4, parts common to the configuration of the image encoding means 1 according to the first embodiment shown in FIG. 2 are assigned the same reference numerals as those in FIG. 2, and detailed descriptions thereof are omitted.

  Comparing the image encoding means 1 according to the first embodiment shown in FIG. 2 with the image encoding means 1 according to the modification of the first embodiment shown in FIG. 4, both means are the image reduction means 10 and the reduced image encoding. Means 11, image enlargement means 12, image cutout means 13 and 14, difference calculation means 15, reinforcement signal encoding means 16, and multiplexing means 17 are the same. However, the image encoding means 1 according to the modification of the first embodiment shown in FIG. 4 includes a local image decoding means 19 in addition to the configuration of the image encoding means 1 according to the first embodiment shown in FIG. Is different.

Specifically, the image encoding unit 1 according to the first embodiment shown in FIG. 2 outputs the reduced image L as an object detection image to the object detection unit 2, and the object detection unit 2 outputs the reduced image L based on the reduced image L. The generated object area information D is input. On the other hand, the image encoding means 1 according to the modification of the first embodiment shown in FIG. 4 uses the object detection means 2 with the local decoded image L ′ obtained by decoding the code string λ of the reduced image L as the object detection image. The object area information D generated based on the local decoded image L ′ is input from the object detection means 2. Also, the image encoding means 1 according to the first embodiment shown in FIG. 2 enlarges the reduced image L, cuts out the partial image B _i , and generates the reinforcement signal E _i . On the other hand, the image encoding means 1 according to the modification of the first embodiment shown in FIG. 4 enlarges the local decoded image L ′, cuts out the partial image B _i , and generates the reinforcement signal E _i .

(Local image decoding means)
In FIG. 4, the local image decoding unit 19 receives the code sequence λ of the low-resolution image that is the reduced image L from the reduced image encoding unit 11, decodes the code sequence λ, and generates a local decoded image L ′. While outputting to the image enlarging means 12, it outputs to the object detection means 2 as an object detection image. The local image decoding unit 19 is a decoding unit that is paired with the reduced image encoding unit 11.

  Specifically, when the reduced image encoding unit 11 uses the PCM encoding method, the local image decoding unit 19 decomposes the code sequence λ into partial code sequences for each pixel, and converts the decomposed partial code sequence into an inverse quantum. And return to the luminance value, and this is decoded by mapping it to each pixel position of the local decoded image L ′ in the order of raster scan, thereby generating a local decoded image L ′.

  Further, the reduced image encoding means 11 uses JPEG, JPEG 2000, MPEG-1, MPEG-2, MPEG-4, MPEG-4 AVC / H. When using an existing image coding method or moving image coding method such as H.264, the local image decoding unit 19 generates a local decoded image L ′ using a decoding method that is paired with the coding method.

(Example of image processed by image encoding means of modification of embodiment 1)
FIG. 5 is a diagram illustrating an example of an image processed in the spatial direction by the image encoding unit 1 according to a modification of the first embodiment. The reinforcement signal E _i which is the reduced image L, the partial images B _i and C _i and the difference image E _i is generated in the same manner as in the first embodiment. The local decoded image L ′ is generated from the code string λ by the local image decoding means 19, the object region information D is generated from the object detection image that is the local decoded image L ′ by the object detection means 2, and the enlarged image K is The image enlargement unit 12 generates the local decoded image L ′.

The code sequence λ of the low resolution image is generated from the reduced image L by the reduced image encoding unit 11, and the code sequence ε for generating the high resolution image is generated by the reinforcement signal encoding unit 16 using the difference image E. a _i is generated from the reinforcing signal _{E i.}

As described above, according to the image encoding device 100 including the image encoding unit 1 according to the modification of the first embodiment, the local image decoding unit 19 of the image encoding unit 1 is a low-resolution image that is a reduced image L. To generate a local decoded image L ′, and the object detection means 2 needs to transmit a high-definition high-resolution image based on the object detection image that is the local decoded image L ′. The object region information D (region information including a predetermined object) that is determined to be generated is generated, and the difference calculation unit 15 of the image encoding unit 1 enhances the spatial region indicated by the object region information D to have a high resolution. The signal E _i is generated by calculating a difference between the image H and the enlarged image K obtained by enlarging the local decoded image L ′, and the multiplexing unit 17 uses the reduced image encoding unit 11 to reduce the reduced image L. Generated from The, the code sequence λ of the low-resolution image, which is generated from the reinforcing signal E _i by the reinforcing signal encoding unit 16, multiplexes the code string ε for high resolution, the image decoding apparatus a code string mu 200 Output to. Thereby, the image decoding apparatus 200 uses the same means as the object detection means 2 included in the image encoding apparatus 100 to obtain the object region information D from the reduced image L obtained by decoding the low-resolution image code string λ. Can be generated. In addition, for the spatial region indicated by the object region information D, a high-definition high-resolution image that reproduces the smooth contour and detailed texture of the subject that is the object is generated from the reinforcement signal E _{i obtained} by decoding the code sequence ε. Can do.

The image encoding unit 1 according to the modification of the first embodiment generates the object region information D by using the local decoded image L ′ generated by the local image decoding unit 19 as the object detection image, and the reinforcement signal E _i. Is generated to generate a code string ε. On the other hand, the image encoding unit 1 according to the first embodiment uses the reduced image L generated by the image reducing unit 10 as an object detection image, thereby generating object region information D and generating a reinforcement signal E _i. A code string ε is generated. In this case, the image decoding apparatus 200 generates object region information D and generates a sharpened image H by using a reduced image L generated by a reduced image decoding unit 22 described later as an object detection image. Become. Here, the reduced image L generated by the image reducing unit 10 of the first embodiment corresponds to the reduced image L generated by the reduced image decoding unit 22 of the image decoding apparatus 200, but a modified example of the first embodiment. The local decoded image L ′ generated by the local image decoding means 19 is closer to the reduced image generated by the reduced image decoding means 22 of the image decoding apparatus 200 than the reduced image L of the first embodiment. Therefore, even when the reduced image encoding unit 11 uses an irreversible encoding method, the image encoding device 100 and the image decoding device 200 can generate the same object region information D. That is, even if image quality deterioration due to the encoding of the reduced image encoding unit 11 occurs, the generated object area information D is not affected, and the object detection does not fail. Further, since the reinforcement signal E _i is generated by the enlarged image K generated using the local decoded image L ′, even if the reduced image encoding unit 11 uses an irreversible encoding method, The signal takes into account distortion of image quality degradation due to encoding. In other words, the image decoding apparatus 200 uses such a reinforcement signal E _i to realize high image resolution regardless of any image quality degradation distortion caused by the encoding of the reduced image encoding unit 11. Can do. That is, the image encoding unit 1 according to the modification of the first embodiment generates the reinforcement signal E _i in consideration of the decoding process of the image decoding apparatus 200 and generates the code string ε. A sharpened image H that further reflects the image H input to the encoding device 100 can be generated.

In addition, according to the image encoding device 100 including the image encoding unit 1 of the modification of the first embodiment, the signal encoded by the reinforcement signal encoding unit 16 of the image encoding unit 1 as in the first embodiment. Is only the reinforcement signal E _i for the spatial area indicated by the object area information D, and the reinforcement signal E _i is generated by the difference calculation of the partial images B _i and C _{i in} the difference calculation means 15. Can be reduced. Further, since the code string output from the image encoding device 100 to the image decoding device 200 does not include the object area information D related to the arrangement of the objects, the code amount can be reduced. By reducing the amount of codes, it is possible to improve the encoding efficiency. That is, it is possible to efficiently improve the spatial resolution of the partial image including the subject that is the object by using the reinforcement signal E _i .

  As in the first embodiment, the image enlargement unit 12 and the image cutout unit 13 included in the image encoding unit 1 may be configured as components of one processing unit. Specifically, as shown in FIG. 4, the image encoding unit 1 may include an image enlargement and cutout unit 18 instead of the image enlargement unit 12 and the image cutout unit 13.

[Example 2]
Next, Example 2 will be described. In the second embodiment, the image encoding device 100 uses the reduced image L of the image H as the object detection image to improve the temporal resolution of the image, generates the object region information D based on the object detection image, Perform image processing in the time direction. FIG. 6 is a diagram illustrating an example of an image processed in the time direction by the image encoding unit 1 according to the second embodiment. Hereinafter, a description will be given with reference to FIG.

  In the second embodiment, the object detection unit 2 of the image encoding device 100 receives a reduced image L that is an object detection image from the image encoding unit 1, and a predetermined object localized in the time direction from the reduced image L. And the area group including the object is output to the image encoding means 1 as object area information D. Specifically, the object detection unit 2 estimates a time interval in which an object exists in the reduced image L that is an object detection image. For example, the object detection means 2 calculates the difference between frames in the reduced image L (between adjacent frames having an interval of 2 frames in the example of FIG. 6), and the absolute value sum d (t) of the differences exceeds the threshold value. A frame is detected and object area information D is generated.

ここで、Δｔは、縮小画像Ｌにおける隣接フレーム間の時間間隔である。 First, the object detection means 2 calculates the difference between adjacent frames in the reduced image L by the following equation, and calculates the absolute value sum d (t) of the difference.

Here, Δt is a time interval between adjacent frames in the reduced image L.

オブジェクト検出手段２は、集合Ｔ内における各時刻ｔについて、時間的に互いに隣接する時刻同士をグループ化し、０個以上（Ｆ個、Ｆは０以上の整数）の領域Ｄ_ｉ（ｉは、０≦ｉ≦Ｆ−１を満たす整数のインデックス）に分割する。この領域Ｄ_ｉは、インデックスｉに対して一定の規則により整順されるものとする。例えば、時刻により整順されるようにしてもよいし、領域Ｄ_ｉ内のある代表時刻τ_ｉにおける絶対値和ｄ（ｔ_ｉ）により整順されるようにしてもよい。 Then, the object detection means 2 obtains a set T at time t when the absolute value sum d (t) exceeds the threshold value θ, as shown in the following equation.

For each time t in the set T, the object detection means 2 groups time adjacent to each other in time, and 0 or more (F, F is an integer of 0 or more) region D _i (i is 0). ≦ i ≦ F−1, which is an integer index). This region D _i is ordered according to a certain rule with respect to the index i. For example, the order may be arranged according to the time, or the order may be arranged according to the sum of absolute values d (t _i ) at a certain representative time τ _i in the region D _i .

また、領域Ｄ_ｉは、時刻順に整順された場合、以下の式で表される。

ここで、領域Ｄ_ｉは、式（３０）における各集合を時間方向に前後それぞれ一定時間拡張した集合、以下の式で表される。

尚、一定時間拡張した結果、複数の領域Ｄ_ｉが時間的に隣接するようになった場合には、これらを一つの集合に併合し、新たな領域Ｄ_ｉとして扱うようにしてもよい。 In the example of FIG. 6, the set T at time t when the absolute value sum d (t) exceeds the threshold θ is expressed by the following equation.

Further, the region D _i is expressed by the following expression when arranged in order of time.

Here, the region _Di is a set obtained by extending each set in Expression (30) for a certain period of time in the time direction, and is expressed by the following expression.

When a plurality of regions D _i are temporally adjacent as a result of expansion for a certain time, they may be merged into one set and treated as a new region D _i .

Object detection means 2 outputs a region D _i thus determined as the object area information D to the image encoding unit 1. Here, the object area information D output by the object detection means 2 is a time information group of a frame including the detected object, and is information on the same time scale as the image H.

  In the second embodiment, the image encoding unit 1 of the image encoding apparatus 100 includes the same unit as the image encoding unit 1 shown in FIG. The image reduction means 10 receives the image H, and in order to reduce the temporal resolution of the image H, the temporal reduction frame in the image H is thinned out and reduced, and a reduced image L having a low temporal resolution is generated and reduced image coding is performed. In addition to outputting to the means 11 and the image enlarging means 12, it outputs to the object detection means 2 as an object detection image. Specifically, the image reduction means 10 generates a reduced image L with low time resolution by sub-sampling of the image H by sampling only frames at a fixed time interval. Note that the image reduction means 10 may smooth the image information of a plurality of frames with respect to the image H and generate the reduced image L.

  In the example of FIG. 6, the image reduction means 10 has an odd time in the time direction with respect to the frames (t = 0, 1, 2, 3, 4, 5, 6, 7,...) Constituting the image H. The frame is thinned out and reduced to generate a reduced image L having a low temporal resolution composed of even-numbered frames (t = 0, 2, 4, 6,...).

ここで、ΔＴは、縮小画像Ｌが存在する時間間隔（サンプリング間隔）であり、ωは、縮小画像Ｌが存在する時刻の位相である。図６の例では、ΔＴ＝２、ω＝０である。 The image enlarging means 12 receives the reduced image L from the image reducing means 10, performs interpolation processing on the reduced image L in the time direction, and stores frames until the image H has the same size (the same frame rate as the image H). The image is inserted and enlarged, and an enlarged image K is generated and output to the image cutout unit 13. Specifically, the image enlarging means 12 performs an interpolation process on the reduced image L by zero-order extrapolation from the previous frame, or an interpolation process by primary extrapolation using two adjacent frames. To generate an enlarged image K. Interpolation processing by zero-order extrapolation is performed according to the following equation.

Here, ΔT is the time interval (sampling interval) in which the reduced image L exists, and ω is the phase of the time at which the reduced image L exists. In the example of FIG. 6, ΔT = 2 and ω = 0.

  In the example of FIG. 6, the image enlarging means 12 interpolates the odd time frames in the time direction with respect to the even time frames (t = 0, 2, 4, 6,...) Constituting the reduced image L. , An enlarged image K composed of frames at odd and even times (t = 0, 1, 2, 3, 4, 5, 6, 7,...) Is generated.

The image cutout means 13 receives the enlarged image K from the image enlargement means 12, receives the object area information D from the object detection means 2, and the time information indicated by the object area information D from the frames constituting the enlarged image K. cut out of frame cutout as the image B _i, and outputs the clipped image B _i in the difference calculation unit 15. The object area information D may be a plurality of aggregate strings (F aggregate strings), and F is an integer of 0 or more. In this case, the cut-out image B _i output by the image cut-out means 13 corresponds to each area D _i (i∈ {0,..., F−1}) included in the object area information D. When F = 0, the cut-out image _Bi is not output. That is, the image cutout unit 13, all of the index i = 0, · · ·, I cut out cut-out image _{B i} relative to F-1, an image sequence of the clipped image _{(B i) i = 0,} ···, F _-1 is output.

In the example of FIG. 6, the image cutout unit 13 calculates the expression (31) from the frames (t = 0, 1, 2, 3, 4, 5, 6, 7,...) Constituting the enlarged image K. The frame of the object area information D (D ₀ = {3, 4, 5}) shown in FIG. 6 is cut out as a cutout image B _i (B ₀ ), and the frame of t = 3, 4, 5 in the enlarged image K is cut out. _i (B ₀ ) is generated as a frame of t = −1, 0, +1.

The image cutout unit 14 inputs the image H, inputs the object region information D from the object detection unit 2, and cuts out the frame of time information indicated by the object region information D from the frames constituting the image H, and outputs the image C _i. And the cut-out image C _i is output to the difference calculation means 15. The cut-out image C _i output by the image cut-out means 14 corresponds to each area D _i (i∈ {0,..., F−1}) included in the object area information D. When F = 0, the cut-out image C _i is not output. That is, the image cutout unit 14 cuts out the cutout image C _i for all indexes i = 0,..., F−1, and the cutout image sequence (C _i ) _{i = 0,. -1} is output.

In the example of FIG. 6, the image cutout unit 14 calculates the expression (31) from the frames (t = 0, 1, 2, 3, 4, 5, 6, 7,...) That form the image H. A frame of the object area information D (D ₀ = {3, 4, 5}) to be shown is cut out as a cut-out image C _i (C ₀ ), and a frame of t = 3, 4, 5 in the image H is cut out as a cut-out image C _i ( C ₀ ) is generated as a frame of t = −1, 0, +1.

The difference calculation means 15 inputs the cut-out image B _i from the image cut-out means 13, inputs the cut-out image C _i from the image cut-out means 14, and calculates the difference image E _i between the cut-out image B _i and the cut-out image C _i. The difference image E _i is calculated and outputted to the reinforcement signal encoding means 16 as the reinforcement signal E _i . For example, the difference calculation unit 15 performs the normal subtraction between the cut image B _i and the clipped image C _i, to generate a difference image E _i. Or, difference calculation means 15, with the cut image B _i and the clipped image C _i, and calculates the difference modulo the number of gradations of the pixel value to generate a difference image E _i. Here, the difference calculation means 15 calculates the difference image E _i for all the indexes i = 0,..., F−1, and the difference image image sequence (E _i ) _{i = 0,. , F-1} is output. In the case of F = 0, the difference calculation means 15 does not output the difference image E _{i because} it does not input the cut images B _i and C _i from the image cut-out means 13 and 14.

In the example of FIG. 6, the difference calculation unit 15 includes a cutout image B _i composed of t = −1, 0, +1 frames and a cutout image C _i composed of t = −1, 0, +1 frames. Thus, a difference image E _i composed of frames of t = −1, 0, +1 is generated.

  The reduced image encoding unit 11, the reinforcement signal encoding unit 16, and the multiplexing unit 17 of the second embodiment perform the same processing as each unit included in the image encoding unit 1 according to the first embodiment shown in FIG. The description is omitted here.

As described above, according to the image encoding device 100 including the image encoding means 1 of the second embodiment, the object detection means 2 is based on the object detection image that is the reduced image L with low time resolution. Object region information D (information on a time region including a predetermined object) that is determined to require transmission of a fine image with high time resolution is generated, and the difference calculation unit 15 of the image encoding unit 1 The reinforcing signal E _i for increasing the resolution of the time domain indicated by the information D is generated by calculating the difference between the image H and the enlarged image K in which the reduced image L is enlarged, and the multiplexing means 17 Is generated from the reduced image L by the reduced image encoding means 11 and the code sequence λ of the image having a low time resolution and the amount of time generated from the reinforcement signal E _i by the reinforcement signal encoding means 16. The code sequence ε for generating a high-resolution image is multiplexed and the code sequence μ is output to the image decoding apparatus 200. As a result, the image decoding apparatus 200 uses the same means as the object detection means 2 included in the image encoding apparatus 100, so that the object region information is obtained from the reduced image L obtained by decoding the code sequence λ of the image with low time resolution. D can be generated. Further, for the time domain indicated by the object area information D, it is possible to generate a high-definition and high-resolution image that reproduces the smooth movement of the subject that is the object from the reinforcement signal E _{i obtained} by decoding the code sequence ε. it can.

Further, according to the image encoding device 100 including the image encoding unit 1 of the second embodiment, the signal encoded by the reinforcement signal encoding unit 16 of the image encoding unit 1 is an object as in the first embodiment. Since only the reinforcement signal E _i for the time domain indicated by the area information D is necessary, and the reinforcement signal E _i is generated by the difference calculation of the clipped images B _i and C _{i in} the difference calculation means 15, the code amount is reduced. Can do. Further, since the code string output from the image encoding device 100 to the image decoding device 200 does not include the object area information D related to the arrangement of the objects, the code amount can be reduced. By reducing the amount of codes, it is possible to improve the encoding efficiency. That is, it is possible to efficiently improve the time resolution of an image including a subject that is an object using the reinforcement signal E _i .

As in the first embodiment, the image enlargement unit 12 and the image cutout unit 13 included in the image encoding unit 1 may be configured as components of one processing unit. Specifically, as shown in FIG. 2, the image encoding unit 1 includes an image enlargement and cutout unit 18 instead of the image enlargement unit 12 and the image cutout unit 13. In this case, the image enlarging / cutting unit 18 inputs the reduced image L from the image reducing unit 10, inputs the object area information D from the object detecting unit 2, and each area D _i (i = 0) included in the object area information D. ,..., F-1), considering the reduction ratio when the reduced image L is generated by thinning a predetermined frame from the image H in the image reducing means 10, and the frames of the respective regions D _i with the same reduction ratio. Decrease the size. Then, the image enlargement / cutout unit 18 cuts out a cutout image from the reduced image L based on the time information of the reduced area D _i , and considers an enlargement ratio when the image enlargement unit 12 inserts a frame to enlarge the image. Then, the frame is inserted into the cutout image at the same enlargement ratio to enlarge the image, and the cutout image _Bi is generated. Thereby, the image encoding means 1 does not generate the enlarged image K, but enlarges only the clipped image. Therefore, the calculation amount and the memory amount can be reduced, and the reinforcement signal E _i can be used for the object. It is possible to improve the time resolution of an image including a certain subject more efficiently.

[Modification of Example 2]
Next, a modification of the second embodiment will be described. In the modification of the second embodiment, the image encoding device 100 generates a locally decoded image L ′ by decoding the code sequence of the reduced image L of the image H in order to improve the temporal resolution of the image, and generates the locally decoded image. L ′ is an object detection image, object area information D is generated based on the object detection image, and image processing in the time direction is performed. FIG. 7 is a diagram illustrating an example of an image processed in the time direction by the image encoding unit 1 according to a modification of the second embodiment. Hereinafter, a description will be given with reference to FIG.

  In the modification of the second embodiment, the object detection unit 2 of the image encoding device 100 inputs the local decoded image L ′ instead of the reduced image L as the object detection image, and the object detection unit 2 of the second embodiment Similar processing is performed. That is, the object detection means 2 receives the local decoded image L ′, which is an object detection image, from the image encoding means 1, detects a predetermined object localized in the time direction from the local decoded image L ′, and The area group including the object is output as object area information D to the image clipping means 13 and 14 of the image encoding means 1.

  In the modification of the second embodiment, the image encoding unit 1 of the image encoding device 100 includes the same unit as the image encoding unit 1 shown in FIG. The same and different points in configuration between the image encoding unit 1 according to the second embodiment and the image encoding unit 1 according to the modification of the second embodiment are between the first embodiment and the modification of the first embodiment. This is the same as that described in the comparison.

  In the modification of the second embodiment, the local image decoding unit 19 receives the code sequence λ of the reduced image L with low temporal resolution from the reduced image encoding unit 11, decodes the code sequence λ, and generates the local decoded image L ′. The generated image is output to the image enlarging unit 12 and is output to the object detecting unit 2 as an object detection image. The image enlarging unit 12 receives the local decoded image L ′ from the local image decoding unit 19, performs interpolation processing on the local decoded image L ′ in the time direction, and has the same size as the image H (the same frame rate as the image H). A frame is inserted and enlarged until the image becomes K, and an enlarged image K is generated and output to the image cutout means 13.

As described above, according to the image encoding device 100 including the image encoding unit 1 according to the modification of the second embodiment, the local image decoding unit 19 of the image encoding unit 1 uses the reduced image L with low time resolution. The code string λ is decoded to generate a local decoded image L ′, and the object detection unit 2 needs to transmit a high-definition image with high time resolution based on the object detection image that is the local decoded image L ′. Object region information D determined to be present (time region information including a predetermined object) is generated, and the difference calculation unit 15 of the image encoding unit 1 reinforces the reinforcement signal E in the frame of the time region indicated by the object region information D. the _i, and the image H, generated by calculating the difference between the enlarged image K which local decoded image L 'is enlarged, the multiplexing means 17, or the reduced image L by the reduced image coding unit 11 And code string generated time resolution of a low image lambda, generated from helper signal E _i by the reinforcing signal encoding unit 16, and a code string ε for generating an image with high time resolution and multiplexed code string μ is output to the image decoding apparatus 200. As a result, the image decoding apparatus 200 uses the same means as the object detection means 2 included in the image encoding apparatus 100, so that the object region information is obtained from the reduced image L obtained by decoding the code sequence λ of the image with low time resolution. D can be generated. Further, for the time domain indicated by the object area information D, it is possible to generate a high-definition and high-resolution image that reproduces the smooth movement of the subject that is the object from the reinforcement signal E _{i obtained} by decoding the code sequence ε. it can.

The image encoding unit 1 according to the modification of the second embodiment generates the object region information D by using the local decoded image L ′ generated by the local image decoding unit 19 as the object detection image, and the reinforcement signal E _i. Is generated to generate a code string ε. On the other hand, the image encoding unit 1 according to the second embodiment uses the reduced image L generated by the image reducing unit 10 as an object detection image, thereby generating object region information D and generating a reinforcement signal E _i. A code string ε is generated. In this case, the image decoding apparatus 200 generates object region information D and generates a sharpened image H by using a reduced image L generated by a reduced image decoding unit 22 described later as an object detection image. Become. Here, the reduced image L generated by the image reducing unit 10 of the second embodiment corresponds to the reduced image L generated by the reduced image decoding unit 22 of the image decoding apparatus 200, but a modified example of the second embodiment. The local decoded image L ′ generated by the local image decoding means 19 is closer to the reduced image L generated by the reduced image decoding means 22 of the image decoding device 200 than the reduced image L of the second embodiment. . Therefore, even when the reduced image encoding unit 11 uses an irreversible encoding method, the image encoding device 100 and the image decoding device 200 can generate the same object region information D. That is, even if the image quality is deteriorated due to the encoding of the reduced image encoding unit 11, the object detection does not fail. Further, since the reinforcement signal E _i is generated by the enlarged image K generated using the local decoded image L ′, even if the reduced image encoding unit 11 uses an irreversible encoding method, The signal takes into account distortion of image quality degradation due to encoding. That is, the image decoding apparatus 200 can achieve high resolution of the image by using such a reinforcement signal E _i . That is, the image encoding means 1 according to the modification of the second embodiment generates the reinforcement signal E _i in consideration of the decoding process of the image decoding apparatus 200 and generates the code string ε. A sharpened image H that further reflects the image H input to the encoding device 100 can be generated.

Further, according to the image encoding device 100 including the image encoding means 1 of the modification of the second embodiment, the signal encoded by the reinforcement signal encoding means 16 of the image encoding means 1 as in the second embodiment. Is only the reinforcement signal E _i for the time domain indicated by the object area information D, and the reinforcement signal E _i is generated by the difference calculation of the partial images B _i and C _{i in} the difference calculation means 15. The code amount of the code string ε generated from _i can be reduced. Further, since the code string output from the image encoding device 100 to the image decoding device 200 does not include the object area information D related to the arrangement of the objects, the code amount can be reduced. By reducing the amount of codes, it is possible to improve the encoding efficiency. That is, it is possible to efficiently improve the time resolution of an image including a subject that is an object using the reinforcement signal E _i .

  As in the second embodiment, the image enlargement unit 12 and the image cutout unit 13 included in the image encoding unit 1 may be configured as components of one processing unit. Specifically, as shown in FIG. 4, the image encoding unit 1 may include an image enlargement and cutout unit 18 instead of the image enlargement unit 12 and the image cutout unit 13.

As described above, the image encoding apparatus 100 according to the embodiment of the present invention has been described separately in the first embodiment, the modified example of the first example, the modified example of the second example, and the modified example of the second example. You may make it combine the modification of Example 1 or Example 1, and the modification of Example 2 or Example 2. FIG. That is, in another embodiment, image processing in the spatial direction and the temporal direction is performed in order to improve the spatial resolution and temporal resolution of the image. Specifically, the object detection means 2 generates object region information D of a spatial region and a time region including the object based on the reduced image L or the local decoded image L ′. Further, the image reduction means 10 of the image encoding means 1 generates a reduced image L with low spatial resolution and temporal resolution, and the image cutout means 13, 14 are the frames of the time domain indicated by the object area information D. In contrast, the image of the spatial area indicated by the object area information D is generated as cut-out images B _i and C _i . Then, the reinforcement signal encoding unit 16 inputs the difference image E _i (reinforcement signal E _i ) of the spatial region and the time region including the object from the difference calculation unit 15, performs encoding, and encodes the spatial region and the time including the object. A code sequence ε of the region reinforcement signal E _i is generated. Thereby, the image decoding apparatus 200 can generate a high-definition spatial resolution and a high temporal resolution image. In addition, it is possible to efficiently improve the spatial resolution and temporal resolution of an image including a subject that is an object by using the reinforcement signal E _i .

[Image decoding device]
Next, an image decoding apparatus according to an embodiment of the present invention will be described. FIG. 8 is a block diagram illustrating a configuration of the image decoding apparatus. The image decoding apparatus 200 includes an image decoding unit 3 and an object detection unit 4. The image decoding apparatus 200 receives the code string μ output from the image encoding apparatus 100 shown in FIG. 1, generates a low-resolution image and a high-definition sharpened image H by image decoding processing and object detection processing, These images are output.

The image decoding unit 3 receives the code string μ from the image encoding device 100, receives the object area information D from the object detection unit 4, performs image decoding based on the code string μ and the object area information D, A resolution image and a sharpened image H are generated and output. The image decoding unit 3 decodes the code string μ, generates an image (object detection image) used by the object detection unit 4 to detect the object, and outputs the object detection image to the object detection unit 4. . The image decoding unit 3 may not output a low resolution image. The code sequence μ input from the image encoding device 100 is a multiplexed code sequence λ in which the reduced image L is encoded and a code sequence ε in which the reinforcement signal E _i is encoded.

  The object detection unit 4 receives the object detection image from the image decoding unit 3, detects the object from the object detection image, and determines whether it is necessary to transmit a high-definition image. Are generated as the object area information D and output to the image decoding means 3. The object detection unit 4 performs the same processing as the object detection unit 2 of the image encoding device 100 shown in FIG.

  Hereinafter, as in the case of the image encoding device 100 shown in FIG. 1, the case where the image is a monochrome image will be described as an example. The value will be handled. Therefore, the image is not limited to a monochrome image but can be applied to a color image or the like.

  Hereinafter, with respect to the image decoding unit 3 of the image decoding apparatus 200 shown in FIG. 8, image decoding for performing image processing in the spatial direction corresponding to the image encoding unit 1 according to the first embodiment and the modified example of the first embodiment described above. The description will be given separately for the means 3 and the image decoding means 3 for performing the image processing in the time direction corresponding to the image encoding means 1 according to the second embodiment and the modification of the second embodiment.

[Image Decoding Unit Corresponding to Embodiment 1 and Modification of Embodiment 1]
First, an image decoding unit 3 that performs image processing in the spatial direction corresponding to the image encoding unit 1 according to the first embodiment and the modification of the first embodiment will be described. The image decoding means 3 uses the reduced image L obtained by decoding the code string μ as an object detection image, causes the object detection means 4 to generate object region information D, performs image processing in the spatial direction, and obtains the code string μ. Using the reinforcement signal E _i obtained by decoding, a sharpened image H in which the resolution of the spatial area indicated by the object area information D is improved is generated.

  FIG. 9 is a block diagram showing the configuration of the image decoding means 3. The image decoding unit 3 includes a demultiplexing unit 20, a reinforcement signal decoding unit 21, a reduced image decoding unit 22, an arrangement unit 23, an image enlargement unit 24, and an addition unit 25. Hereinafter, the coordinates of the horizontal position x and the vertical position y of the image are (x, y), and the pixel value of the image I at the coordinates (x, y) is I (x, y).

(Demultiplexing means)
The demultiplexing means 20 receives the code string μ from the image coding apparatus 100, and in the preset order, the code string λ obtained by coding the reduced image L and the code obtained by coding the reinforcement signal E _i. The code sequence λ is output to the reduced image decoding unit 22, and the code sequence ε is output to the reinforcement signal decoding unit 21.

  In addition, when the positions of the code sequence λ and the code sequence ε are described in additional information such as header information or footer information in the code sequence μ, the demultiplexing unit 20 uses the additional information based on the additional information. , The code string μ is separated into a code string λ and a code string ε.

(Reinforcing signal decoding means)
The reinforcement signal decoding unit 21 receives the code sequence ε obtained by encoding the reinforcement signal E _i from the demultiplexing unit 20, decodes the code sequence ε, generates the reinforcement signal E _i, and outputs it to the arrangement unit 23. To do. The reinforcement signal decoding unit 21 is a decoding unit that generates a code string ε and is paired with the reinforcement signal encoding unit 16 in the image encoding unit 1 of the image encoding device 100.

Specifically, the reinforcement signal decoding unit 21 decomposes the code sequence ε into partial code sequences for each pixel when the reinforcement signal encoding unit 16 uses a PCM encoding method, and the decomposed partial code sequence is dequantized. The luminance value is converted into a luminance value, which is decoded by mapping it to each pixel position of the reinforcement signal E _{i in} the order of raster scanning, and the reinforcement signal E _i is generated.

Further, the reinforcement signal encoding means 16 uses JPEG, JPEG 2000, MPEG-1, MPEG-2, MPEG-4, MPEG-4 AVC / H. When an existing image coding method or moving image coding method such as H.264 is used, the reinforcement signal decoding unit 21 generates the reinforcement signal E _i using a decoding method paired with the coding method.

Here, the reinforcement signal E _i generated by the reinforcement signal decoding means 21 is a signal for reinforcing the spatial area indicated by the object area information D and generating a high resolution (high spatial resolution) image. Hereinafter, with respect to the i-th (i = 0,..., F-1 (F is the number of objects)) index, the representative point (for example, the center of gravity, the pixel with the smallest x coordinate) in the region including the i-th object Let E _i (x, y) be the reinforcement signal at coordinates (x, y) when the origin is the image coordinate of the pixel with the smallest y coordinate in the group.

(Reduced image decoding means)
The reduced image decoding unit 22 receives the code sequence λ obtained by encoding the reduced image L from the demultiplexing unit 20, generates the reduced image L by decoding the code sequence λ, and outputs the reduced image L to the image enlargement unit 24. At the same time, the image is output to the object detection means 4 as an object detection image. The reduced image decoding unit 22 is a decoding unit that generates the code string λ and is paired with the reduced image encoding unit 11 in the image encoding unit 1 of the image encoding device 100.

  Specifically, when the reduced image encoding unit 11 uses a PCM encoding method, the reduced image decoding unit 22 decomposes the code sequence λ into partial code sequences for each pixel, and converts the decomposed partial code sequence into an inverse quantum. The luminance value is converted into a luminance value, which is decoded by mapping to each pixel position of the reduced image L in the order of raster scan, and the reduced image L is generated.

  Further, the reduced image encoding means 11 uses JPEG, JPEG 2000, MPEG-1, MPEG-2, MPEG-4, MPEG-4 AVC / H. When an existing image encoding method or moving image encoding method such as H.264 is used, the reduced image decoding unit 22 generates a reduced image L using a decoding method that is paired with the encoding method.

(Arrangement means)
The placement means 23 receives the reinforcement signal E _i from the reinforcement signal decoding means 21, receives the object area information D from the object detection means 4, and superimposes the reinforcement signal E _i based on the spatial area indicated by the object area information D. A position to be determined is determined, and a reinforcing image G is generated and output to the adding means 25. Specifically, the arrangement unit 23 includes a frame memory having the size (number of pixels and aspect ratio) of the sharpened image H to be output by the image decoding unit 3, and after determining the superposition position of the reinforcement signal E _i The reinforcement signal E _i is stored at the address position of the frame memory corresponding to the superposition position, and the image on the frame memory is generated as the reinforcement image G.

For example, the placement unit 23 generates the reinforcement image G by storing the pixel value of the reinforcement signal E _i at the address of the frame memory corresponding to the overlapping position. However, when the regions of the plurality of reinforcement signals E _i overlap, the arrangement unit 23 calculates a representative value (for example, an average value) of the pixel values in the region, and uses the pixel value of the calculation result as the pixel value of the region. Use.

For example, the region _{D i} included in the object area information D is bounding box (upper left vertex of the image coordinates _{(p i, q} i), the image coordinates of the lower right apex _(r i, _{s i)} and information) represented and, the reinforcing signal _{E i} is the difference calculation unit 15 in the image coding unit 1 of the image coding apparatus 100, the partial image _B i, the difference between _{C i} or the partial image _B i,, gradation of _{C i} When the difference is calculated as a modulo difference, the arrangement unit 23 generates the reinforcement image G by the following equation.

(Image enlargement means)
The image enlarging means 24 receives the reduced image L from the reduced image decoding means 22, and enlarges the reduced image L by inserting pixels until it reaches a predetermined size (for example, the size of the sharpened image H). An image K is generated and output to the adding means 25. Since the image enlarging means 24 performs the same processing as the image enlarging means 12 of the image encoding means 1 according to the first embodiment shown in FIGS. 2 and 4 and the modification of the first embodiment, the description thereof is omitted here.

(Addition means)
The adding means 25 inputs the reinforcing image G from the arranging means 23, inputs the enlarged image K from the image enlarging means 24, adds the reinforcing image G and the enlarged image K as shown in the following formula, and sharpens it. An image H is generated and output.

(Example of image processed by image decoding means corresponding to the first embodiment and the modification of the first embodiment)
FIG. 10 is a diagram illustrating an example of an image processed in the spatial direction by the image decoding unit 3 according to the first embodiment and a modification of the first embodiment. The enlarged image K is generated from the reduced image L by the image enlargement unit 24, and the areas D ₀ and D ₁ included in the object area information D are generated from the object detection image that is the reduced image L by the object detection unit 4. . Further, the reinforcement image G is generated from the reinforcement signals E ₀ and E ₁ and the areas D ₀ and D ₁ included in the object area information D by the arrangement unit 23, and the sharpened image H is enlarged by the addition unit 25. And the reinforcing image G. In the sharpened image H, as a result of the addition of the enlarged image K that receives a blurred impression in which no broad spectrum component exists and the reinforcing image G, the region indicated by the object region information D becomes a high-resolution image.

As described above, according to the image decoding device 200 including the image decoding means 3 that performs image processing in the spatial direction in accordance with the first embodiment and the modification of the first embodiment, The column μ is input, and the object detection unit 4 is configured to select object region information D (predetermined object is stored on the basis of the object detection image that is a reduced resolution L of the low resolution decoded from the code sequence λ included in the code sequence μ. (Information of the area to be included) was generated. Further, the arrangement unit 23 of the image decoding unit 3 arranges the reinforcement signal E _i decoded from the code sequence ε included in the code sequence μ at the position of the spatial region indicated by the object region information D, and generates a high-resolution image. A reinforcing image G is generated, and the adding unit 25 adds the enlarged image K obtained by enlarging the reduced image L by the image enlarging unit 24 and the reinforcing image G to generate a sharpened image H. . As a result, the sharpened image H becomes a high-definition high-resolution image that reproduces the smooth outline and detailed texture of the subject that is the object in the spatial region of the object region information D. The code sequence μ input from the image encoding device 100 is the code sequence λ of the low-resolution image and the code sequence ε of the reinforcement signal E _i in the spatial region indicated by the object region information D, and the object region related to the object arrangement Since the information D is not included, the code amount is small, and the burden of the decoding process can be reduced. Therefore, it is possible to efficiently improve the resolution of the partial image including the subject that is the object by using the reinforcement signal E _i .

In the image decoding unit 3 shown in FIG. 9, the arrangement unit 23 and the addition unit 25 have been described as components of separate processing units, but these may be configured as components of one processing unit. Good. Specifically, as shown in FIG. 9, the image decoding unit 3 includes an arrangement addition unit 26 instead of the arrangement unit 23 and the addition unit 25. The arrangement adding means 26 receives the enlarged image K from the image enlarging means 24, receives the reinforcement signal E _i from the reinforcement signal decoding means 21, receives the object area information D from the object detection means 4, and receives the object area information D based on the spatial area shown, to determine the position to be added reinforcement signal E _i, to the determination position of the enlarged image K image, by adding the helper signal E _i of the determined position, the enlarged image after addition K is output as a sharpened image H. As a result, the image decoding unit 3 directly generates the sharpened image H without generating the reinforcing image G, so that the calculation amount and the memory amount are small, and the burden of decoding processing is further reduced. Can do. Therefore, it is possible to improve the resolution of the partial image including the subject as the object more efficiently by using the reinforcement signal E _i .

[Image Decoding Unit Corresponding to Second Embodiment and Modified Example of Second Embodiment]
Next, an image decoding unit 3 that performs image processing in the time direction corresponding to the image encoding unit 1 according to the second embodiment and the modification of the second embodiment will be described. The image decoding means 3 uses the reduced image L obtained by decoding the code string μ as an object detection image, causes the object detection means 4 to generate object region information D, performs image processing in the time direction, and obtains the code string μ. Using the reinforcement signal E _i obtained by decoding, a sharpened image H with improved time domain resolution indicated by the object region information D is generated.

  The image decoding means 3 corresponding to the second embodiment and the modification of the second embodiment is configured to include the same means as the image decoding means 3 shown in FIG. The demultiplexing unit 20, the reinforcement signal decoding unit 21, and the reduced image decoding unit 22 perform the same processing as each unit in the image decoding unit 3 corresponding to the modification of the first embodiment and the first embodiment, and will be described here. Is omitted. Hereinafter, the time is t, and the pixel value at the time t of the image I and the image coordinates (x, y) is I (t, x, y).

(Image enlargement means)
The image enlarging means 24 receives the reduced image L from the reduced image decoding means 22, performs interpolation processing on the reduced image L in the time direction, and has a predetermined size (for example, the same frame rate as the sharpened image H). The frame is inserted and enlarged to generate an enlarged image K and output to the adding means 25. The image enlarging means 24 performs the same processing as the image enlarging means 12 of the image encoding means 1 according to the second embodiment and the modification of the second embodiment shown in FIGS.

(Arrangement means)
The placement unit 23 receives the reinforcement signal E _i from the reinforcement signal decoding unit 21, receives the object region information D from the object detection unit 4, and obtains the representative time τ _i from the region D _i included in the object region information D. For example, from the time in the region D _i , the minimum time, the predetermined time from the smallest, the central time, the maximum time, the average time, and the like are obtained as the representative time τ _i .

但し、Ｅ_ｉ（ｔ，ｘ，ｙ）の値が定義されない時刻ｔに対しては、Ｅ_ｉ（ｔ，ｘ，ｙ）＝０とする。 Then, the placement unit 23 determines the frame position where the reinforcement signal E _i is to be placed according to the following formula, places the reinforcement signal E _i while adjusting the position (time) of the frame, and generates the reinforcement image G. And output to the adding means 25.

_However, for the E i (t, x, y ) value of no defined time _t, and E i (t, x, y ) = 0.

(Addition means)
The adding means 25 inputs the reinforcing image G from the arranging means 23, inputs the enlarged image K from the image enlarging means 24, adds the reinforcing image G and the enlarged image K by the following formula, and obtains the sharpened image H. Generate and output.

The adding means 25 may calculate the addition of the remainder modulo the number of gradations of the reinforcing image G and the enlarged image K by the following formula and generate the calculation result as a sharpened image H. .

(Example of image processed by image decoding means corresponding to the second embodiment and a modification of the second embodiment)
FIG. 11 is a diagram illustrating an example of an image processed in the time direction by the image decoding unit 3 according to the second embodiment and a modification of the second embodiment.

First, the processing of the object detection unit 4 in FIG. 11 will be described. As described above, the object detection unit 4 detects a predetermined object localized in the time direction from the reduced image L, which is an object detection image, and generates a group of regions including the object as object region information D. Specifically, the object detection unit 4 estimates a time interval in which an object exists in the reduced image L that is an object detection image. For example, the object detection unit 4 calculates a difference between frames in the reduced image L, calculates an absolute value sum d (t) of the differences, and a set of times t at which the absolute value sum d (t) exceeds the threshold θ. T is obtained, and object area information D is generated with the set T as a sub-area D _i . The absolute value sum d (t) of the differences is calculated by the above equation (27), and the set T is obtained by the above equation (28).

また、部分集合の領域Ｄ_ｉは、時刻順に整順された場合、以下の式で表される。

ここで、部分集合の領域Ｄ_ｉは、式（３９）における各集合を時間方向に前後それぞれ一定時間拡張した集合、以下の式で表される。

尚、一定時間拡張した結果、複数の部分集合の領域Ｄ_ｉが時間的に隣接するようになった場合には、これらを一つの集合に併合し、新たな部分集合の領域Ｄ_ｉとして扱うようにしてもよい。 In the example of FIG. 11, the set T at time t when the absolute value sum d (t) exceeds the threshold θ is expressed by the following equation.

Further, the subset region D _i is expressed by the following expression when arranged in order of time.

Here, the region D _i of the subset is a set obtained by extending each set in Formula (39) for a certain period of time in the time direction, and is expressed by the following formula.

When a plurality of subset areas D _i are temporally adjacent as a result of expansion for a certain period of time, they are merged into one set and treated as a new subset area D _i. It may be.

In FIG. 11, an enlarged image K is generated from the reduced image L by the image enlargement unit 24, and areas D ₀ and D ₁ included in the object area information D are object detection images that are reduced images L by the object detection unit 4. Generated from Further, the reinforcement image G is generated by the arrangement means 23 from the reinforcement signals E ₀ and E ₁ and the areas D ₀ and D ₁ included in the object area information D. Here, the representative times τ ₀ = 4, τ ₁ = 8 are obtained from the regions D ₀ and D ₁ included in the object region information D _n , and the reinforcement signal is transmitted to the frames of t = 3, 4, and 5 in the reinforcement image G. E ₀ is arranged, and the reinforcement signal E ₁ is arranged in the frames of t = 7, 8, and 9. The sharpened image H is generated from the enlarged image K and the reinforcing image G by the adding means 25. The sharpened image H becomes an image with a high time resolution due to the reinforcing image G arranged in the time region indicated by the object region information D.

As described above, according to the image decoding device 200 including the image decoding unit 3 that performs image processing in the time direction in accordance with the second embodiment and the modification of the second embodiment, The sequence μ is input, and the object detection unit 4 uses the object area information D (predetermined object) based on the object detection image which is the reduced image L with low temporal resolution decoded from the code sequence λ included in the code sequence μ. (Time domain information including). Further, the arrangement unit 23 of the image decoding unit 3 arranges the reinforcement signal E _i decoded from the code sequence ε included in the code sequence μ at the position of the time region indicated by the object region information D, and generates the reinforcement image G. Then, the adding unit 25 adds the enlarged image K obtained by enlarging the reduced image L by the image enlarging unit 24 and the reinforcing image G to generate the sharpened image H. As a result, the sharpened image H becomes an image with high time resolution that reproduces the smooth movement of the subject that is the object in the time region of the object region information D. The code sequence μ input from the image encoding device 100 is a code sequence λ of an image with low time resolution and a code sequence ε of the reinforcement signal E _i in the time domain indicated by the object area information D, and includes the object area information D. Therefore, the code amount is small, and the burden of decoding processing can be reduced. Therefore, it is possible to efficiently improve the time resolution of an image including a subject that is an object using the reinforcement signal E _i .

In the image decoding means 3 corresponding to the second embodiment and the modification of the second embodiment, the arranging means 23 and the adding means 25 have been described as components of separate processing units. You may make it be a component. Specifically, as shown in FIG. 9, the image decoding unit 3 includes an arrangement addition unit 26 instead of the arrangement unit 23 and the addition unit 25. The arrangement adding means 26 receives the enlarged image K from the image enlarging means 24, receives the reinforcement signal E _i from the reinforcement signal decoding means 21, receives the object area information D from the object detection means 4, and receives the object area information D Based on the time domain shown, the time position to which the reinforcement signal E _i is to be added is determined, the image at the determined position in the enlarged image K and the reinforcement signal E _{i at the} determined position are added, and the sharpened image H is generated and output. As a result, the image decoding means 3 directly generates the sharpened image H without generating the reinforcement image G, so that the calculation amount and the memory amount can be reduced, and the burden of decoding processing can be reduced. it can. Therefore, it is possible to more efficiently improve the time resolution of an image including a subject that is an object using the reinforcement signal E _i .

As described above, with respect to the image decoding apparatus 200 according to the embodiment of the present invention, the image decoding unit 3 corresponding to the modified example of Example 1 and Example 1, and the image decoding unit 3 corresponding to the modified example of Example 2 and Example 2. However, when the image coding apparatus 100 is configured by combining the modification example of the first embodiment or the first embodiment and the modification example of the second embodiment or the second embodiment, the image decoding apparatus 200 is described. Performs image processing in the spatial direction and the temporal direction, and generates a sharpened image H with improved spatial resolution and temporal resolution. Specifically, the object detection unit 4 generates the object area information D of the space area and the time area including the object based on the object detection image that is the reduced image L, and the arrangement unit 23 of the image decoding unit 3 , based on the time domain indicated by the object area information D, determines to be time position adds reinforcement signal E _i, based on the spatial area indicated by the object area information D, and spatial position to adding the reinforcing signal E _i Then, the reinforcing image G is generated. Then, the adding means 25 adds the enlarged image K obtained by enlarging the reduced image L and the reinforcement image G, and generates a sharpened image H. As a result, it is possible to more efficiently improve the spatial resolution and temporal resolution of an image including a subject that is an object using the reinforcement signal E _i .

The present invention has been described with reference to the embodiments. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the technical idea thereof. The image encoding device and the image decoding device according to the present invention transmit the entire image with a low-resolution code sequence, and transmit the code sequence of the reinforcement signal E _i in order to increase the resolution of the image including the object that is the object. For example, the present invention is also applicable to a mobile terminal having a small screen display as an image decoding device.

  For example, the portable terminal normally displays the reduced image L with low resolution decoded by the reduced image decoding means 22 on the display. Then, when the user performs an object display operation, the mobile terminal cuts out the image of the area including the object from the sharpened image H output by the adding unit 25, enlarges the image, and displays the image on the display. indicate. Specifically, the image decoding unit 3 of the portable terminal includes a display unit, and the arrangement unit 23 obtains the coordinate information of the spatial region including the object or the frame information of the time region including the object from the object region information D. Generate. The display means inputs a low resolution image, which is the reduced image L, from the reduced image decoding means 22 and displays it on the display. Then, the display unit inputs coordinate information or frame information from the arrangement unit 23, inputs a sharpened image H from the adding unit 25, and displays the region indicated by the coordinate information or frame information from the sharpened image H in accordance with a user operation. The image is cut out and enlarged, and displayed on the display. As a result, since the image of the area including the object is displayed with high resolution, it is possible to present an easy-to-view image that is followed by the user's operation by transmission with excellent encoding efficiency.

In addition, the image encoding device 100 transmits the code sequence μ obtained by multiplexing the code sequence λ of the reduced image L and the code sequence ε of the reinforcement signal E _i to the image decoding device 200 via the broadcast transmission path. Alternatively, it may be transmitted via a communication network such as the Internet.

Further, the image encoding unit 1 of the image encoding device 100 does not necessarily need to include the multiplexing unit 17. When the multiplexing unit 17 is not provided, the image encoding device 100 transmits the code sequence λ of the reduced image L and the code sequence ε of the reinforcement signal E _i to the image decoding device 200 via the broadcast transmission path independently. You may make it transmit, and you may make it transmit via a communication network. The image encoding device 100 transmits the code sequence λ of the reduced image L to the image decoding device 200 via the broadcast transmission path, and the image decoding device 200 transmits the code sequence ε of the reinforcement signal E _i via the communication network. The code sequence λ of the reduced image L may be transmitted to the image decoding device 200 via the communication network, and the code sequence ε of the reinforcement signal E _i may be transmitted via the broadcast transmission path. You may make it transmit to the decoding apparatus 200. FIG. Further, the image coding apparatus 100, to the code string λ of the reduced image L, and transmitted to the image decoding apparatus 200 via the high communication network reliability subjected to processing such as advanced error measures, the reinforcing signal E _i The code sequence ε may be transmitted to the image decoding apparatus 200 via a communication network with low reliability. Thereby, the amount of codes can be reduced, and the encoding efficiency can be improved. In addition, the image encoding device 100 transmits the code sequence λ of the reduced image L to the image decoding device 200 free of charge, and transmits the code sequence ε of the reinforcement signal E _{i to} the image decoding device 200 for a fee by charging processing. You may do it. When the image encoding device 100 independently transmits the code sequence λ of the reduced image L and the code sequence ε of the reinforcement signal E _i , the image decoding unit 3 of the image decoding device 200 includes the demultiplexing unit 20. There is no need.

  Note that a normal computer can be used as a hardware configuration of the image encoding device 100 and the image decoding device 200. The image encoding device 100 and the image decoding device 200 are configured by a computer including a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, an interface, and the like. The functions of the image encoding unit 1 and the object detection unit 2 provided in the image encoding device 100 are realized by causing the CPU to execute a program describing these functions. Each function of the image decoding unit 3 and the object detection unit 4 provided in the image decoding device 200 is realized by causing the CPU to execute a program describing these functions. These programs can be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

DESCRIPTION OF SYMBOLS 1 Image encoding means 2, 4 Object detection means 3 Image decoding means 10 Image reduction means 11 Reduced image encoding means 12, 24 Image enlargement means 13, 14 Image clipping means 15 Difference calculation means 16 Reinforcement signal encoding means 17 Multiplexing Means 18 Image enlargement and cutout means 19 Local image decoding means 20 Demultiplexing means 21 Reinforcement signal decoding means 22 Reduced image decoding means 23 Arrangement means 25 Addition means 26 Arrangement addition means 100 Image encoding apparatus 200 Image decoding apparatus

Claims (8)

In an image encoding apparatus for inputting an image, reducing the resolution of the image, encoding the image with the reduced resolution, and outputting a code string,
An object detection means for detecting a predetermined object included in the low-resolution image;
The input image is reduced in resolution and encoded into a first code string, and a reinforcement signal for increasing the resolution of the image is generated based on the image including the object detected by the object detection unit, Image encoding means for encoding a reinforcement signal into a second code string and outputting the first code string and the second code string;
The object detection means includes
Detecting a predetermined object included in the low-resolution image, and generating a spatial region including the object in the image as object region information;
The image encoding means includes
Image reduction means for thinning out pixels from the input image to generate a reduced image with reduced resolution, and outputting the reduced image to the object detection means as an object detection image for detecting the predetermined object A reduced image encoding unit that encodes the reduced image generated by the image reduction unit and generates a code string; and an image that generates an enlarged image by inserting pixels into the reduced image generated by the image reduction unit From the enlargement means, the first image cutout means for cutting out the image of the spatial area indicated by the object area information generated by the object detection means from the enlarged image generated by the image enlargement means, and the input image A second image for cutting out an image of the spatial area indicated by the object area information generated by the object detection means; A difference calculating means for calculating a difference between the cutout means, the image cut out by the first image cutout means, and the image cut out by the second image cutout means, and generating a reinforcement signal; A reinforcement signal encoding means for encoding the reinforcement signal generated by the difference calculation means and generating a code string;
While outputting the code string generated by the reduced image encoding means and the code string generated by the reinforcement signal encoding means,
The image encoding means further decodes the code string generated by the reduced image encoding means, generates a locally decoded image, and uses the locally decoded image for object detection for detecting the predetermined object. A local image decoding means for outputting to the object detection means as an image,
The image enlarging means enlarges the local decoded image generated by the local image decoding means to generate an enlarged image,
The image coding apparatus according to claim 1, wherein the object detection unit detects a predetermined object based on the object detection image output by the local image decoding unit and generates object region information. An image encoding apparatus that inputs an image, reduces the resolution of the image, encodes the reduced-resolution image, and outputs a code string, and includes a predetermined object included in the reduced-resolution image Object detection means for detecting, and for reducing the resolution of the input image and encoding it into a first code string, and for increasing the resolution of the image based on the image including the object detected by the object detection means Image encoding means for generating a reinforcement signal, encoding the reinforcement signal into a second code string, and outputting the first code string and the second code string; and the object detection means Detecting a predetermined object included in the low-resolution image, generating a spatial region including the object in the image as object region information, The image encoding means generates a reduced image with reduced resolution by thinning out pixels from the input image, and uses the reduced image as an object detection image for detecting the predetermined object. An image reduction means for outputting to the image, a reduced image generated by the image reduction means, a reduced image encoding means for generating a code string, and a pixel inserted in the reduced image generated by the image reduction means, An image enlarging means for generating an enlarged image; a first image clipping means for cutting out an image of a spatial area indicated by the object area information generated by the object detecting means from the enlarged image generated by the image enlarging means; From the input image, an image of the spatial area indicated by the object area information generated by the object detection means is obtained. A difference for calculating a difference between the second image cutout unit to be cut out, the image cut out by the first image cutout unit, and the image cut out by the second image cutout unit, and generating a reinforcement signal A code signal generated by the reduced image encoding means; and the reinforcement signal, comprising: a calculation means; and a reinforcement signal encoding means for encoding the reinforcement signal generated by the difference calculation means and generating a code string. A code sequence of a low resolution image and a code sequence of a reinforcing signal for obtaining a high resolution image are input from an image encoding device that outputs the code sequence generated by the encoding means , and the code sequence is decoded. In the image decoding device that outputs an image,
Object detection means for detecting a predetermined object included in the low resolution image using a low resolution image generated by decoding the code string;
The input code string is decoded, a low-resolution image and a reinforcement signal are generated, a reinforcement image is generated by arranging the reinforcement signal in a region where the object is detected, and the low-resolution image and the reinforcement image And an image decoding unit that outputs an image of the addition result. The image decoding device according to claim 2 ,
From the image encoding device, enter the code sequence,
The object detection means includes
A predetermined object included in the low-resolution image is detected, and a spatial region including the object in the image is generated as object region information;
The image decoding means includes
A reduced image decoding means for decoding a code string of the low resolution image and generating a reduced image;
Reinforcing signal decoding means for decoding the code string of the reinforcing signal and generating a reinforcing signal;
An image enlarging unit that inserts pixels into the reduced image generated by the reduced image decoding unit and generates an enlarged image;
Arrangement means for arranging a reinforcement signal generated by the reinforcement signal decoding means in a spatial region indicated by the object area information generated by the object detection means, and generating a reinforcement image;
An image decoding apparatus comprising: an addition unit that adds the enlarged image generated by the image enlargement unit and the reinforcement image generated by the arrangement unit and outputs an image after the addition. The image decoding device according to claim 3 ,
Instead of the arrangement means and the addition means, provided with an arrangement addition means,
The arrangement addition means adds the reinforcement signal generated by the reinforcement signal decoding means to the image of the spatial area indicated by the object area information generated by the object detection means in the enlarged image generated by the image enlargement means. And outputting an enlarged image after the addition. The image decoding device according to claim 3 ,
The object detection means generates F (F is an integer of 0 or more) areas Di (i = 0,..., F−1) each including the object as object area information,
The reinforcing signal decoding means decodes the code string of the reinforcing signal to generate F reinforcing signals Ei,
The arrangement means arranges the F reinforcement signals Ei generated by the reinforcement signal decoding means in F areas Di indicated by the object area information generated by the object detection means, and generates a reinforcement image. An image decoding apparatus characterized by that. The image decoding device according to claim 3 ,
Furthermore, a display means for displaying the reduced image generated by the reduced image decoding means on the screen,
The display means cuts out an image of an area indicated by the object area information generated by the object detection means from a post-addition image output by the addition means according to a user operation, and displays the image on a screen. An image decoding device. An image encoding program for causing a computer to function as the image encoding apparatus according to claim 1 . Image decoding program for a computer to function as the image decoding apparatus according to any one of claims 2 to 6.

Images