JP2005260464A - Image encoding device, image decoding device, image encoding method, image decoding method, image encoding program, image decoding program, image encoding program recording medium, and image decoding program recording medium - Google Patents

Abstract

Coding efficiency is improved in coding of a region at the end of the screen when the screen is divided into a plurality of regions, coding for dividing a high-resolution image into a plurality of regions, and the like.
A prediction mode search unit searches for a prediction mode in accordance with geometric information between a reference frame and a current frame from prediction modes corresponding to a plurality of patterns that define image information used when creating a prediction image. And decide. The predicted image creation unit 15 creates a predicted image using the geometric information according to the determined prediction mode, and the difference encoding unit 16 encodes the difference between the input image information and the predicted image. The prediction mode encoding unit 14 encodes the prediction mode.
[Selection] Figure 7

Description

  The present invention is a technique related to image coding of a plurality of frames using an inter-frame predictive coding method.

  In general, in moving picture coding, inter-frame prediction coding is used in order to achieve high coding efficiency using temporal correlation. The frame encoding mode can be predicted from an I frame that is encoded without using correlation between frames, a P frame that is predicted from one previously encoded frame, and two previously encoded frames. There are B frames.

  In the B frame, it is necessary to store the decoded image for two frames in the reference image memory. In particular, video encoding system H.264. H.263 and H.H. In H.264, decoded images of two or more frames are accumulated in a reference image memory, and a reference image can be selected and predicted from the reference image memory.

  A reference image can be selected for each block, and reference image designation information for designating a reference image is encoded. The reference image memory has a short time (STRM) and a long time (LTRM). The decoded image of the current frame is stored in the STRM, and the image stored in the STRM is selected in the LTRM. accumulate. As a document describing the control method of LTRM and STRM, for example, the following Non-Patent Document 1 can be cited.

  Reference image designation information includes H.264. A method using TR (Temporal Reference) such as H.263 Annex N, H.264 H.263 Annex U or H.264. As in H.264, a method for separately defining an index has been proposed.

  TR is MPEG-1, MPEG-2 and H.264. 261, H.M. This is information on the output time of each frame employed in international standard video encoding such as H.263, and is fixed-length encoded for each frame. A reference time interval is set in advance in the system, and the time from the beginning of the sequence is indicated by the product of the time interval and TR. The encoder sets time information of the input image to TR to encode each frame, and the decoder outputs the decoded image of each frame at the time specified by TR.

  When this TR is used, since the TR is generally encoded with a fixed length, it does not reflect the occurrence frequency of the reference image designation information, and the encoding efficiency is poor. In a method using an index separately, entropy encoding can be performed by reflecting the frequency of occurrence of reference image designation information by, for example, variable-length encoding the index.

  In particular, in the case of a still image, there is intra prediction encoding in which a prediction image is created using image information already decoded in the current frame and a residual signal is encoded.

  FIG. 18 is a diagram illustrating an example of pixel positions used for prediction in JPEG-LS. For example, in JPEG-LS, a predicted image is created from image information around an encoding target pixel as shown in FIG. In FIG. 18, the predicted image of pixel x is created from the decoded images of pixels a, b, c, and d.

  FIG. 2 is a diagram illustrating an example of intra prediction in H.264. H. In the H.264 intra prediction, a method of creating a predicted image from decoded images of surrounding blocks is employed. In such intra prediction, there are a plurality of modes for creating a predicted image, a mode with the highest coding efficiency is searched, and the mode information is also coded. 19, a, b,..., P indicate the pixels of the current block, and A, B,..., Q indicate already decoded pixels. The predicted image at the pixel position placed on the arrow shown in FIG. 19 is created from the image information of the pixels D, E, and F.

  FIG. 20 is a diagram illustrating an example of interpolation. Instead of creating a predicted image by extrapolation as shown in FIG. 19, as shown in FIG. 20, image information at pixel positions that have not yet been encoded is used and the image information is already encoded and decoded. An interpolation method for creating a predicted image by interpolation with existing image information has also been proposed (see, for example, Non-Patent Document 2). In this case, the image information at the uncoded position and the prediction method are searched and determined. In this case, the image information at the uncoded position, the prediction mode, and the prediction residual are encoded. Of course, it is also possible to create a prediction image from image information that has already been encoded and decoded for image information at a position that has not yet been encoded, and encode the prediction residual.

  The only difference between these interpolations and extrapolations is the predicted order. Basically, both are predicted using the image information in the current frame. In general, the interpolation efficiency is higher in the prediction efficiency because the image information used for the prediction can be selected more than in the interpolation interpolation.

  On the other hand, there are omnidirectional images that are shot using an ultra-wide-angle camera. As a method for encoding an omnidirectional video, a method has been proposed in which an omnidirectional video is once mapped onto the surface of a three-dimensional polygon, and further, an all-polygon plane is mapped to a rectangular two-dimensional image and encoded as a two-dimensional video. ing.

  FIG. 21 is a diagram illustrating an example of projection of an omnidirectional video onto an octahedron. Although the shape of the three-dimensional polygon is arbitrary, for example, Non-Patent Document 3 proposes a method using an octahedron as shown in FIG. From the omnidirectional image projected on the octahedron as shown in FIG. 21A, an image of the upper quadrangular pyramid and an image of the lower quadrangular pyramid as shown in FIG. 21B are created and rearranged. , An image in which an omnidirectional video as shown in FIG. 21C is mapped onto one rectangular screen is created.

  In these methods, an omnidirectional video is mapped to one rectangular screen, but may be mapped to a plurality of screens. For example, in the case of an octahedron, image information from the upper quadrangular pyramid may be handled as one screen, and image information from the lower quadrangular pyramid may be handled as one screen, and may be handled as a total of two screens of image information. In this way, it is possible to have a configuration having a plurality of images having a lower spatial resolution than the method of mapping an omnidirectional video on one screen. On the decoding side, if the screen has a low resolution, there is an advantage that the time for decoding the image information of the entire screen is reduced and the playback screen can be displayed more quickly.

  Such an image composed of a plurality of screens may be individually encoded for each screen, or each screen may be encoded as one video in association with each frame of one video. These videos may be encoded every other frame in one video. For each frame, in addition to the time information TR, category information indicating which video originally belonged is encoded.

  FIG. 22 is a diagram illustrating an example of projection of an omnidirectional video onto a cube. The polyhedron for projecting the omnidirectional video is not limited to the octahedron as shown in FIG. 21, but may be a cube as shown in FIG. Assume that there is a video S6 on the back side of the video S1, a video S5 on the back side of the video S2, and a video S4 on the back side of the video S3. When there is such a geometric information relationship, the video S1 to the video S6 may be collectively encoded as one video. These videos are encoded every other frame in one video. For each frame, in addition to the time information TR, category information indicating which video is originally from the video S1 to the video S6 is encoded.

  As for high-resolution image encoding, there is a method of encoding each divided region by dividing the screen into a plurality of regions.

  FIG. 23 is a diagram illustrating an example of screen division of a high resolution image. For example, as shown in FIG. 23, there is a method of dividing the screen into two in the vertical direction. This division method is used to shorten the computation time by providing a decoder for each area, for example, when the resolution is high and it takes a lot of computation time to decode, or to transmit encoded data This is used to recover from transmission errors.

Note that there are Non-Patent Document 4, Non-Patent Document 5, Non-Patent Document 6, and the like as documents describing techniques used in the embodiments of the present invention described later.
Thomas Wiegand, Xiaozheng Zhang, and Bernd Girod, “Long-Term Memory Motion-Compensated Prediction,” IEEE Transactions on Circuits and Systems for Video Technology, vol.9, no.1, pp.70-84, Feb.1999 Hatori: “Interpolation prediction coding of image signals using feedback difference quantization,” IEICE vol.J66-B, no.5, pp.599-606, May 1983 “Report on 3DAV Exploration,” N5878, ISO / IEC JTC1 / SC29 / WG11 (MPEG), 2003 Hitoshi Uekura, Hiroshi Watanabe, Naoki Kobayashi, Susumu Ichinose, Hiroshi Yasuda, “Global Motion / Brightness Change Compensated Video Coding Considering Computational Complexity,” IEICE Theory (B), vol.J82-B, no.9 , Pp.1676-1688, 1999 Hideaki Kimata, Masaki Kitahara, and Yoshiyuki Yashima, “RECURSIVELY WEIGHTING PIXEL DOMAIN INTRA PREDICTION ON H.264,” SPIE Visual Communication and Image Processing 2003 (VCIP), vol.5150, pp.2035-2042, 2001 Marta Karczewicz, Jani Lainema, Joni Vahteri and Bogdan Dobrin, “Nokia MVC H.26L Proposal Description,” ITU-T SG16 Q.6 document Q15-J-19, May, 2000

  In coding that divides the screen into regions and performs intra prediction, coding efficiency increases when the difference (prediction residual) between image information in the region and the predicted image is small. Search and determine the mode.

  In the conventional intra prediction encoding, a prediction image is created using image information of the current frame that has already been encoded and decoded. Therefore, in the end portion of the screen, the portion that has already been encoded and decoded is limited, and there is a problem that the prediction method is limited. In addition, when a high-resolution image is divided into a plurality of regions and encoded, there is a problem that the encoding efficiency is reduced because each divided region is individually encoded.

  The present invention solves the above-mentioned problems, and in the encoding of the area at the end of the screen when the screen is divided into a plurality of areas, the encoding efficiency for dividing a high resolution image into a plurality of areas, etc. It aims at improving.

  In order to solve the above problems, an image coding apparatus according to the present invention accumulates a reference image and encodes image information in an area obtained by dividing the current frame using inter-frame prediction coding. The apparatus uses a reference image memory for storing a reference image of a plurality of frames, and geometric information between a reference frame and a current frame that are set in advance, based on the image information of the reference frame and the image information of the current frame. A prediction image generation unit that generates a prediction image of a frame, and a difference encoding unit that encodes a difference between the image information in the region into which the current frame is divided and the prediction image are provided.

  An image decoding apparatus according to the present invention is an image decoding apparatus that accumulates reference images and decodes image information in an area obtained by dividing a current frame using inter-frame prediction encoding, and includes a plurality of frames. The reference frame image information and the current frame image information are stored using a reference image memory for storing the reference image, a difference decoding unit for decoding the difference image information, and geometric information between the preset reference frame and the current frame. And a decoded image generating unit that generates a decoded image from the predicted image and the difference image information.

  Thus, when encoding a plurality of images taken by cameras at different shooting points and shooting directions as a plurality of frames, the image information of the already encoded and decoded frame (reference frame) and the current frame are encoded. The prediction image can be created from the image information of the pixel position that has already been encoded and decoded.

  For example, when the images from the regions D1 to D4 are encoded with the images as shown in FIG. 21, if the images from the regions U1 to U4 are already encoded and decoded, the image information is shown in FIG. By considering any one of the pixels A to P, a predicted image as shown in FIG. 19 can be created.

  Also, for example, when the screen is divided as shown in FIG. 23, if the lower part of the image is first encoded and decoded and this is used as a reference frame, the upper part of FIG. When creating a predicted image for information, it is possible to use the image information (reference frame) in the lower part. Therefore, if there is already encoded and decoded image information in the upper part, it becomes the same as the interpolation shown in FIG. 20, and the prediction efficiency can be improved. If the purpose of the division is to distribute the amount of computation, when the second line is encoded in the upper part, the first line in the lower part has already been decoded, so that it can be used for prediction. Is possible.

  Geometric information between the reference frame and the current frame is used when selecting a pixel position to be used for prediction from within the reference frame. As in the above example, image information at the end of the reference frame may be used, or image information inside the reference frame may be used.

  FIG. 1 is a diagram illustrating an example of images from a plurality of cameras. For example, in the case of an image of camera A and camera B (FIG. 1B) having a geometric information relationship as shown in FIG. 1A, the inside of the image (reference frame) of camera A (position X) And the image information of the image of the camera B (current frame) are used to create a predicted image. In FIG. 1B, the position X of the image of the camera A corresponds to the left end of the image of the camera B.

  Further, the present invention may be applied to moving images. For example, when each region shown in FIG. 21 forms a moving image, it is possible to select candidates from more reference frames when creating a predicted image of the current frame. This is particularly effective when the camera imaging system moves.

  FIG. 2 is a diagram illustrating an example of creating a predicted image when the camera moves. As shown in FIG. 2A, a procedure when the position of one camera is moved (position A → position B) is shown. In this case, a predicted image is created from image information of reference frames at different times and image information of frames at the current time in one video. The geometric information of the video at each time is measured, and the geometric information between the reference frame and the current frame is determined as shown in FIG. In addition to the time information TR, such geometric information is encoded for each frame. In FIG. 2B, the position X of the image at the position A corresponds to the left end of the image at the position B.

  Note that an appropriate filter may be applied between the image information of the reference frame used to create the predicted image and the decoded image at the encoding target position in the current frame. By this filter processing, distortion between the reference frame and the current frame is averaged, and subjective quality can be improved.

  Further, after selecting image information to be used from the reference frame, the value of this image information may be changed by calculation and then used. For example, when the average pixel value is different between the reference frame and the current frame, the pixel value may be used after being corrected by the method of Non-Patent Document 4, for example.

  In the case of lossless coding, the original image of the reference frame may be used instead of using the already encoded and decoded image information as the reference frame.

  Geometric information between images may be measured in advance when the imaging system is constructed, or if the camera imaging system moves, the movement amount of the position information of the camera imaging system may be used. Good. It is also possible to measure this position information using GPS or the like. In particular, when the imaging system is fixed in a moving image, the geometric information need not be encoded every frame, and may be set in advance.

When the geometric information changes with time, the geometric information can be encoded and included in the encoded data, and the information can be used on the decoding side. When encoding geometric information,
・ Camera position information,
-Image position information in the absolute coordinate system,
・ Relative position information of video,
Any of the above may be the encoding target.

  In particular, when the camera moves, relative position information is suitable. A method using the amount of movement between screens is also suitable as the relative position information of the video.

  FIG. 3 is a diagram illustrating an example of a method for expressing the positional relationship between two screens. For example, as shown in FIG. 3, if there are a screen A and a screen B, the relative position information thereof includes the position information of the four corner pixels of the screen A based on the positions of the four corner pixels of the screen B, That is, it may be expressed by positional information of (x1, y1), (x2, y2), (x3, y3), (x4, y4).

  Further, the geometric information may be estimated from the image information so that the encoding efficiency of the current frame is the highest. Geometric information may be included in the encoded data. In this case, image information to be used is selected from the reference frame using the geometric information in the encoded data.

  According to the present invention, when encoding video from a wide-angle camera, video from a plurality of cameras, or video from a moving camera, using geometric information between the reference frame and the current frame, Coding efficiency can be improved by creating a predicted image from a decoded image of an image from another camera or a past image and a decoded image of the current frame.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, creation of a predicted image in the embodiment of the present invention will be described.

  FIG. 4 is a diagram illustrating an example of pixel positions used for creating a predicted image of the present embodiment. In the creation of a predicted image in the present embodiment, a predicted image is created using image information having the arrangement shown in FIGS. 4A to 4O for macroblocks of 16 pixels in length and width. FIGS. 4A to 4D show examples in the case where a predicted image is created using image information in one of the upper, lower, left and right directions, and FIGS. FIGS. 4K to 4N show an example of creating a predicted image using image information in any one of three directions (up, down, left, and right). FIG. 4 (O) shows an example in which a predicted image is created using image information in all directions of up, down, left, and right.

  FIG. 5 is a diagram showing the positional relationship of the surrounding pixels of the current macroblock. As shown in FIG. 5, the pixels arranged on the current macroblock (the macroblock that is the target for creating the predicted image at this time) are represented by p1 [i] (i = 1,..., 16), left P2 [j] (j = 1,..., 16), the pixel arranged on the right is p3 [j] (j = 1,..., 16), and the pixel arranged below P4 [i] (i = 1,..., 16).

In the present embodiment, these pixels are used, and any one of the following pattern prediction modes is applied, and the pixels in macroblock q [j] [i] (i, j = 1,..., 16) are Ask. However, i shows the position of a horizontal direction and j shows the position of a vertical direction. In addition, Σ _k represents the total sum from 1 to 16 in the equation following Σ _k .

○ Pattern 1
Pixel used: FIG. 4 (A)
Predictive image creation formula:
q [j] [i] = p2 [j] (1).
○ Pattern 2
Pixel used: FIG. 4B
Predictive image creation formula:
q [j] [i] = p1 [i] (2).
○ Pattern 3
Pixels used: FIG. 4C
Predictive image creation formula:
q [j] [i] = p3 [j] (3).
○ Pattern 4
Pixel used: FIG. 4 (D)
Predictive image creation formula:
q [j] [i] = p4 [i] (4).
○ Pattern 5
Pixel used: FIG. 4 (E)
Predictive image creation formula:
q [j] [i] = (Σ _k (p1 [k] + p2 [k])) / 32 (5).
○ Pattern 6
Pixel used: FIG. 4 (F)
Predictive image creation formula:
q [j] [i] = (Σ _k (p1 [k] + p3 [k])) / 32 (6).
○ Pattern 7
Pixel used: FIG. 4 (G)
Predictive image creation formula:
q [j] [i] = (Σ _k (p4 [k] + p3 [k])) / 32 (7).
○ Pattern 8
Pixel used: FIG. 4 (H)
Predictive image creation formula:
q [j] [i] = (Σ _k (p4 [k] + p2 [k])) / 32 (8).
○ Pattern 9
Pixel used: FIG. 4 (I)
Predictive image creation formula:
q [j] [i] = (p1 [i] + p4 [i]) / 2 (9).
○ Pattern 10
Pixel used: FIG. 4 (J)
Predictive image creation formula:
q [j] [i] = (p2 [j] + p3 [j]) / 2 (10).
○ Pattern 11
Pixel used: FIG. 4 (K)
Predictive image creation formula:
q [j] [i] = (Σ _k (p1 [k] + p2 [k] + p4 [k])) / 48 (11).
○ Pattern 12
Pixel used: FIG. 4 (L)
Predictive image creation formula:
q [j] [i] = (Σ _k (p1 [k] + p2 [k] + p3 [k])) / 48 (12).
○ Pattern 13
Pixel used: FIG. 4 (M)
Predictive image creation formula:
q [j] [i] = (Σ _k (p1 [k] + p3 [k] + p4 [k])) / 48 (13).
○ Pattern 14
Pixel used: FIG. 4 (N)
Predictive image creation formula:
q [j] [i] = (Σ _k (p2 [k] + p3 [k] + p4 [k])) / 48 (14).
○ Pattern 15
Pixel used: FIG. 4 (O)
Predictive image creation formula:
q [j] [i] = (Σ _k (p1 [k] + p2 [k] + p3 [k] + p4 [k])) / 64 (15).
Here, in each pattern, a calculation formula for creating a predicted image may be set differently from the above. For example, in the pattern 5, the following calculation formula may be used instead of the above formula (5).

○ Pattern 5-2
Pixel used: FIG. 4 (E)
Predictive image creation formula:
q [j] [i] = (p1 [i] + p2 [j]) / 2 (16).
In addition, instead of creating a prediction image with the same calculation formula in a macroblock as described above, a prediction image may be created by recursively using a prediction image that has already been created as in Non-Patent Document 5. . Further, a macroblock may be divided into small areas and a predicted image may be created for each area. Further, as in Non-Patent Document 6, a parameter indicating a change in a predicted image within a block may be encoded.

[First Embodiment]
Next, image encoding and image decoding in the embodiment of the present invention will be described. First, a procedure for encoding an omnidirectional video as shown in FIG. 22 will be described as the first embodiment of the present invention.

  FIG. 6 is a diagram for explaining an example of an encoding order of videos from a plurality of cameras. As shown in FIG. 22, the omnidirectional video is projected onto a cube, and as shown in FIG. 6, video S1 to video S6 are encoded alternately for the same TR.

  Here, on the assumption that the geometric information shown in FIG. 22 exists between video S1 and video S6, the pixel position used for predictive coding is determined from the reference frame. At this time, an example is shown in which the frame of the image S6 is divided into 16 pixels vertically and horizontally macroblocks, and a prediction image creation method is selected and encoded so that the code amount of differentially encoded data is minimized for each macroblock. It is assumed that images at the current time from image S1 to image S5 are already stored in the reference image memory.

  FIG. 7 shows a configuration example of the image coding apparatus according to the present embodiment. The image encoding device includes an image input unit 11 that captures image information, a reference image memory 12 that can store reference images of a plurality of frames, a prediction mode search unit 13 that searches for a prediction mode, and a prediction mode. Created by a prediction mode encoding unit 14 for encoding, a prediction image generating unit 15 for generating a prediction image, a difference encoding unit 16 for encoding a difference between input image information and a prediction image, and a difference encoding unit 16 A decoding unit 17 that decodes the differentially encoded data and generates a decoded image; and a reference image switching unit 18 that switches a reference image used in the predicted image generation unit 15.

  The reference image memory 12 is provided with a memory capable of storing images for six frames from the image S1 to the image S6, and stores image information of a frame immediately before each corresponding image. The image input unit 11 is assumed to divide the input image into macro blocks.

  FIG. 8 is a use pixel determination process flowchart when creating a predicted image in the present embodiment. The prediction mode searching unit 13 and the predicted image creating unit 15 can be used when creating a predicted image according to the flowchart shown in FIG. 8 for the macroblock (hereinafter also referred to as MB) of the image S6 to be encoded. Determine the pixel. Here, encoding is performed in order from the upper left MB.

  FIG. 9 is a diagram illustrating an example of pixel positions of a cubic reference frame in the present embodiment. The positions around the image S6 when the cube is developed around the image S6 are represented as B1 to B8. At this time, images S2 to S5 exist at the positions B1 to B4, respectively. Since the image at the current time from the image S1 to the image S5 has already been stored in the reference image memory 12, the image information at the end from the image S2 to the image S5 adjacent to the image S6 is created as the predicted image S6. Use when.

  When the encoding target MB is the upper left MB of the image S6 (step 100), the pixels of the image S2 and the image S3 at the positions of B1 and B2 are determined as pixels that can be used to create a predicted image ( Step 101). When the encoding target MB is the upper right MB of the image S6 (step 102), the pixels of the images S2 and S4 at the positions of B1 and B3 and the pixels of the current frame (image S6) are predicted images. It is determined that the pixel can be used for creation of (step 103). When the encoding target MB is the upper MB of the image S6 (step 104), the pixel of the image S2 at the position of B1 and the pixel of the current frame (image S6) can be used to create a predicted image. The pixel is determined to be a correct pixel (step 105).

  When the encoding target MB is the lower left MB of the image S6 (step 106), the pixel of the image S3 and the image S5 at the positions of B2 and B4 and the pixel of the current frame (image S6) are predicted images. Are determined to be usable pixels (step 107). When the encoding target MB is the lower right MB of the image S6 (step 108), the pixels of the images S4 and S5 at the positions of B3 and B4 and the pixels of the current frame (image S6) are predicted. It is determined that the pixel can be used to create an image (step 109). When the encoding target MB is the lower MB of the image S6 (step 110), the pixel of the image S5 at the position of B4 and the pixel of the current frame (image S6) can be used to create a predicted image. A pixel is determined (step 111).

  When the encoding target MB is the left MB of the image S6 (step 112), the pixel of the image S3 at the position of B2 and the pixel of the current frame (image S6) can be used to create a predicted image. A pixel is determined (step 113). If the MB to be encoded is the right MB of the image S6 (step 114), the pixel of the image S4 at the position of B3 and the pixel of the current frame (image S6) can be used to create a predicted image. The pixel is determined to be a correct pixel (step 115).

  When the encoding target MB does not correspond to any of the above (when the encoding target MB is not the end MB of the image S6), the pixel of the current frame (image S6) can be used to create a predicted image. (Step 116).

  The prediction mode search unit 13 creates a prediction image defined by various prediction modes from the image information of the reference frame or the current frame, and selects a prediction mode having the smallest sum of absolute differences between the input image and the prediction image. , The prediction mode of the macroblock is determined.

  The prediction mode search unit 13 and the prediction image creation unit 15 create a prediction image using the image information of the arrangement shown in FIG. However, only the image information of the pixel position determined in the flowchart of FIG. 8 is used.

  FIG. 10 is a flowchart of an image encoding process in the present embodiment. First, the image input unit 11 takes in an input image (for example, the image S6 in FIG. 22) and divides it into macro blocks (step 200). Each divided macro block is encoded as follows.

  The prediction mode search unit 13 determines pixel positions that can be used for the prediction image for the macroblock to be encoded according to the flowchart shown in FIG. 8 (step 201). Next, among the prediction modes from pattern 1 to pattern 15 described above, only a pattern using the determined pixel position is sequentially applied to create a prediction image, and the absolute value difference sum between the original image and the prediction image is calculated. Then, the prediction mode of the pattern having the smallest value is obtained (step 202). At this time, the reference image switching unit 18 switches the reference image so that the corresponding pixel can be read from the reference image memory 12 in the development view shown in FIG.

  After determining the prediction mode in this way, the predicted image creating unit 15 creates a predicted image (step 203). At this time, the reference image switching unit 18 switches the reference image again so as to create a predicted image in the determined prediction mode.

  The differential encoding unit 16 encodes the difference between the original image and the predicted image (step 204), and the decoding unit 17 decodes the encoded data to create a decoded image (step 205). The data is stored in the reference image memory 12 for S6 (step 206). Moreover, the prediction mode encoding part 14 encodes the prediction mode determined by the prediction mode search part 13 (step 207).

  The above processing (steps 201 to 207) is executed for all macroblocks of the current input image (step 208). From the next macroblock, a predicted image is created and encoded using the decoded image of the current image stored in the reference image memory 12. In this way, all current input images are encoded.

  FIG. 11 is a diagram illustrating a configuration example of the image decoding device according to the present embodiment. The image decoding apparatus includes a differential decoding unit 21 that decodes differentially encoded data, a reference image memory 22 that can store a plurality of frames of reference images, a prediction mode decoding unit 23 that decodes a prediction mode, and a prediction image A predicted image creating unit 24 that creates a decoded image from the difference image and the predicted image, and a reference image switching unit 26 that switches a reference image used in the predicted image creating unit 24.

  The reference image memory 22 includes a memory capable of storing images for six frames from the image S1 to the image S6, and stores image information of a frame immediately before each corresponding image.

  As in the case of the image encoding device described above, the predicted image creation unit 24 creates a predicted image according to the flow shown in FIG. 8 for the macroblock of the image to be decoded next (for example, the image S6 in FIG. 22). Pixels that can be used at this time are determined, and one of the prediction modes from pattern 1 to pattern 15 decoded by the prediction mode decoding unit 23 is applied to create a prediction image.

  FIG. 12 is a flowchart of image decoding processing according to the present embodiment. The image decoding apparatus decodes the encoded data of the image (for example, the image S6) encoded by the above-described image encoding apparatus for each macroblock as follows.

  The prediction mode decoding unit 23 decodes the prediction mode (step 300), and the prediction image creation unit 24 creates a prediction image (step 301). At this time, the reference image switching unit 26 switches the reference image so as to create a predicted image according to the decoded prediction mode.

  The difference decoding unit 21 decodes the difference encoded data to obtain difference image information (step 302), and the decoded image creation unit 25 creates a decoded image using the difference image information and the predicted image (step 303). ). The decoded image is output and stored in the reference image memory 22 for the image S6 (step 304).

  The above processing (steps 300 to 304) is executed for all macroblocks of the currently decoded image (step 305). Here, from the next macroblock, a predicted image is created and decoded using the decoded image of the current image stored in the reference image memory 22. In this way, all the current input images are decoded.

[Second Embodiment]
Next, as a second embodiment of the present invention, an image encoding procedure when the camera moves will be described. FIG. 13 is a diagram illustrating an example of creating a predicted image when the camera moves in the present embodiment. As shown in FIG. 13A, the camera moves in the order of position A → position B → position C and performs imaging. In FIG. 13B, the position A-1 of the image at position A and the position B-1 of the image at position B correspond to the left end of the image at position C.

  Hereinafter, an example of creating a predicted image using an image when the camera is at position A or B when encoding an image when the camera is at position C will be described. It is assumed that decoded images at positions A and B are already stored in the reference image memory.

  In the present embodiment, the pixel position used for predictive coding is determined from the reference frame on the assumption that the geometric information shown in FIG. 13 exists between position A and position C, and position B and position C. To do. At this time, the image at position C is divided into 16 pixels vertically and horizontally macroblocks, and the reference image and the prediction image creation method are selected and encoded so that the code amount of differentially encoded data is minimized for each macroblock. To do.

  FIG. 14 is a diagram illustrating a configuration example of an image encoding device according to the present embodiment. The image encoding device includes an image input unit 31 that captures image information to be encoded, a reference image memory 32 that can store a plurality of frames of reference images, a prediction mode search unit 33 that searches for a prediction mode, A prediction mode encoding unit 34 that encodes a prediction mode, a prediction image generation unit 35 that generates a prediction image, a difference encoding unit 36 that encodes a difference between input image information and a prediction image, and a difference encoding unit A decoding unit 37 that decodes the differentially encoded data generated in 36 to generate a decoded image, a reference image switching unit 38 that switches a reference image used in the predicted image generating unit 35, and a reference image specification that specifies a reference image And a reference image designation information encoding unit 39 that encodes information.

  It is assumed that the decoded images at position A and position B have already been stored in the reference image memory 32. It is further assumed that a memory capable of storing the decoded image at position C is provided. It is assumed that the image input unit 31 divides the input image into macro blocks.

  The predicted image creation unit 35 creates a predicted image for the leftmost macroblock of the image at position C from the image at position A or position B in the reference image memory 32 and the decoded image at position C. For macroblocks at other positions, a predicted image is created from the decoded image at position C in the reference image memory 32.

  The prediction mode search unit 33 creates a prediction image defined by various prediction modes from image information of the reference frame or the current frame, and predicts the reference image having the smallest sum of absolute differences between the input image and the prediction image. The mode is determined as the reference image and prediction mode of the macroblock.

  FIG. 15 is a flowchart of an image encoding process in the present embodiment. First, the image input unit 31 takes in the input image at the position C and divides it into macro blocks (step 400). Each divided macroblock is encoded as follows.

  The prediction mode search unit 33 sets the reference image to the image at position A, sequentially applies the prediction modes from pattern 1 to pattern 15 to create a prediction image, and calculates the sum of absolute value differences between the current image and the prediction image. And the prediction mode of the pattern having the smallest value is obtained (step 401). Next, a reference image is set as an image at position B, prediction modes from pattern 1 to pattern 15 are sequentially applied to generate a prediction image, and an absolute value difference sum between the current image and the prediction image is obtained. A pattern having the smallest value is obtained (step 402).

  Comparing the minimum absolute value difference sum when the image of the position A obtained in step 401 is a reference image and the minimum absolute value difference sum when the image of the position B obtained in step 402 is a reference image (Step 403). If the minimum absolute value difference sum when the image at position A is the reference image is smaller, the image at position A is determined as the reference image of the macroblock (step 404), and the prediction mode obtained at step 401 is determined. Is determined as the prediction mode of the macroblock (step 405). If the minimum absolute value difference sum when the image at position B is used as the reference image is smaller, the image at position B is determined as the reference image of the macroblock (step 406), and the prediction mode obtained at step 402 is determined. Is determined as the prediction mode of the macroblock (step 407).

  Here, when creating a predicted image, among the prediction modes from pattern 1 to pattern 15, the image at position A or the image at position B and the pixel position of the current image that has been encoded and decoded are used. Apply pattern only.

  After determining the reference image and the prediction mode in this way, the predicted image creation unit 35 creates a predicted image in the determined prediction mode (step 408). At this time, the reference image switching unit 38 switches the reference image so as to create a predicted image with the determined reference image.

  The difference encoding unit 36 encodes the difference between the current image and the predicted image (step 409), and the decoding unit 37 decodes the encoded data to create a decoded image (step 410). The reference image memory 32 for position C is stored (step 411). The prediction mode encoding unit 34 encodes the prediction mode (step 412), and the reference image designation information encoding unit 39 encodes the reference image designation information (step 413).

  The above processing (steps 401 to 413) is executed for all macroblocks of the current image (step 414). Here, from the next macroblock, a predicted image is created and encoded using the decoded image at position C stored in the reference image memory 32. In this way, all the images at the input position C are encoded.

  FIG. 16 is a diagram illustrating a configuration example of an image decoding device according to the present embodiment. The image decoding apparatus includes a differential decoding unit 41 that decodes differentially encoded data, a reference image memory 42 that can store reference images of a plurality of frames, a prediction mode decoding unit 43 that decodes a prediction mode, and a prediction image A predicted image creating unit 44 that creates a decoded image from the difference image and the predicted image, a reference image switching unit 46 that switches a reference image used in the predicted image creating unit 44, and a reference image A reference image designation information decoding unit 47 for decoding the reference image designation information for designating.

  A case will be described in which the decoded images of the position A and the position B are already stored in the reference image memory 42, and the encoded data of the image of the position C is decoded from now on. As in the case of the above-described image encoding device, the predicted image creation unit 44 determines the position A or the position B in the reference image memory 42, the decoded image at the position C, and the like for the leftmost macroblock of the position C image. Create a prediction image from For macroblocks at other positions, a predicted image is created from the decoded image at position C in the reference image memory 42. Under such a premise, the encoded data encoded by the image encoding device of FIG. 14 is decoded for each macroblock as follows.

  FIG. 17 is a flowchart of image decoding processing according to the present embodiment. First, the prediction mode decoding unit 43 decodes the prediction mode (step 500), and the reference image designation information decoding unit 47 decodes the reference image designation information (step 501).

  The predicted image creation unit 44 creates a predicted image according to the decoded prediction mode (step 502). At this time, the reference image switching unit 46 switches the reference image so as to create a predicted image using the reference image specified by the decoded reference image specifying information.

  The difference decoding unit 41 decodes the difference encoded data to obtain difference image information (step 503), and the decoded image creation unit 45 creates a decoded image using the difference image information and the predicted image (step 504). ). The generated decoded image is output and the decoded image is stored in the reference image memory 42 for position C (step 505).

  The above processing (steps 500 to 505) is executed for all macroblocks of the current image (step 506). From the next macroblock, a predicted image is created and decoded using the decoded image at position C stored in the reference image memory 42.

  The processing performed by the image encoding device and the image decoding device in the first and second embodiments described above can be realized not only using hardware and firmware, but also using a computer and a software program. The program can be provided by being recorded on a computer-readable recording medium or can be provided through a network.

  In the first and second embodiments described above, the geometric information between the reference frame and the current frame is preset and is not included in the encoded data, but is included in the encoded data. May be.

  Further, in the present embodiment, the position where the sum of absolute value differences is set to the smallest in the prediction mode search in the image coding apparatus, for example, the position where the code amount of the actually encoded data is minimized. May be set.

  In the present embodiment, the decoding units 17 and 37 in the image encoding device create a decoded image from the difference image information and the predicted image and store it in the reference image memories 12 and 32. Images may be stored in the reference image memories 12 and 32.

It is a figure which shows the example of the image | video from a some camera. It is a figure which shows the example of prediction image creation when a camera moves. It is a figure which shows the example of the expression method of the positional relationship between 2 screens. It is a figure which shows the example of the position of the pixel used for preparation of an estimated image. It is a figure which shows the positional relationship of the surrounding pixel of the present macroblock. It is a figure explaining the example of the encoding order of the image | video from several cameras. It is a figure which shows the structural example of the image coding apparatus in this Embodiment. It is a use pixel determination process flowchart at the time of the prediction image preparation in this Embodiment. It is a figure which shows the example of the pixel position of the cube reference frame in this Embodiment. It is an image encoding process flowchart in this Embodiment. It is a figure which shows the structural example of the image decoding apparatus in this Embodiment. It is an image decoding process flowchart in this Embodiment. It is a figure which shows the example of prediction image preparation when the camera in this Embodiment moves. It is a figure which shows the structural example of the image coding apparatus in this Embodiment. It is an image encoding process flowchart in this Embodiment. It is a figure which shows the structural example of the image decoding apparatus in this Embodiment. It is an image decoding process flowchart in this Embodiment. It is a figure which shows the example of the pixel position used for the prediction in JPEG-LS. H. 2 is a diagram illustrating an example of intra prediction in H.264. It is a figure which shows the example of interpolation. It is a figure which shows the example of the projection to the octahedron of an omnidirectional image | video. It is a figure which shows the example of the projection to the cube of an omnidirectional image | video. It is a figure which shows the example of the screen division | segmentation of a high resolution image.

Explanation of symbols

11, 31 Image input unit 12, 32 Reference image memory 13, 33 Prediction mode search unit 14, 34 Prediction mode encoding unit 15, 35 Prediction image creation unit 16, 36 Difference encoding unit 17, 37 Decoding unit 18, 38 Reference Image switching unit 39 Reference image designation information encoding unit 21, 41 Difference decoding unit 22, 42 Reference image memory 23, 43 Prediction mode decoding unit 24, 44 Predicted image creating unit 25, 45 Decoded image creating unit 26, 46 Reference image switching 47 Reference image designation information decoding unit

Claims (14)

An image encoding apparatus that stores reference images and encodes image information in an area obtained by dividing a current frame using inter-frame predictive encoding,
A reference image memory for storing multiple frames of reference images;
A prediction image creation unit that creates a prediction image of the current frame from the image information of the reference frame and the image information of the current frame using geometric information between the reference frame set in advance and the current frame;
An image encoding apparatus comprising: a difference encoding unit that encodes a difference between image information in a region obtained by dividing the current frame and a predicted image. An image encoding apparatus that stores reference images and encodes image information in an area obtained by dividing a current frame using inter-frame predictive encoding,
An image input unit for capturing image information of the current frame to be encoded;
A reference image memory for storing multiple frames of reference images;
When creating a predicted image when creating a predicted image of the current frame from the image information of the reference frame and the image information of the current frame using geometric information indicating the spatial positional relationship between the reference frame and the current frame A prediction mode search unit for searching for and determining a prediction mode used to create a prediction image according to the geometric information from a plurality of prediction modes corresponding to a plurality of patterns defining image information used for;
A prediction mode encoding unit for encoding the prediction mode;
A predicted image creating unit for creating a predicted image according to the prediction mode;
A differential encoding unit for encoding the difference between the input image information and the predicted image;
An image encoding apparatus comprising: a decoding unit that decodes differentially encoded data generated by the differential encoding unit to generate a decoded image and stores the decoded image in the reference memory. The image encoding device according to claim 2,
An image coding apparatus comprising: a reference image designation information encoding unit that encodes reference image designation information that is referred to by the prediction image generation unit among reference images stored in the reference image memory. In the image coding device according to claim 2 or 3,
An image encoding apparatus comprising: a geometric information encoding unit that encodes the geometric information. An image decoding apparatus that stores reference images and decodes image information in an area obtained by dividing a current frame using inter-frame prediction encoding,
A reference image memory for storing multiple frames of reference images;
A differential decoding unit for decoding the differential image information;
A prediction image creation unit that creates a prediction image of the current frame from the image information of the reference frame and the image information of the current frame using geometric information between the reference frame set in advance and the current frame;
An image decoding apparatus comprising: a decoded image generation unit that generates a decoded image from a predicted image and difference image information. An image decoding apparatus that stores reference images and decodes image information in an area obtained by dividing a current frame using inter-frame prediction encoding,
Using the encoded data of the difference image information of the image to be decoded and the geometric information indicating the spatial positional relationship between the reference frame and the current frame, the current frame image information is obtained from the reference frame image information and the current frame image information. When creating a prediction image, the encoded data of the prediction mode determined and encoded by the image encoding device from among a plurality of prediction modes corresponding to a plurality of patterns that define image information to be used when creating the prediction image An input unit for inputting
A reference image memory for storing multiple frames of reference images;
A prediction mode decoding unit that decodes the encoded data of the input prediction mode;
A differential decoding unit for decoding the differential image information;
A predicted image creating unit that creates a predicted image of the current frame using image information of a reference image stored in the reference image memory according to the decoded prediction mode;
An image decoding apparatus comprising: a decoded image generating unit that generates a decoded image from a predicted image and difference image information and stores the decoded image in the reference image memory. The image decoding device according to claim 6,
An input unit that inputs encoded data of designation information of a reference image that is referred to by the prediction image generation unit among the reference images stored in the reference image memory;
A reference image designation information decoding unit for decoding the encoded data of the input reference image designation information,
The prediction image creating unit reads a reference image indicated by the designation information of the decoded reference image from the reference image memory and uses the reference image to create a predicted image. The image decoding device according to claim 6 or 7,
An image decoding apparatus comprising: a geometric information decoding unit that decodes the geometric information. An image encoding method for storing reference images and encoding image information in an area obtained by dividing a current frame using inter-frame predictive encoding,
A predicted image creating step for creating a predicted image of the current frame from the image information of the reference frame and the image information of the current frame using geometric information between the reference frame set in advance and the current frame;
An image encoding method, comprising: executing a difference encoding step of encoding a difference between image information in a region obtained by dividing the current frame and a predicted image. An image decoding method for storing reference images and decoding image information in an area obtained by dividing a current frame using inter-frame predictive coding,
A predicted image creating step for creating a predicted image of the current frame from the image information of the reference frame and the image information of the current frame using geometric information between the reference frame set in advance and the current frame;
A differential decoding step for decoding the differential image information;
An image decoding method comprising: executing a decoded image creating step of creating a decoded image from a predicted image and difference image information.   An image encoding program for causing a computer to realize the function of each unit of the image encoding device according to any one of claims 1 to 4.   The image decoding program for making a computer implement | achieve the function of each part of the image decoding apparatus described in any one of Claim 5-8.   An image encoding program for causing a computer to realize the function of each unit of the image encoding device according to any one of claims 1 to 4 is recorded on a computer-readable recording medium. A characteristic image encoding program recording medium.   An image decoding program for causing a computer to realize the function of each unit of the image decoding device according to any one of claims 5 to 8 is recorded on a computer-readable recording medium. An image decoding program recording medium.

Images