JP5915168B2 - Encoding device, decoding device, moving image processing system, encoding method, decoding method, encoding program, and decoding program - Google Patents

Description

  The present invention relates to an encoding device, a decoding device, a moving image processing system, an encoding method, a decoding method, an encoding program, and a decoding program.

  In recent years, as video display devices have become more sophisticated, such as high resolution, high bit depth display, and stereoscopic video, video encoding technology corresponding to the aforementioned functions has been disclosed. It is coming.

  For example, for scalability corresponding to multiple bit depths, techniques with bit depth scalability are disclosed using different color encodings for two or more versions having different bit depths. In addition, in stereoscopic video encoding by scalable video encoding, a specific viewpoint image is encoded as a base layer among two viewpoint images, another viewpoint image is encoded as an enhancement layer, and encoding of an enhancement layer is performed. Discloses a technique for predicting from a base layer. In addition, in a method of encoding video data, a technique for rearranging the transmission order of the encoded enhancement layers and delaying the frames of the enhancement layer with respect to the frames of the base layer is disclosed (for example, the following patent document) 1-3.)

Special table 2010-532936 gazette Special table 2009-513074 gazette Special table 2003-527004 gazette

  However, using the above-described conventional technology, an image is extracted and encoded as a region to be encoded as a base layer and a region to be encoded as an enhancement layer from a moving image encoded at a high resolution. When performing the above, the encoding efficiency of the enhancement layer frame decreases. For example, it is assumed that an object shown in the basic layer has moved to the enhancement layer. At this time, if the reference to the past base layer frame cannot be made from the enhancement layer frame, the enhancement layer frame has a code amount related to the object, and the coding efficiency is lowered.

  The present invention provides an encoding device, a decoding device, a moving image processing system, an encoding method, a decoding method, an encoding program, and a decoding program that can improve the encoding efficiency in order to eliminate the above-described problems caused by the prior art. The purpose is to provide.

  In order to solve the above-described problems and achieve the object, according to one aspect of the present invention, a first region is obtained from any one of a series of images with reference to a storage unit that stores the series of images. The image in the second area adjacent to the first area is extracted, the image in the first area is encoded into an encoding frame for the first area, and one of the images is encoded In this case, the dummy frame of the encoded frame for the first area of the previous image of any image is specified based on the image in the first area of the previous image and can be reproduced with the self-encoded frame. The number of frames generated is equal to or less than the number of frames included in the appearance interval of the encoded frame, and the dummy frames generated for the generated number are set as the next encoded frame of the encoded frame for the first region of the previous image. Code for the first region of the image The image of the preceding encoded frame preceding the number of generations of the dummy frame from the encoded frame for the first region of any image in the preceding encoded frame group preceding the frame, and the second region of any image And encoding the image in the second area of any image into an encoded frame for the second area of any image based on the image in each of the images and each code for the first and second areas An encoding device, an encoding method, and an encoding program that output the number of generated frames, the number of generated dummy frames, and information indicating the number of generated dummy frames and the set position are proposed.

  According to one aspect of the present invention, a series of encoded frames for a series of partial images in a first area of a series of images, and a second area adjacent to the first area of the series of images. Identification information corresponding to at least one of a series of encoded frames for a series of partial images and a series of encoded frames for a series of partial images in the first or second region And information indicating the number and position of encoded frames that are not to be output among a series of encoded frames for a series of partial images in the first or second area, and in the first and second areas A series of encoded frames for the series of partial images in the first region is selected from the series of encoded frames for the series of partial images based on the identification information, and the selected series of encoded frames is selected. Decoding into a series of partial images, based on the information indicating the number and position of the encoded frames that are not the output target of the decoded partial images, other than the partial images corresponding to the encoded frames that are not the output target A decoding device, a decoding method, and a decoding program for outputting a partial image are proposed.

  According to one aspect of the present invention, a series of encoded frames for a series of partial images in a first area of a series of images, and a second area adjacent to the first area of the series of images. Identification information corresponding to at least one of a series of encoded frames for a series of partial images and a series of encoded frames for a series of partial images in the first or second region The display area is designated from the area obtained by combining the first area and the second area, and the identification information is obtained from a series of encoded frames for a series of partial images in the first and second areas. Based on this, a series of encoded frames for a series of partial images in the first region are selected, and a series of encoded frames for the series of partial images in the selected first region are selected. To the partial image And when a display area is designated by the designation unit, a series of encoded frames for a series of partial images in the second area are decoded into a series of partial images in the second area, and the first area The first image sequence in the display area is extracted from the series of partial images in the second area, and the second image sequence in the display area is extracted from the series of partial images in the second area. A decoding apparatus, a decoding method, and an image sequence of a display area obtained by combining the images in the image sequence and the images in the second image sequence with the same image in a series of images, And a decoding program is proposed.

  According to another aspect of the present invention, there is provided a moving image processing system including an encoding device that encodes a series of images and a decoding device that decodes the encoded series of images. An image in the first area and an image in the second area adjacent to the first area are extracted from any one of the series of images, and the image in the first area is extracted from the first area. Encoding into an encoded frame, encoding an image in the second region into an encoded frame for the second region based on the image of the encoded frame for the first region and the image in the second region; The identification information indicating at least one of the first and second regions is assigned to the corresponding encoded frame, and the encoded frames for the first and second regions are output to the decoding device, The first and second identification information is given Each encoded frame for a region is received, and from each encoded frame for the first and second regions, an encoded frame for the first region is selected based on the identification information, and a code for the selected first region is selected A moving image processing system is proposed that decodes a frame into a decoded image and outputs the decoded image.

  According to one aspect of the present invention, there is an effect that the encoding efficiency can be improved.

FIG. 1 is an explanatory diagram illustrating an operation example of the encoding device. FIG. 2 is an explanatory diagram illustrating an example of reference constraints for the base view and the extended view. FIG. 3 is a block diagram illustrating an example of a hardware configuration of the encoding device. FIG. 4 is a block diagram illustrating an example of a hardware configuration of the decoding device. FIG. 5 is a block diagram illustrating a functional configuration example of the encoding device and the decoding device. FIG. 6 is an explanatory diagram showing an example of screen extraction at the time of horizontal position shooting. FIG. 7 is an explanatory diagram illustrating an example of a screen extraction method during vertical position shooting. FIG. 8 is an explanatory diagram illustrating an example of a reference direction for motion compensation. FIG. 9 is an explanatory diagram illustrating an example of a reference destination when a dummy frame is set. FIG. 10 shows an example of setting dummy frames in a stream. FIG. 11 is an explanatory diagram showing a setting example of vertical / horizontal information and dummy frame information. FIG. 12 is an explanatory diagram illustrating an example of the encoding process. FIG. 13A is an explanatory diagram (part 1) illustrating an example of a motion vector search range during horizontal position shooting. FIG. 13B is an explanatory diagram (part 2) illustrating an example of a motion vector search range at the time of horizontal position shooting. FIG. 14A is an explanatory diagram (part 1) illustrating an example of a motion vector search range during vertical position shooting. FIG. 14B is an explanatory diagram (part 2) illustrating an example of a motion vector search range during vertical position shooting. FIG. 15 is an explanatory diagram showing an example of the decoding process. FIG. 16A is an explanatory diagram (part 1) illustrating a specific example of output image combining processing. FIG. 16B is an explanatory diagram (part 2) of the specific example of the output image combining process. FIG. 16C is an explanatory diagram (part 3) of the specific example of the output image combining process. FIG. 17 is a flowchart illustrating an example of an encoding processing procedure. FIG. 18A is a flowchart (part 1) illustrating an example of the MVC encoding process. FIG. 18B is a flowchart (part 2) illustrating an example of the MVC encoding process. FIG. 19 is a flowchart illustrating an example of a motion vector search range determination process in the base view. FIG. 20A is a flowchart (part 1) illustrating an example of a motion vector search range determination process in the extended view at the time of horizontal position shooting. FIG. 20B is a flowchart (part 2) illustrating an example of the motion vector search range determination process in the extended view at the time of horizontal position shooting. FIG. 21A is a flowchart (part 1) illustrating an example of a motion vector search range determination process in the extended view at the time of vertical position shooting. FIG. 21B is a flowchart (part 2) illustrating an example of the motion vector search range determination process in the extended view at the time of vertical position shooting. FIG. 22A is a flowchart (part 1) illustrating an example of a decoding processing procedure. FIG. 22B is a flowchart (part 2) illustrating an example of the decoding processing procedure.

  Exemplary embodiments of an encoding device, a decoding device, a moving image processing system, an encoding method, a decoding method, an encoding program, and a decoding program according to the present invention will be described below in detail with reference to the accompanying drawings.

  FIG. 1 is an explanatory diagram illustrating an operation example of the encoding device. FIG. 1 illustrates the operation of the encoding apparatus 101 that encodes a moving image. The encoding device 101 captures an imaging range 110 including the imaging object a. The user is shooting while checking the display 102. The display 102 has a display area of horizontal 1920 [pixels] and vertical 1080 [pixels], which is full HD (High Definition). The encoding apparatus 101 accepts the information shown in the shooting range 110 as an input moving image with horizontal 3840 [pixel] and vertical 2160 [pixel], and converts it into a moving image with horizontal 3840 [pixel] and vertical 1080 [pixel]. Encode.

  Hereinafter, the resolution of full HD is referred to as “2k1k”, the resolution of horizontal 3840 [pixel] and vertical 2160 [pixel] is referred to as “4k2k”, and the resolution of horizontal 3840 [pixel] and vertical 1080 [pixel] is “ 4k1k ".

  As described above, in the application in which a 4k1k moving image expanded in the horizontal direction from the resolution of 2k1k full HD is recorded and the center 2k1k is displayed at the time of reproduction, the encoding apparatus 101 displays the moving image to be displayed on the display 102. The purpose is to reduce the amount of decoding.

  First, the encoding apparatus 101 according to the present embodiment is an H.264. In accordance with the H.264 MVC standard, the input image 111 is handled as a moving image taken from a plurality of viewpoints. H. In H.264 MVC, encoding is performed as one base view and one or more non-base views. The base view is H.264. It is encoded as a high profile stream defined by H.264 AVC (Advanced Video Coding). The non-base view is H.264. Using the profile and syntax newly extended with H.264 MVC, encoding is performed with reference to frames included in other views or in one's own view.

  Images in the base view are encoded in a format that can be played independently, and images in the non-base view are encoded by referring to the images in the base view. Encoding efficiency can be improved. H. H.264 MVC is assumed to be used in 3D moving image content, and it is possible to improve encoding efficiency by using the feature that images observed from a plurality of viewpoints are pixels close to each other at the same time. it can.

  The encoding apparatus 101 that has received the input moving image extracts an image in the base view and an extended view image that is a non-base view from any one of the input images 111 of the input moving image, and encodes each of them. The base view is set to a 2k1k area at the center of 4k2k, and the extended view is set to a 1k1k area on the left side of the base view and a 1k1k area on the right side of the base view.

  Subsequently, the encoding apparatus 101 encodes the image in the base view and the image in the extended view, and outputs the encoded image to the stream 121. The stream 121 is data accessed sequentially from the top. The stream 121 is recorded in a recording device in the encoding device 101.

  Specifically, the encoding apparatus 101 encodes an image in the base view and outputs an encoded frame 122 to the stream 121. Also, the encoding apparatus 101 encodes an image in the extended view and outputs an encoded frame 123 to the stream 121. The encoded frame 123 is encoded with reference to a local decoded image at the time of base view encoding. In addition, identification information 124 indicating that an image in the extended view has been encoded is assigned to the encoded frame 123. Note that the encoding apparatus 101 may add identification information indicating that an image in the base view has been encoded to the encoded frame 122.

  Next, at the time of reproduction after shooting, the encoding apparatus 101 detects the identification information 124 out of the encoded frame 122 and the encoded frame 123 and corresponds to the base view to which the identification information 124 is not given. The encoded frame 122 is selected, and the encoded frame 122 is decoded. Subsequently, the encoding apparatus 101 displays the decoded image on the display 102. In addition, when performing reproduction at the time of shooting, the encoding apparatus 101 displays an image in the base view on the display 102 as it is without encoding or decoding.

  As described above, the encoding apparatus 101 is the H.264 standard. Using H.264 MVC, an image in the base view and an image in the extended view are extracted from a 4k2k image, with 2k1k as an independent base view and left and right regions of the base view as extended views that refer to the base view. Subsequently, the encoding apparatus 101 assigns identification information 124 to either the encoded frame 122 or the encoded frame 123 obtained by encoding the extracted image. By providing the identification information 124, an apparatus that detects the identification information 124 and decodes the base view can limit the decoding target to the encoded frame for the base view based on the identification information 124, and thus the amount of decoding can be reduced. The time required for decoding can be shortened. Next, the H.P. The reference restriction of the base view and the extended view according to the H.264 MVC standard will be described.

  FIG. 2 is an explanatory diagram illustrating an example of reference constraints for the base view and the extended view. A table 201 shown in FIG. 2A shows a list of reference constraints of the base view and the extended view, and FIG. 2B shows a specific example when the reference constraints shown in the table 201 are applied. FIG.

  H. In the H.264 MVC standard, an I (Intra-coded) picture encoded without using inter-frame prediction, a P (Predictive) picture encoded with reference to a frame existing in the forward direction, one or two There are B (Bi-predictive) pictures encoded with reference to frames. H. In H.264 MVC, it is possible to select which inter-frame prediction is used for each slice obtained by dividing one frame. Specifically, H.M. In H.264 MVC, each slice is encoded with reference to an I slice coded without using inter-frame prediction, a P slice coded with reference to a slice existing in the forward direction, or one or two slices. B slice can be specified. In the present embodiment, it is assumed that one frame is one slice for simplification of description.

  FIG. 2B shows the image b0 in the base view and the image d0 in the extended view at the time t0, the image b1 in the base view and the image d1 in the extended view at the time t1, and the base view at the time t2. It is the figure which showed image b2 of this, and the image d2 in an extended view.

  If the reference source image belongs to the base view and the reference destination image belongs to the base view, H.264 MVC specifies that a reference image can be encoded with reference to past and future frames in the base view. The case where the reference source image belongs to the base view and the reference destination image belongs to the base view is the field indicated by (1) in Table 201. For example, the encoding device 101 can refer to the image b0 and the image b2 indicated by the arrow indicated by (1) when the image b1 is encoded.

  When the reference source image belongs to the base view and the reference destination image belongs to the extended view, H.264 MVC stipulates that the reference image cannot be encoded by referring to the image belonging to the extended view. For example, the encoding device 101 cannot refer to the images d0 to d2 when encoding the image b1.

  When the reference source image belongs to the extended view and the reference destination image belongs to the base view, H.264 MVC specifies that a reference image can be encoded with reference to images belonging to a base view at the same time. The case where the reference source image belongs to the extended view and the reference destination image belongs to the base view is the field indicated by (2) in Table 201. For example, the encoding device 101 can refer to the image b1 indicated by the arrow indicated by (2) when encoding the image d1.

  When the reference source belongs to the extended view and the reference destination belongs to the extended view, H.264 MVC stipulates that a reference image can be encoded with reference to a past frame and a future frame. The case where the reference source belongs to the extended view and the reference destination belongs to the extended view is the field indicated by (3) in Table 201. For example, the encoding apparatus 101 can refer to the image d0 and the image d2 indicated by the arrow indicated by (3) when the image d1 is encoded. In this way, H.C. H.264 MVC makes it possible to reproduce the base view alone by preventing the encoded frame of the base view from referring to the image of the extended view.

  Subsequently, the hardware according to the present embodiment will be described. The hardware employed in the present embodiment is an encoding device 101 that records and reproduces a moving image, and a decoding device that reproduces the moving image. The encoding apparatus 101 will be described with reference to FIG. 3, and the decoding apparatus will be described with reference to FIG.

(Example of hardware configuration of encoding apparatus 101)
FIG. 3 is a block diagram illustrating an example of a hardware configuration of the encoding device. In FIG. 3, an encoding apparatus 101 that records and reproduces a moving image includes a CPU (Central Processing Unit) 301, a ROM (Read-Only Memory) 302, and a RAM (Random Access Memory) 303. The encoding device 101 also includes an image sensor 304, an image sensor IF (InterFace) 305, a gyro sensor 306, an operation panel controller 307, a recording medium 308, and an external IF 309. The encoding apparatus 101 includes an image processing unit 310 and an MVC encoding / decoding processing unit 311.

  The encoding apparatus 101 includes a display 102, a display output IF 312 and an image output IF 313. The CPU 301 to RAM 303, the image sensor IF 305, the external IF 309 to MVC encoding / decoding processing unit 311, the display output IF 312, and the image output IF 313 are mutually connected by a bus controller 314.

  The CPU 301 is an arithmetic processing device that controls the entire encoding device 101. The ROM 302 is a non-volatile memory that stores programs such as a boot program of the encoding apparatus 101. A RAM 303 is a volatile memory used as a work area for the CPU 301.

  The image sensor 304 converts light emitted from the object into an electrical signal. For example, the imaging device 304 is a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like.

  The image sensor IF 305 controls the image sensor 304 at the time of recording, thereby converting a signal from the image sensor 304 into a predetermined image format and storing it in the RAM 303. The predetermined image format is, for example, YUV420. The signal from the image sensor 304 is in accordance with, for example, the RGB Bayer format.

  The gyro sensor 306 detects the angle of the encoding device 101. An operation panel controller 307 is a liquid crystal touch panel and operation buttons included in the encoding device 101. The recording medium 308 is a storage device such as a flash ROM, and records a stream 121 including an encoded frame. Further, the recording medium 308 may record the encoding program according to the present embodiment. The external IF 309 controls the gyro sensor 306, the operation panel controller 307, and the recording medium 308. Further, the external IF 309 may be connected to a device other than the encoding device 101 via a communication line such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet.

  The image processing unit 310 performs image processing on the image stored in the RAM 303. Specifically, at the time of recording, the image processing unit 310 reads the input image 111 from the image sensor 304 from the RAM 303, and based on the vertical and horizontal information specified by the CPU 301, the base view image and the extended view image are input from the input image 111. Is extracted and stored in the RAM 303 again.

  The vertical / horizontal information is information related to the angle of the encoding apparatus 101. The angle of the encoding device 101 is the angle of the encoding device with respect to the horizontal plane. For example, when the encoding device 101 is perpendicular to the horizontal plane, the information related to the angle of the encoding device 101 is information that the image is oriented vertically. Further, when the encoding device 101 is parallel to the horizontal plane, the information regarding the angle of the encoding device 101 is information indicating that the device is facing sideways.

  Also, the image processing unit 310 reads the corresponding base view decoded image and extended view decoded image from the RAM 303 based on the vertical / horizontal information and the dummy frame information specified by the CPU 301 during reproduction, and the base view decoded image. And the decoded image of the extended view. Next, the image processing unit 310 cuts out an image of the display area designated by the user, and then stores the image in the RAM 303 again. The dummy frame information is information on the number and position of dummy frames set to delay the base view.

  The MVC encoding / decoding processing unit 311 obtains the base view image and the extended view image designated by the processor at the time of recording from the RAM 303, and outputs the H.264 image. H.264 MVC encoding processing is performed. A stream 121 having an encoded frame is written to the recording medium 308 through the external IF 309. In addition, the MVC encoding / decoding processing unit 311 decodes the encoded frame in the stream 121 from the recording medium 308 and stores the decoded image in the RAM 303 at the time of reproduction. In addition, the MVC encoding / decoding processing unit 311 records vertical / horizontal information and dummy frame information in the stream 121 in the CPU 301.

  The image processing unit 310 and the MVC encoding / decoding processing unit 311 according to the present embodiment may be realized by an application-specific IC such as a structured ASIC (Application Specific Integrated Circuit) in order to perform processing at high speed. Further, the encoding apparatus 101 may cause the CPU 301 to execute processing in the image processing unit 310 and the MVC encoding / decoding processing unit 311.

  The display 102 displays a base view image during recording. The display 102 may display an image in a display area designated by the user, which is output from the image processing unit 310 during reproduction. For example, the display 102 can employ a liquid crystal display or the like. The display output IF 312 controls the display 102.

  The image output IF 313 reads the image of the display area designated by the user output from the image processing unit 310 at the time of reproduction from the RAM 303 and outputs the image of the display area to a television receiver or the like connected to the encoding device 101. .

(Hardware configuration example of decoding device 400)
FIG. 4 is a block diagram illustrating an example of a hardware configuration of the decoding device. In FIG. 4, a decoding device 400 that reproduces a moving image includes a CPU 401, a ROM 402, a RAM 403, an operation panel controller 404, a recording medium 405, and an external IF 406. In addition, the decoding device 400 includes an image processing unit 407, an MVC decoding processing unit 408, and an image output IF 409. The CPU 401 to RAM 403 and the external IF 406 to the image output IF 409 are connected to each other by a bus controller 410.

  The CPU 401 governs overall control of the decoding device 400. The ROM 402 stores a program such as a boot program for the decoding device 400. The RAM 403 is used as a work area for the CPU 401.

  The operation panel controller 404 is a liquid crystal touch panel or operation buttons that the decoding device 400 has. The recording medium 405 is a storage device such as a flash ROM, and stores a stream 121 including an encoded frame. Further, the recording medium 405 may record the decryption program according to the present embodiment. The external IF 406 controls the operation panel controller 404 and the recording medium 405. The external IF 406 may be connected to a network such as a LAN, WAN, or the Internet via a communication line, and may be connected to another device other than the decoding device 400 via the network.

  The image processing unit 407 reads the corresponding base view decoded image and extended view decoded image from the RAM 403 based on the vertical / horizontal information and dummy frame information specified by the CPU 401, and decodes the base view decoded image and the extended view decoded image. Merge images. Next, the image processing unit 407 cuts out an image of the display area designated by the user, and then stores the image in the RAM 303 again.

  The MVC decoding processing unit 408 decodes the encoded frame in the stream 121 from the recording medium 405 and stores the decoded image in the RAM 403. In addition, the MVC decoding processing unit 408 notifies the CPU 401 of vertical / horizontal information and dummy frame information in the stream 121.

  Although an example of connection of the moving image processing system is not shown, for example, the encoding device 101 and the decoding device 400 are connected via an external IF 309 and an external IF 406.

(Functional configuration examples of the encoding device 101 and the decoding device 400)
Next, functional configuration examples of the encoding device 101 and the decoding device 400 will be described. FIG. 5 is a block diagram illustrating a functional configuration example of the encoding device 101 and the decoding device 400. The encoding apparatus 101 includes a reception unit 501, an acquisition unit 502, a determination unit 503, an extraction unit 504, a first encoding unit 505, a second encoding unit 506, a generation unit 507, and a setting unit 508. And an output unit 509. Decoding device 400 includes reception unit 521, selection unit 522, designation unit 523, conversion unit 524, decoding unit 525, extraction unit 526, combining unit 527, and output unit 528.

  In the present embodiment, of the receiving unit 501 to the output unit 509 of the encoding apparatus 101, the extraction unit 504 is a function of the image processing unit 310, and the first encoding unit 505 and the second encoding unit 506 are MVC codes. This is a function of the decoding / decoding processing unit 311. Regarding the functional units other than the extraction unit 504 to the second encoding unit 506, the CPU 301 executes the program stored in the storage device, thereby realizing the function. Specifically, the storage device is, for example, the ROM 302, the RAM 303, the recording medium 308, or the like shown in FIG. The image processing unit 310 and the MVC encoding / decoding processing unit 311 may execute functional units other than the extraction unit 504 to the second encoding unit 506.

  Of the receiving unit 521 to the output unit 528 of the decoding device 400, the decoding unit 525 is a function of the MVC decoding processing unit 408, and the extraction unit 526 and the combining unit 527 are functions of the image processing unit 407. Regarding the functional units other than the decoding unit 525 to the combining unit 527, the CPU 401 executes the program stored in the storage device, thereby realizing the function. Specifically, the storage device is, for example, the ROM 402, the RAM 403, the recording medium 405, etc. shown in FIG. Further, the image processing unit 407 and the MVC decoding processing unit 408 may execute functional units other than the extraction unit 504 to the second encoding unit 506.

  The accepting unit 501 accepts a series of images. For example, the reception unit 501 receives an input moving image. The encoding apparatus 101 can detect a trigger for encoding the input moving image by the operation of the reception unit 501. The received input image is stored in a storage area such as the RAM 303.

  The acquisition unit 502 acquires information about the angle of the own device when a series of images is received by the reception unit 501. Here, the own device may be a device that executes the acquisition unit 502 or a device that is physically connected to a device that executes the acquisition unit 502. The information regarding the angle may be an angle of the own device such as 0 degrees or 90 degrees, or may be information such that the own device is inclined horizontally or vertically. For example, the acquisition unit 502 acquires the angle of the encoding device 101 using the gyro sensor 306.

  The user performs horizontal shooting if the shape of the shooting target a is horizontally long, and performs vertical shooting if the shape of the shooting target a is vertically long. Therefore, the encoding apparatus 101 can perform an encoding process in accordance with the shape of the imaging target a by the operation of the acquisition unit 502. The acquired information regarding the angle of the own device is stored in a storage area such as the RAM 303.

  The determining unit 503 determines the position information of the first and second regions according to the angle of the own device acquired by the acquiring unit 502. The first area is, for example, a base view, and the second area is an extended view. The extended view shown in FIG. 1 is adjacent to the left and right of the base view, but may be, for example, an area adjacent to the left end of the base view or an image adjacent to the upper end, right end, and lower end.

  For example, it is assumed that the information related to the angle of the own device indicates sideways. At this time, the determination unit 503 determines the position information of the first region to be a rectangular region in which the upper left coordinates are (1k, 0.5k) and the vertical 1k [pixel] and the horizontal 2k [pixel]. Further, the determination unit 503 obtains the position information of the second area, the upper left coordinates are (0, 0.5k), the rectangular area is 1k [pixel] in length and 1k [pixel] in width, and the upper left coordinates are (3k, 0.5k), and a rectangular area having 1k [pixel] in length and 1k [pixel] in width is determined. The encoding apparatus 101 can determine an imaging region corresponding to the shape of the imaging target a by the operation of the determination unit 503. The determined position information is stored in a storage area such as the RAM 303.

  The extraction unit 504 extracts an image in the first area and an image in the second area adjacent to the first area from any one of the series of images. For example, at time t0 illustrated in FIG. 2, the extraction unit 504 extracts the image b in the first region and the image d in the second region from the input image 111. The extraction unit 504 may extract an image in the first area according to the position information and an image in the second area according to the position information from any of the images. The encoding apparatus 101 uses the operation of the extraction unit 504 to perform H.264 encoding. Images of multiple views according to the H.264 MVC standard can be prepared. Note that the extracted image is stored in a storage area such as the RAM 303.

  The first encoding unit 505 encodes the image in the first area into an encoded frame for the first area. For example, the first encoding unit 505 encodes the image b into encoded frames such as an I picture, a P picture, and a B picture using inter-frame prediction. The encoding apparatus 101 can generate a moving image in the first region by the operation of the first encoding unit 505. Note that the encoded frame is stored in a storage area such as the recording medium 308.

  The second encoding unit 506 converts the image in the second area into an encoded frame for the second area based on the local decoded image at the time of encoding the first area and the image in the second area. Encode. The second encoding unit 506 also performs preceding encoding that precedes the number of generations of dummy frames from the preceding encoded frame among the preceding encoded frame groups preceding the encoded frame for the first region of any image. You may encode based on the local decoded image of a flame | frame. Here, the preceding encoded frame group is an encoded frame group preceding the encoded frame for the first region of any image. A local decoded image is an image used when encoding a frame to be referenced. It does not go through the encoding / decoding process in the stream 121.

  For example, it is assumed that the setting unit 508 sets three dummy frames as a frame next to the first encoded frame for the first region. At this time, the second encoding unit 506 is based on the image in the second region of any image and the image obtained by decoding the encoded frame three previous to the first region of any image. Thus, an image in the second region of any image is encoded into an encoded frame.

  Further, when the dummy frame has been set, the second encoding unit 506 encodes an image in the second region of any image into the encoded frame without generating the dummy frame by the generating unit 507. May be.

  For example, it is assumed that one of the images is the fifth image, and the setting unit 508 sets three dummy frames after the first frame in the first area. At this time, the encoded frame in the first area of the fifth image is at the eighth position from the beginning because the dummy frame has already been set. The second encoding unit 506 executes the fifth based on the image in the second area of the fifth image and the encoded frame three previous to the encoded frame in the first area of the fifth image. The image in the second region of the image is encoded. Note that the second encoding unit 506 may encode using the difference between the image in the second area and the decoded image, or perform motion compensation from the image combined with the image in the second area. Encoding may be performed.

  The second encoding unit 506 also performs subsequent encoding that follows the number of generations of dummy frames from the above-described encoded frame in the subsequent encoded frame group subsequent to the encoded frame for the first region of any image. The frame may be decoded and encoded based on the decoded image. Here, the subsequent encoded frame group is an encoded frame group that follows the encoded frame for the first region of any image.

  For example, it is assumed that the setting unit 508 sets two dummy frames as the first encoded frame for the second region. At this time, the second encoding unit 506 sets the fourth encoded frame for the second region as the local of the fourth encoded frame in the frame group subsequent to the second encoded frame of the first region. Encoding is performed based on the decoded image and the second image in the second area. The encoding apparatus 101 can generate a moving image in the second region by the operation of the second encoding unit 506. Note that the encoded frame is stored in a storage area such as the recording medium 308.

  When the first encoding unit 505 encodes any image, the generation unit 507 reproduces the dummy frame of the encoded frame for the first region of the image before the any image with the self-encoded frame. As many as the number of frames included in the appearance interval of possible specific encoded frames are generated. In addition, the generation unit 507 generates a dummy frame based on the image in the first area among the previous images.

  The specific encoded frame that can be reproduced by the self-encoded frame is an I picture or an IDR (Instantaneous Decoder Refresh) picture. An IDR picture prohibits reference to a picture preceding the IDR picture in the subsequent picture. Accordingly, the IDR picture appearance interval is the maximum referenceable number. The generation unit 507 may generate one specific encoded frame, or may generate as many as the number of frames included in the IDR picture appearance interval.

  The preceding image group of any image is an image group existing in the past direction from any image. The dummy frame may be a frame that refers to an encoded frame for the first region of the previous image. For example, if the encoded frame for the first region of the previous image is an I picture, the dummy frame is a P picture or a B picture, and a skip macroblock is designated.

  For example, it is assumed that the first encoding unit 505 encodes the first image in the first area and encodes the second image. At this time, when encoding the second image to be any image, the generation unit 507 inputs the first image to be the previous image to the first encoding unit 505 as the number of generations three times. As a result, the first encoding unit 505 generates three dummy frames.

  In addition, the generation unit 507 generates dummy frames by a number one more than the number of encoded frames having a plurality of reference destinations in the encoded frame group encoded by the first encoding unit 505. May be. An encoded frame having a plurality of reference destinations is a B picture. For example, when two B pictures are set in succession, the generation unit 507 generates three dummy frames.

  In addition, when the second encoding unit 506 encodes any image, the generation unit 507 can reproduce a dummy frame for the second area of the previous image of any image with the self-encoded frame. The number of frames included in the appearance interval of a specific encoded frame is generated. For example, it is assumed that the second encoding unit 506 encodes the first image in the second area and encodes the second image. At this time, the generation unit 507 inputs the first image, which is one of the images, to the second encoding unit 506 as the generation number twice. As a result, the second encoding unit 506 generates two dummy frames. The encoding apparatus 101 can cause a frame delay by the operation of the generation unit 507. The generated frame is stored in a storage area such as the RAM 303.

  The setting unit 508 sets the dummy frame as the next encoded frame after the encoded frame for the first region of the reference destination image. For example, the setting unit 508 sets a dummy frame next to the first encoded frame for the first region.

  Further, the setting unit 508 may set the dummy frame as the next encoded frame of the encoded frame for the second region of the reference destination image. For example, the setting unit 508 sets a specific encoded frame next to the first encoded frame for the second region. The encoding apparatus 101 can set the position where the frame delay is generated by the operation of the setting unit 508.

  The output unit 509 adds identification information 124 indicating at least one of the first and second regions to the corresponding encoded frame, and outputs the encoded frames for the first and second regions. Further, the output unit 509 detects the identification information 124 as an output destination and outputs it to a device that decodes the encoded frame for the first region. A device that detects the identification information 124 and decodes the encoded frame for the first region is, for example, the decoding device 400. The identification information 124 may indicate a first area or a second area. Specifically, H.M. In H.264 MVC, a NAL unit type, which will be described later, is set to be a frame obtained by encoding an image in a base view or a frame obtained by encoding an image in an extended view.

  The output unit 509 may output the specific encoded frame generated by the generation unit 507 and information indicating the number of generated specific encoded frames and the set position. For example, the information indicating the set position may indicate the start position and the number of a specific encoded frame, or may indicate the position of each encoded frame. The output unit 509 may output information related to the angle of the own device. The encoding apparatus 101 can output the encoded frame to the outside by the operation of the output unit 509. The output unit 509 may output the output result to a storage device such as the recording medium 308 and the RAM 303 as an output destination, or may output to an external device via the external IF 309.

  The reception unit 521 includes a series of encoded frames for a series of partial images in the first area in the series of images, and a series of partial images in the second area adjacent to the first area in the series of images. A series of encoded frames. Furthermore, the reception unit 521 receives the identification information 124 corresponding to one of a series of encoded frames for a series of partial images in the first or second region. In addition, the reception unit 521 may receive information indicating the number and position of encoded frames that are not output targets among a series of encoded frames for a series of partial images in the first or second region.

  The receiving unit 521 may receive information related to the angle of the device when the device that has received a series of images has received a series of images. For example, the accepting unit 521 accepts that the device that accepted the image is vertical. The decoding apparatus 400 can detect a trigger for decoding a series of encoded frames by the operation of the reception unit 521. Each received information is stored in a storage area such as the RAM 403.

  The selection unit 522 selects a series of encoded frames for the series of partial images in the first area from the series of encoded frames for the series of partial images in the first and second areas, based on the identification information 124. select. For example, when the identification information 124 indicates the first area, the selection unit 522 selects a series of encoded frames to which the identification information 124 is added. When the identification information 124 indicates the second area, the selection unit 522 selects a series of encoded frames to which the identification information 124 is not added. The decoding apparatus 400 can limit the decoding target to the base view by the operation of the selection unit 522. The selected encoded frame is stored in a storage area such as the RAM 403.

  The designation unit 523 designates a display area from an area obtained by combining the first area and the second area. For example, the designation unit 523 designates a display area that is moved rightward from the initial display area by the user. The user who operates the decoding device 400 operates the display area so that the photographing object a is reflected. Therefore, the decoding device 400 can perform a decoding process in accordance with the display position of the shooting target object a by the operation of the specifying unit 523. Note that the designated display area is stored in a storage area such as the RAM 403.

  The conversion unit 524 converts the designated display area according to the information regarding the angle. For example, when the information regarding the angle indicates the vertical direction and the display area is designated to move in the right direction, the conversion unit 524 converts the display area so as to move in the downward direction. The decoding device 400 can display the decoded image with the same screen orientation as that in the state in which the shooting target object a is shot, by the operation of the conversion unit 524. Note that the converted display area is stored in a storage area such as the RAM 403.

  The decoding unit 525 decodes a series of encoded frames for the first region selected by the selection unit 522 into a series of partial images in the first region. For example, the decoding unit 525 decodes an I picture, a P picture, and a B picture as each frame for the first region. Further, when the display area is designated by the designation unit 523, the decoding unit 525 may decode a series of encoded frames for the second area into a series of partial images in the second area. The decoding device 400 can reproduce a moving image of a series of images by the operation of the decoding unit 525. The decoded image is stored in a storage area such as the RAM 403.

  The extraction unit 526 extracts the first image sequence in the display area from the series of partial images in the first area, and extracts the second image sequence in the display area from the series of partial images in the second area. To extract. An image sequence is data in which a plurality of images are arranged in chronological order. For example, the first image sequence is arranged in chronological order, such as an image in the display area at the 0th time, an image in the display area at the first time, and an image in the display area at the second time. It becomes data.

  In addition, the extraction unit 526 extracts the first partial image in the display region from the partial images other than the partial image corresponding to the encoded frame that is not the output target among the series of partial images in the first region decoded by the decoding unit 525. One image sequence is extracted. Note that the extraction unit 526 performs extraction based on information indicating the number and position of encoded frames that are not output targets. Further, the extraction unit 526 may extract the second image sequence in the display area from the series of partial images in the second area.

  For example, it is assumed that information indicating the number and position of encoded frames that are not output targets indicates the start position and number of encoded frames that are not output targets. At this time, the extraction unit 526 sets the encoded frame corresponding to the head position and the encoded frame from the corresponding encoded frame to the immediately preceding encoded frame as the encoded frame that is not the output target, and the code that is not the output target. Partial images are extracted from other than the frame. Further, it is assumed that information indicating the number and position of encoded frames that are not output targets indicates the positions of encoded frames that are not output targets. At this time, the extraction unit 526 performs extraction on a partial image obtained by decoding an encoded frame other than the encoded frame corresponding to each position.

  For example, when the information indicating the number and position of encoded frames that are not output targets indicates the first to third, and the decoding unit 525 decodes images 0 to 4 as partial images in the first region. The extraction unit 526 extracts the first image sequence in the display area from the images 0 and 4.

  In addition, the extraction unit 526 extracts the first image sequence in the display area after conversion by the conversion unit 524 from the series of partial images in the first area, and from the series of partial images in the second area. You may extract the 2nd image sequence in the display area after conversion. The decoding device 400 can acquire an image sequence that is the range of the display area by the processing of the extraction unit 526. Note that the extracted image sequence is stored in a storage area such as the RAM 403.

  The combining unit 527 combines each image of the first image sequence extracted by the extraction unit 526 and each image of the second image sequence with the same image in the series of images. For example, the combining unit 527 combines the 0th of the first image sequence and the 0th of the second image sequence.

  In addition, the combining unit 527 selects other images other than the image corresponding to the encoded frame that is not the output target from among the images of the first image sequence based on the information indicating the number and position of the encoded frames that are not the output target. You may combine each image of a 2nd image sequence by the same image in a series of images.

  For example, it is assumed that information indicating the number and position of encoded frames that are not output targets indicates the start position and number of encoded frames that are not output targets. At this time, the combining unit 527 includes an image other than the image corresponding to the encoded frame corresponding to the head position and the encoded frame from the corresponding encoded frame to the immediately preceding encoded frame, and the second Are combined with each other in the series of images. Further, it is assumed that information indicating the number and position of encoded frames that are not output targets indicates the positions of encoded frames that are not output targets. At this time, the combining unit 527 converts the image other than the image corresponding to the encoded frame other than the encoded frame corresponding to each position, and each image of the second image sequence into the same image in the series of images. Join each other.

  The decoding device 400 can reproduce a moving image corresponding to the display area by the operation of the combining unit 527. Note that the extracted image sequence is stored in a storage area such as the RAM 403.

  The output unit 528 outputs a decoded image other than the decoded image corresponding to the encoded frame that is not the output target based on the information indicating the number and position of the encoded frames that are not the output target in the series of decoded images. For example, it is assumed that information indicating the number and position of encoded frames that are not output targets indicates the start position and number of encoded frames that are not output targets.

  At this time, the output unit 528 sets the encoded frame corresponding to the head position and the encoded frame from the corresponding encoded frame to the immediately preceding encoded frame as the encoded frame that is not the output target, and the code that is not the output target. A decoded image for other than the frame is output. Further, it is assumed that information indicating the number and position of encoded frames that are not output targets indicates the positions of encoded frames that are not output targets. At this time, the output unit 528 outputs a decoded image obtained by decoding an encoded frame other than the encoded frame corresponding to each position.

  Further, the output unit 528 may output the image sequence of the display area obtained by combining the combining unit 527, or output the image sequence of the display area after conversion obtained by combining the combining unit 527. Also good. The encoding apparatus 101 can output the encoded frame to the outside by the operation of the output unit 509. Note that the output unit 528 may output the output result to a storage device such as the RAM 403 as an output destination, or may output it to a television receiver or the like via the image output IF 409.

  Next, an example of extracting the input image 111 will be described. As the extraction method of the input image 111, in this embodiment, there are an extraction method at the time of horizontal position shooting and an extraction method at the time of vertical position shooting. FIG. 6 illustrates an example of a screen extraction method during horizontal position shooting, and FIG. 7 illustrates an example of a screen extraction method during vertical position shooting.

  FIG. 6 is an explanatory diagram showing an example of screen extraction at the time of horizontal position shooting. As shown by (A) in FIG. 6, the encoding apparatus 101 reads from the input image 111 the image b in the base view that is the initial display area when not specified by the user and the extended view included in the extended area. The image d_L and the image d_R are extracted. Note that the image d_L is an image adjacent on the left side of the base view, and the image d_R is an image adjacent on the right side of the base view.

  For example, the encoding apparatus 101 converts a rectangular image having an upper left coordinate (1k, 0.5k), 1k [pixel], and 2k [pixel] from the input image 111 as the image b in the base view. Extract. Also, the encoding device 101 extracts a rectangular image whose upper left coordinates are (0, 0.5k) and whose vertical length is 1k [pixel] and horizontal 1k [pixel] as the image d_L in the extended view. Further, the encoding apparatus 101 extracts a rectangular image having an upper left coordinate (3k, 0.5k) and a vertical 1k [pixel] and a horizontal 1k [pixel] as the image d_R in the extended view.

  The encoding apparatus 101 performs such extraction processing for each input image 111. For example, as indicated by (B) in FIG. 6, the encoding apparatus 101 extracts an image b0 and an image d0 obtained by combining the image d0_L and the image d0_R from the input image 111 at time t0. Similarly, the encoding apparatus 101 extracts an image b1 and an image d1 obtained by combining the image d1_L and the image d1_R from the input image 111 at time t1. Also, the encoding apparatus 101 extracts the image b2 and the image d2 obtained by combining the image d2_L and the image d2_R from the input image 111 at time t2.

  FIG. 7 is an explanatory diagram illustrating an example of a screen extraction method during vertical position shooting. When vertical position shooting is performed, the encoding apparatus 101 extracts the input image 111 as an extraction image indicated by (A) in FIG. Specifically, the input image 111 has a length of 2k [pixels] and a width of 4k [pixels], and is included in the image b in the base view and the extended region as an initial display region when not specified by the user. The image d_L and the image d_R in the extended view are extracted.

  In order to extract as indicated by (A) in FIG. 7, the encoding apparatus 101 extracts the input image 111 as indicated by (B) in FIG. Specifically, as the image b in the base view, a rectangular image whose upper left coordinates are (1k, 0.5k) and whose vertical length is 1k [pixel] and horizontal 2k [pixel] is extracted from the input image 111. Also, the encoding apparatus 101 extracts a rectangular image whose upper left coordinates are (1k, 0) and whose vertical length is 0.5k [pixel] and horizontal 2k [pixel] as the image d_L in the extended view. Furthermore, the encoding apparatus 101 extracts a rectangular image whose upper left coordinates are (1k, 1.5k) and whose vertical length is 0.5k [pixel] and horizontal 2k [pixel] as the image d_R in the extended view.

  The encoding apparatus 101 performs such extraction processing for each input image 111. The extraction method from time t0 to time t2 indicated by (C) in FIG. 7 is the same as that in FIG. Hereinafter, when there is no description in particular, the description will be made with an example of performing an extraction method at the time of horizontal position photographing.

  Note that whether the image was taken in the horizontal position as shown in FIG. 6 or the image was taken in the vertical position as shown in FIG. 7 is recorded as vertical / horizontal information. The vertical / horizontal information will be described later with reference to FIG.

  FIG. 8 is an explanatory diagram illustrating an example of a reference direction for motion compensation. (A) in FIG. 8 shows that the objects (1) and (2) move in the input image 111, and (B) in FIG. 8 shows the reference direction of the motion vector. .

  The object (1) exists in the image d0 and the image d1 in the extended view at both time t0 and time t1. When the image d1 is encoded, the encoding device 101 can refer to the image d0 as shown in Table 201, so that the encoding efficiency for the object (1) is improved.

  The object (2) exists in the image b0 in the base view at time t0, and exists in the image d1 in the extended view at time t1. When the image d1 is encoded, the encoding device 101 cannot refer to the image b0 as shown in Table 201, and thus the encoding efficiency of the object (2) is deteriorated.

  As described above, when the movement between different views occurs, the encoding efficiency deteriorates. Therefore, in this embodiment, by inserting a dummy frame, the images of the base view and the extended view can be referred to at the same time, thereby improving the coding efficiency. A case where a dummy frame is set will be described with reference to FIG.

  FIG. 9 is an explanatory diagram illustrating an example of a reference destination when a dummy frame is set. (A) in FIG. 9 shows an example of moving from the image b in the base view to the image d_R in the extended view, like the object (2) shown in FIG.

  Next, (B) in FIG. 9 shows an encoded frame group in the base view and an encoded frame group in the extended view in a state where no dummy frame is set. Specifically, in (B) of FIG. 9, the encoded frame fb0 for the base view at time t0, the encoded frame fd0 for the extended view, the encoded frame fb1 for the base view at time t1, and the encoded frame fd1 for the extended view. There is. Furthermore, there are an encoded frame fb2 for the base view at time t2, an encoded frame fd2 for the extended view, an encoded frame fb3 for the base view at time t3, and an encoded frame fd3 for the extended view.

  The encoded frame fb0 is an I picture, the encoded frames fb3, fd0, and fd3 are P pictures, and the encoded frames fb1, fb2, fd1, and fd2 are B pictures.

  At this time, since the encoded frame fd1 to the encoded frame fd3 are not at the same time as the encoded frame fb0, when the encoding apparatus 101 encodes the encoded frame fd1 to the encoded frame fd3, reference is impossible. The encoding efficiency deteriorates.

  Next, in (C) of FIG. 9, encoded frames fb1_d, fb2_d, and fb3_d that are dummy frames are set after the encoded frame fb0. The entity of the dummy frame is a set of skip macroblocks, for example. The skip macroblock is a macroblock indicating that the reference source image is the same as the reference destination image. Also, the encoded frames fb1 to fb3 subsequent to the encoded frame fb0 are an encoded frame fb4_1 at time t4, an encoded frame fb5_2 at time t5, and an encoded frame fb6_3 at time t6, respectively.

  At this time, the encoded frame fd3 is at the same time as the encoded frame fb3_d. Therefore, when encoding the encoded frame fd3, the encoding device 101 can refer to the encoded frame fb3_d, and the encoding efficiency is improved. Next, a setting example of dummy frames in the stream 121 will be described with reference to FIG.

  FIG. 10 shows an example of setting dummy frames in a stream. In FIG. 10A and FIG. 10B, solid arrows indicate reference frames, indicating that the images connected by dotted lines are extracted from the same input image 111. .

  In (A) of FIG. 10, the encoding apparatus 101 performs a dummy frame using the I picture at the time t0 indicated by (1) in the encoded frame for the base view, from time t1 to time indicated by (2). A P picture and a B picture at t3 are set. Also, the encoding apparatus 101 uses the P frame and the B picture at time tn + 1 to time tn + 3 indicated by (4) as dummy frames using the P picture at time tn indicated by (3) in the encoded frame for the extended view. Set.

  As described above, in FIG. 10A, the encoding apparatus 101 sets the same number of dummy frames as the encoded frames for the base view in the encoded frames for the extended view. As a result, the number of encoded frames for the extended view and the number of encoded frames for the base view are the same. For example, when the decoding apparatus 400 checks whether the number of encoded frames for the extended view and the number of encoded frames for the base view are the same, the encoding apparatus 101 sets the number of frames to be the same and the stream 121 that can pass the check. Can be generated.

  In FIG. 10A, a dummy frame is set in the base view so that the encoded frame for the extended view can be referred to in the past direction of the base view. In order to allow the encoded frame for the extended view to be referenced in the future direction of the base view, a dummy frame may be set in the extended view. An example in which a dummy frame is set in the extended view is shown in FIG.

  In (B) of FIG. 10, the encoding apparatus 101 sets the time t1 and time indicated by (6) as a dummy frame using the P picture at time t0 indicated by (5) in the encoded frame for the extended view. A B picture at t2 is set. Also, the encoding apparatus 101 sets B pictures at time tm + 1 and time tm + 2 indicated by (8) as dummy frames using the P picture indicated by (7) at time tm in the encoded frame for the base view. . The position and number where the dummy frame is set are recorded as dummy frame information. The dummy frame information will be described later with reference to FIG.

  FIG. 11 is an explanatory diagram showing a setting example of vertical / horizontal information and dummy frame information. FIG. 11 shows an example in which vertical / horizontal information and dummy frame information are set in the stream 121. The encoding apparatus 101 sets a User data unregistered SEI (Supplemental Enhancement Information) message 1101 in the stream 121. The User data unregistered SEI message 1101 is hereinafter simply referred to as an SEI message 1101. The SEI message 1101 is one of NAL (Network Abstraction Layer) units and is a SEI message that can be defined by the user.

  The SEI message 1101 includes a NAL header field 1111 and an RBSP field (Raw Byte Sequence Payload) 1112. The NAL header field 1111 stores the NAL unit type and the like. The RBSP field 1112 is a payload portion of the NAL unit.

  The RBSP field 1112 includes a payload_type field 1121, a payload_size field 1122, a uuid_iso_iec_11578 field 1123, and a user_data_payload_byte field 1124. The payload_type field 1121 stores the type stored in the RBSP field 1112, for example, 5 indicating user_data_unregistered. The payload_size field 1122 stores the size of the SEI message 1101. The uuid_iso_iec_11578 field 1123 stores identification information that can be uniquely identified.

  The user_data_payload_byte field 1124 stores a value unique to the user. In the present embodiment, the user_data_payload_byte field 1124 includes a vertical / horizontal information field 1131 and a dummy frame information field 1132. For example, in FIG. 11, the vertical and horizontal information field 1131 stores an identifier indicating horizontal, and the dummy frame information field 1132 stores 3 as the number of frames.

  As a position where the dummy frame is set, for example, the SEI message 1101 may be set before the dummy frame. In this case, the set position of the SEI message 1101 indicates the position where the dummy frame is set. For example, in the stream 121 shown by (A) in FIG. 10, the SEI message 1101 is set before the encoded frame for the base view at time t1 and before the encoded frame for the extended view at time tn + 1.

  In FIG. 11, vertical and horizontal information and dummy frame information are set in the stream 121, but the encoding apparatus 101 may write vertical and horizontal information and dummy frame information in a file associated with the stream 121.

  FIG. 12 is an explanatory diagram illustrating an example of the encoding process. The encoding apparatus 101 executes the encoding process. The image processing unit 310 in the encoding apparatus 101 receives a 4k2k input image 111, executes input image extraction processing using vertical and horizontal information, and outputs an image in the base view and an image in the extended view. Subsequently, the MVC encoding / decoding processing unit 311 performs MVC encoding processing from the aspect information, the image in the base view, and the image in the extended view, and the encoded frame 122 and the extended view in the base view Is output to the stream 121. Within the MVC encoding process, the MVC encoding / decoding processing unit 311 executes a motion vector search range determination process. An example of the motion vector search range is shown in FIGS. 13A, 13B, 14A, and 14B.

  13A and 13B are explanatory diagrams illustrating an example of a motion vector search range at the time of horizontal position shooting. In (A) of FIG. 13A, the MVC encoding / decoding processing unit 311 sets the motion vector search range of each macroblock to a certain range centered on the current macroblock. Hereinafter, the macroblock is referred to as MB.

  Since the motion vector search range in each MB differs depending on the MB position in the screen, the search range is determined based on the following parameters. First, with the current MB as the center, the number of pixels to the left end of the motion vector search range is lx, the number of pixels to the upper end is uy, the number of pixels to the right end is rx, and the number of pixels to the lower end is dy. For example, lx = uy = rx = dy = 16.

  13A, mbnumx is the number of horizontal pixels / 16 of the base view and the extended view, and mbnumy is the number of vertical pixels / 16 of the base view and the extended view. Also, let mbadrx be the MB address x of the current MB. mbadrx indicates the horizontal position in MB units in the screen. Further, let mbadry be the MB address y of the current MB. mbadry indicates the vertical position in MB units in the screen.

  For example, if the image in the base view and the image in the extended view are 1920 × 1088 pixels, mbnumx is 1920/16 = 120, and mbnumy is 1088/16 = 68. The range that mbadrx can take is 0 to 119, and the range that mbadry can take is 0 to 67. Also, for images in the extended view, when mbadrx is 0 to 59, the current MB is in the image d_L, and when mbadrx is 60 to 119, the current MB is in the image d_R.

  Next, in FIG. 13B, (A) in FIG. 13B shows a specific example of a vector search range corresponding to the position of each MB. For MB_a to MB_i, MB_a to MB_c are MBs in the base view, and for MB_d to MB_i, they are MBs in the extended view.

  In the motion vector search range of MB_a, since MB_a exists at the left end of the image b and the image d_L in the extended view cannot be referred to, lx becomes 0. The motion vector search range of MB_b remains lx = uy = rx = dy = 16 because MB_b exists inside the image b. In the motion vector search range of MB_c, rx is 0 because MB_c exists at the right end of the image b and the image d_R in the extended view cannot be referred to.

  In the motion vector search range of MB_d, MB_d exists at the upper end in the image d_L, and uy is 0. The motion vector search range of MB_e remains lx = uy = rx = dy = 16 because MB_e exists inside the image d_L.

  In the motion vector search range of MB_f, since MB_f exists at the right end in the image d_L and the image b in the base view at the same time can be referred to, rx does not become 0 but remains 16. Further, the vector search range to the image b corresponding to the region (R) shown by (B) in FIG. 13B is converted from the vector search range of the image d_L. Assuming that the number of horizontal image pixels of the base view and the extended view is picsize, the vertical image size is picsize, the coordinates of the reference pixels before conversion are px, py, and the coordinates of the reference pixels after conversion are ox, oy, as follows: Is required.

ox = px-picsizeex / 2
oy = py

  In the motion vector search range of MB_g, since MB_g exists at the left end in the image d_R and the image b in the base view at the same time can be referred to, lx does not become 0 but remains 16. Further, the vector search range to the image b corresponding to the region (L) shown by (B) in FIG. 13B is converted from the vector search range of the image d_R, and is obtained as follows.

ox = px + picsizeex / 2
oy = py

  The motion vector search range of MB_h remains lx = uy = rx = dy = 16 because MB_h exists inside the image d_R. In the motion vector search range of MB_i, rx and dy are 0 because MB_i exists at the lower right of the image d_R.

  14A and 14B are explanatory diagrams illustrating an example of a motion vector search range during vertical position shooting. The vector search range of MB shown in (A) in FIG. 14A is the same as that at the time of horizontal position shooting shown in (A) of FIG.

  The definition of mbnumx, mbnumy, mbadrx, and mbadry shown by (B) in FIG. 14A is the same as the definition shown by (B) in FIG. Also, for images in the extended view, when mbadry is 0 to 33, the current MB is in the image d_L, and when mbadry is 34 to 67, the current MB is in the image d_R.

  Next, FIG. 14B shows a specific example of a vector search range corresponding to the position of each MB. For MB_a to MB_i, MB_a to MB_c are MBs in the base view, and for MB_d to MB_i, they are MBs in the extended view.

  In the motion vector search range of MB_a, uy is 0 because MB_a exists at the upper end of the image b and the image d_L in the extended view cannot be referred to. The motion vector search range of MB_b remains lx = uy = rx = dy = 16 because MB_b exists inside the image b. In the motion vector search range of MB_c, dy is 0 because MB_c exists at the lower end of the image b and the image d_R in the extended view cannot be referred to.

  In the motion vector search range of MB_d, MB_d exists at the upper end of the image d_L, and uy is 0. The motion vector search range of MB_e remains lx = uy = rx = dy = 16 because MB_e exists inside the base view.

  In the motion vector search range of MB_f, since MB_f exists at the lower end in the image d_L and the image b in the base view at the same time can be referred to, dy does not become 0 but remains 16. Further, the vector search range to the image b corresponding to the region (D) shown by (B) in FIG. 14B is converted from the vector search range of the image d_L, and is obtained as follows.

ox = px
oy = py-picsize / 2

  In the motion vector search range of MB_g, since MB_g exists at the upper end in the image d_R and the image b in the base view at the same time can be referred to, uy does not become 0 but remains 16. Further, the vector search range to the image b corresponding to the region (U) shown by (B) in FIG. 14B is converted from the vector search range of the image d_R, and is obtained as follows.

ox = px
oy = py + picsize / 2

  The motion vector search range of MB_h remains lx = uy = rx = dy = 16 because MB_h exists inside the image d_R. In the motion vector search range of MB_i, rx and dy are 0 because MB_i exists at the lower right of the image d_R.

  FIG. 15 is an explanatory diagram showing an example of the decoding process. The decoding process can be executed by the encoding apparatus 101 and the decoding apparatus 400. In the description of FIG. 15, a case where the decoding device 400 executes a decoding process will be described.

  The MVC decoding processing unit 408 in the decoding device 400 receives the stream 121, executes the MVC decoding process, and outputs a series of images in the 2k1k base view and a series of images in the 2k1k extended view. Subsequently, the image processing unit 407 executes output image combination processing from the aspect information, the series of images in the 2k1k base view, the series of images in the 2k1k extended view, and the designated display area. And a 2k1k output image is output. Next, a specific example of the output image combining process is shown in FIGS. 16A to 16C.

  FIG. 16A is an explanatory diagram (part 1) illustrating a specific example of output image combining processing. (A) in FIG. 16A shows a state in which the subject a is partially moved from the base view and enters the extended view. FIG. 16A illustrates a case where the user operating the decoding device 400 has designated the display area to move to the right by several pixels.

  (B) in FIG. 16A shows a state in which a series of encoded frames for the base view is first decoded from a series of partial images in the base view. Specifically, the decoding apparatus 400 decodes a series of encoded frames for the base view into a series of partial images dec_b0 to dec_b4 in the base view. Also, the decoding apparatus 400 decodes a series of encoded frames for the extended view into a series of partial images dec_d0 to dec_d4 in the extended view. From this state, the decoding apparatus 400 extracts an image sequence in the display area from a partial image other than the partial image corresponding to a frame that is not an output target among a series of partial images in the base view.

  For example, it is assumed that the dummy frame information indicates the second to fourth frames. At this time, the decoding apparatus 400 extracts images ext_b0 and ext_b4 in the display area from the partial images dec_b0 and dec_b4 other than the second to fourth partial images dec_b1 to dec_b3 corresponding to the times t1 to t3. Further, the decoding device 400 extracts images ext_d0 to ext_d4 in the display area from the partial images dec_d0 to dec_d4.

  Subsequently, the decoding apparatus 400 outputs an image com1 obtained by combining the image ext_b0 and the image ext_d0, and an image com2 obtained by combining the image ext_b4 and the image ext_d1.

  FIG. 16B is an explanatory diagram (part 2) of the specific example of the output image combining process. The output image combining process shown in FIG. 16A is performed after decoding and before extraction, except for a frame that is not an output target. However, the output image combining process shown in FIG. 16B extracts a process that excludes a frame that is not an output target. After and before joining. Since the process up to the decoding process is the same as that shown in FIG. 16A, the process up to the process of decoding the series of partial images dec_b0 to dec_b4 in the base view and the series of partial images dec_d0 to dec_d4 in the extended view. Description is omitted.

  After decoding, the decoding apparatus 400 extracts images ext_b0 and ext_b4 in the display area from the partial images dec_b0 to dec_b4. Further, the decoding device 400 extracts images ext_d0 to ext_d4 in the display area from the partial images dec_d0 to dec_d4.

  Subsequently, the decoding apparatus 400 combines the images ext_b0 and ext_b4 other than the second to fourth images ext_b1 to ext_b3 corresponding to the time t1 to the time t3, and outputs the combined image. Specifically, the decoding device 400 outputs an image com1 obtained by combining the image ext_b0 and the image ext_d0, and an image com2 obtained by combining the image ext_b4 and the image ext_d1.

  FIG. 16C is an explanatory diagram (part 3) of the specific example of the output image combining process. In FIG. 16C, output image combination processing when the decoding apparatus 400 receives vertical / horizontal information and the vertical / horizontal information indicates vertical is described.

  When the vertical / horizontal information indicates vertical, the decoding device 400 converts the designated display area. Specifically, when the user operating the decoding device 400 specifies that the display area moves to the right by several pixels, the decoding device 400 converts the display area so as to move down several pixels. Subsequently, the decoding, extraction, and combination processing of the encoded frame is the same as that shown in FIG. After the combining process, the decoding apparatus 400 outputs com1 and com2 that are image strings in the converted display area obtained by combining.

  Next, the flowchart of the process shown in FIGS. 16A to 16C will be described. FIG. 17 shows a flowchart of the encoding process. 18A to 21B show flowcharts of the MVC encoding process, the motion vector search range determination process in the base view, and the motion vector search range determination process in the extended view, which are executed from the encoding process. Yes. 22A and 22B show flowcharts of the decoding process. The encoding apparatus 101 performs the motion vector search range determination process in the extended view, and the encoding apparatus 101 and the decoding apparatus 400 execute the decoding process. In the present embodiment, the decoding process will be described by taking a case where the decoding apparatus 400 executes as an example.

  FIG. 17 is a flowchart illustrating an example of an encoding processing procedure. In FIG. 17, a process for encoding an input moving image will be described. The encoding apparatus 101 determines whether or not the horizontal position shooting is performed (step S1701). In addition, regarding the determination method of whether it is horizontal position imaging | photography, the angle from the gyro sensor 306 is acquired and it determines according to an angle. In the case of horizontal position shooting (step S1701: Yes), the encoding apparatus 101 sets the vertical / horizontal information horizontally (step S1702). If it is not horizontal position shooting (step S1701: No), the encoding apparatus 101 sets vertical and horizontal information vertically (step S1703).

  After the process of step S1702 or step S1703, the encoding apparatus 101 determines whether the vertical / horizontal information indicates horizontal (step S1704). When the image is horizontal (step S1704: Yes), the encoding apparatus 101 determines the position information of the base view as the center of the input image and the position information of the extended view as the left and right of the base view (step S1705). When the vertical position is indicated (step S1704: NO), the encoding apparatus 101 determines the position information of the base view at the center of the input image and the position information of the extended view at the top and bottom of the base view (step S1706).

  After the process of step S1702 or step S1703, the encoding apparatus 101 extracts, from the input image 111, an image in the base view and an image in the extended view according to the determined position information (step S1707). Next, the encoding apparatus 101 outputs the image in the base view and the image in the extended view to the RAM 303 (step S1708), and executes the MVC encoding process (step S1709).

  Subsequently, the encoding apparatus 101 sets vertical / horizontal information and dummy frame information in the stream 121 (step S1710). Next, the encoding apparatus 101 determines whether or not the photographing is finished (step S1711). If shooting has not ended (step S1711: NO), the encoding apparatus 101 proceeds to the process of step S1704. When the imaging is finished (step S1711: Yes), the encoding apparatus 101 ends the encoding process. As described above, by executing the processing illustrated in FIG. 17, the encoding apparatus 101 can encode the input moving image.

  Note that the processing of step S1707 and step S1708 is executed by the image processing unit 310, and the processing of step S1709 is executed by the image processing unit 310. For example, the CPU 301 may prepare two threads as a process for executing the encoding process. The first thread is a thread that causes the image processing unit 310 to execute the processes in steps S1707 and S1708 when the input image 111 is received. The second thread accepts that the RAM 303 is confirmed at a constant cycle or output from the first thread, and if there is an image in the base view and an image in the extended view, the MVC encoding / decoding processing unit 311 Is a thread for executing the processing of step S1709.

  18A and 18B are flowcharts illustrating an example of the MVC encoding process. 18A and 18B, processing for encoding an image in the base view and an image in the extended view will be described. First, in FIG. 18A, the encoding apparatus 101 executes a motion vector search range determination process in the base view (step S1801). Next, the encoding apparatus 101 performs MVC encoding processing on the image in the base view (step S1802). For the MVC encoding process, see H.264. Since the encoding process conforming to the H.264 MVC standard is performed, the description thereof is omitted.

  After executing the MVC encoding process and encoding any of the images, the encoding apparatus 101 outputs an encoded frame for the base view to the stream 121 (step S1803). Subsequently, the encoding apparatus 101 determines whether or not a dummy frame has been set (step S1804). If no dummy frame is set (step S1804: No), the encoding apparatus 101 encodes the first image in the base view by the number of frames of the dummy frame information (step S1805).

  Next, the encoding apparatus 101 sets the generated dummy frame as a frame next to the first encoded frame with respect to the base view (step S1806). Note that, in the processing of step S1805 and step S1806, the case where a dummy frame is set at the timing of encoding the second image has been described. For example, when a dummy frame is set at the timing of encoding the third image, the process of step S1805 is “encode the second image in the base view for the number of frames of the dummy frame information”. . Also, the processing in step S1806 is “set the generated dummy frame as the frame next to the second encoded frame for the base view”.

  Next, in FIG. 18B, after the process of step S1806 or when a dummy frame has been set (step S1804: Yes), the encoding apparatus 101 determines whether the vertical / horizontal information indicates horizontal (step S1804). S1807). If it indicates horizontal (step S1807: Yes), the encoding apparatus 101 executes a motion vector search range determination process in the extended view at the time of horizontal position shooting (step S1808). When the vertical direction is indicated (step S1807: No), the encoding apparatus 101 executes a motion vector search range determination process in the extended view at the time of vertical position shooting (step S1809).

  After executing the process of step S1808 or the process of step S1809, the encoding apparatus 101 executes the MVC encoding process for the extended view image (step S1810). The encoding apparatus 101 adds identification information 124 indicating the extended view to the encoded frame for the extended view (step S1811). Next, the encoding apparatus 101 outputs an encoded frame for the extended view to the stream 121 (step S1812). Subsequently, the encoding apparatus 101 determines whether or not the photographing is finished (step S1813). When the photographing is finished (step S1813: Yes), the encoding apparatus 101 encodes the last image in the extended view by the number of frames of the dummy frame information (step S1814).

  Next, the encoding apparatus 101 sets the generated dummy frame as a frame next to the last encoded frame for the extended view (step S1815). After the end of step S1812, or when shooting has not ended (step S1813: No), the encoding apparatus 101 ends the MVC encoding process. As described above, by executing the processing illustrated in FIGS. 18A and 18B, the encoding apparatus 101 can encode the image in the base view and the image in the extended view.

  FIG. 19 is a flowchart illustrating an example of a motion vector search range determination process in the base view. FIG. 19 illustrates processing for determining a motion vector search range when the current MB exists in the base view. The encoding apparatus 101 sets lx and rx to 16 and sets uy and dy to 16 (step S1901). Next, the encoding apparatus 101 determines whether or not mbadrx of the current MB is 0 (step S1902). If it is 0 (step S1902: YES), the encoding apparatus 101 sets lx to 0 (step S1903).

  After execution of step S1903 or when mbadrx is not 0 (step S1902: No), the encoding apparatus 101 determines whether mbadrx of the current MB is mbnumx−1 (step S1904). When mbadrx is mbnumx-1 (step S1904: Yes), the encoding apparatus 101 sets rx to 0 (step S1905).

  After execution of step S1905 or when mbadrx is not mbnumx-1 (step S1904: No), the encoding apparatus 101 determines whether mbadry of the current MB is 0 (step S1906). When mbadry is 0 (step S1906: Yes), the encoding apparatus 101 sets uy to 0 (step S1907).

  After execution of step S1907, or when mbadry is not 0 (step S1906: No), the encoding apparatus 101 determines whether or not mbadry of the current MB is mbnumy-1 (step S1908). When mbadry is mbnumy-1 (step S1908: Yes), the encoding apparatus 101 sets dy to 0 (step S1909).

  After execution of step S1909, if mbadry is not mbnumy-1 (step S1908: No), the encoding apparatus 101 acquires a search range image in the base view (step S1910). The search range image in the base view is an image in which the current MB can be referred to among images obtained by decoding the encoded frames corresponding to the base view. For example, if the current MB is a P picture, it is an image obtained by decoding a past P picture or I picture.

  Next, the encoding apparatus 101 executes a motion vector search process in the base view (step S1911), and ends the motion vector search range determination process in the base view. As described above, by executing the processing illustrated in FIG. 19, the encoding apparatus 101 can determine the search range of the motion vector when the current MB exists in the base view. The motion vector search process in the base view is a process for searching for the MB having the closest pixel value to the current MB from the image acquired in step S1910. The motion vector search process is described in H.264. Since the encoding process conforming to the H.264 MVC standard is performed, the description thereof is omitted.

  20A and 20B are flowcharts illustrating an example of a motion vector search range determination process in the extended view at the time of horizontal position shooting. 20A and 20B, processing for determining a motion vector search range when the current MB exists in the extended view at the time of horizontal position shooting will be described. Steps S2002 to S2009 are the same processing as steps S1902 to S1909, and thus description thereof is omitted. First, in FIG. 20A, the encoding apparatus 101 sets lx and rx to 16, sets uy and dy to 16, and sets area (L) and area (R) in the extended view (step S2001). ). After executing the process of step S2001, the encoding apparatus 101 executes the process of step S2002.

  Next, in FIG. 20B, after executing step S2008 or step S2009, encoding apparatus 101 determines whether or not mbadrx of the current MB is mbnumx / 2-1 (step S2010). When mbadrx is mbnumx / 2-1 (step S2010: Yes), the encoding apparatus 101 sets the region (R) in the base view (step S2011).

  After execution of step S2011, or when mbadrx is not mbnumx / 2-1, (step S2010: No), the encoding apparatus 101 determines whether mbadrx of the current MB is mbnumx / 2 (step S2012). When mbadrx is mbnumx / 2 (step S2012: Yes), the encoding apparatus 101 sets the region (L) in the base view (step S2013).

  After executing step S2013, or when mbadrx is not mbnumx / 2 (step S2012: No), the encoding apparatus 101 acquires a search range image in the extended view (step S2014). The search range image in the extended view is an image in which the current MB can be referred to among images obtained by decoding the encoded frames corresponding to the extended view.

  Next, the encoding apparatus 101 determines whether the region (L) is in the base view or the region (R) is in the base view (step S2015). When either the region (L) or the region (R) is in the base view (step S2015: Yes), the encoding apparatus 101 acquires a search range image in the base view (step S2016). The image in the base view acquired in step S2016 is an image at the same time as the image to which the current MB belongs, among the images in which the encoded frames corresponding to the base view are decoded.

  When the step S2016 or the region (L) and the region (R) are not in the base view (step S2015: No), the encoding apparatus 101 executes a motion vector search process in the extended view (step S2017). Note that the motion vector search process in the extended view searches for the MB having the closest pixel value to the current MB from the image acquired in step S2014.

  Subsequently, the encoding apparatus 101 determines whether the region (L) is in the base view or whether the region (R) is in the base view (step S2018). When either the region (L) or the region (R) is in the base view (step S2018: Yes), the encoding apparatus 101 performs a motion vector search process in the base view (step S2019). Note that the motion vector search process in the extended view searches for the MB having the closest pixel value to the current MB from the image acquired in step S2016.

  After the process of step S2019, or when the region (L) and the region (R) are not in the base view (step S2018: No), the encoding apparatus 101 determines the motion vector search range in the extended view at the time of horizontal position shooting. The process ends. As described above, by executing the processing shown in FIGS. 20A and 20B, the encoding apparatus 101 can determine the search range of the motion vector when the current MB is present in the extended view at the time of horizontal position shooting. .

  FIGS. 21A and 21B are flowcharts illustrating an example of a motion vector search range determination process in the extended view at the time of vertical position shooting. 21A and 21B illustrate a process for determining a motion vector search range when the current MB is present in the extended view during vertical position shooting. Steps S2101 to S2109 are the same processes as steps S2001 to S2009, and thus description thereof is omitted.

  In FIG. 21A, after executing step S2108 or step S2109, encoding apparatus 101 determines whether or not the mbadry of the current MB is mbnumy / 2-1 (step S2110). When mbadry is mbnumy / 2-1 (step S2110: Yes), the encoding apparatus 101 sets the region (D) in the base view (step S2111).

  After execution of step S2111 or when mbadry is not mbnumy / 2-1 (step S2110: No), the encoding apparatus 101 determines whether or not mbadry of the current MB is mbnumy / 2 (step S2112). When mbadry is mbnumy / 2 (step S2112: Yes), the encoding apparatus 101 sets the region (U) in the base view (step S2113).

  After executing step S2113, or when mbadry is not mbnumy / 2 (step S2112: No), the encoding apparatus 101 acquires a search range image in the extended view (step S2114). The search range image in the extended view is an image in which the current MB can be referred to among images obtained by decoding the encoded frames corresponding to the extended view.

  Next, the encoding apparatus 101 determines whether the region (U) is in the base view or the region (D) is in the base view (step S2115). When either the region (U) or the region (D) is in the base view (step S2115: Yes), the encoding apparatus 101 acquires a search range image in the base view (step S2116). The image in the base view acquired in step S2116 is an image at the same time as the image to which the current MB belongs, among images obtained by decoding the encoded frames corresponding to the base view.

  After execution of step S2116 or when the region (U) and the region (D) are not in the base view (step S2115: No), the encoding apparatus 101 executes motion vector search processing in the extended view (step S2117). ). Note that the motion vector search process in the extended view searches for an MB having the closest pixel value to the current MB from the image acquired in step S2114.

  Subsequently, the encoding apparatus 101 determines whether the region (U) is in the base view or the region (D) is in the base view (step S2118). When either the region (U) or the region (D) is in the base view (step S2118: Yes), the encoding apparatus 101 executes a motion vector search process in the base view (step S2119). Note that the motion vector search process in the extended view searches for the MB having the closest pixel value to the current MB from the image acquired in step S2016.

  After the process of step S2119, or when the area (U) and the area (D) are not in the base view (step S2118: No), the encoding apparatus 101 determines the motion vector search range in the extended view at the time of vertical position shooting. End the process. As described above, by executing the processing illustrated in FIGS. 21A and 21B, the encoding apparatus 101 can determine the search range of the motion vector when the current MB is present in the extended view during vertical position shooting. .

  22A and 22B are flowcharts illustrating an example of the decoding processing procedure. 22A and 22B describe a process for decoding an encoded frame. When receiving a moving image playback request from a user or the like, the decoding apparatus 400 receives vertical / horizontal information and dummy frame information from the stream 121 (step S2201). Next, the decoding apparatus 400 selects an encoded frame for the base view from the stream 121 based on the identification information 124 (step S2202), and determines whether a display area is designated by the user (step S2203).

  When the display area is not specified (step S2203: No), the decoding apparatus 400 performs an MVC decoding process on the encoded frame for the base view (step S2204). Next, the decoding device 400 outputs the partial image in the base view to the RAM 403 (step S2205). Subsequently, the decoding apparatus 400 sets an image other than the partial image corresponding to the encoded frame that is not the output target among the partial images in the base view as the output target (step S2206). Next, the decoding apparatus 400 outputs an output target image in the base view (step S2207).

  Step S2207 or step S2203 shown in FIG. 22B: After a series of processing that becomes the route of Yes, the decoding apparatus 400 determines whether or not the reproduction has ended (step S2208). When the reproduction has not ended (step S2208: No), the decoding device 400 proceeds to the process of step S2202. When the reproduction ends (step S2208: Yes), the decoding device 400 ends the decoding process.

  Next, when the display area is designated in FIG. 22B (step S2203: Yes), the decoding apparatus 400 selects an encoded frame for the extended view from the stream 121 (step S2209). Through the processing in steps S2202 and S2209, the code frame for the base view to be decoded and the encoded frame for the extended view at the same time are selected. Subsequently, the decoding apparatus 400 performs MVC decoding processing on the encoded frame for the base view and the encoded frame for the extended view (step S2210). The MVC decoding process is the same as that of H.264. Since the decoding process conforming to the H.264 MVC standard is performed, the description thereof is omitted.

  Next, the decoding apparatus 400 outputs the decoded partial images of the base view and the extended view to the RAM 403 (Step S2211). Subsequently, the decoding apparatus 400 sets an image other than the partial image corresponding to the encoded frame that is not the output target among the partial images in the base view as the output target (step S2212). Next, the decoding apparatus 400 determines whether the vertical / horizontal information indicates horizontal (step S2213).

  When the vertical direction is indicated (step S2213: No), the decoding apparatus 400 converts the designated display area according to the vertical and horizontal information (step S2214). Specifically, when the display area is designated to move 10 pixels in the right direction from the central portion, the converted display area is an area that has been moved 10 pixels in the downward direction from the central portion. After executing step S2214 or when showing sideways (step S2213: Yes), the decoding apparatus 400 extracts the image sequence in the display area from the partial image in the base view and displays it from the partial image in the extended view. An image sequence in the region is extracted (step S2215).

  Next, the decoding apparatus 400 combines the images in the image sequence in the base view and the images in the image sequence in the extended view by using the same image (step S2216). Next, the decoding apparatus 400 outputs an image of the designated display area (step S2217), and proceeds to the process of step S2208. In this way, by executing the processing shown in FIGS. 22A and 22B, the decoding device 400 can decode the encoded frame and reproduce the moving image.

  As described above, according to the encoding device, the moving image processing system, the encoding method, and the encoding program, an extended view that uses 2k1k as an independent base view and refers to the left and right regions as base views from a 4k2k image. As such, view identification information is given. Thereby, since the decoding apparatus which decodes a base view can limit a decoding object to the encoding frame with respect to a base view from identification information, decoding amount can be reduced rather than converting after decoding the whole screen.

  By reducing the amount of decoding, for example, the decoding device can shorten the time from when a playback instruction is received from the user to when playback of a moving image starts. Also, the decoding device can shorten the time until the trick play is started even when performing trick play such as fast forward, rewind, and cue. For example, when performing cueing, the decoding device can speed up cueing processing. In addition, the decoding device during moving image reproduction can reduce a phenomenon in which reproduction of moving images is unnaturally interrupted.

  Also, the encoding apparatus may generate dummy frames for the number of sheets that can be referred to the past and set them as the base view. Thereby, when encoding an extended view, it is possible to refer to a past base view image, and the encoding apparatus can improve encoding efficiency. Since the dummy frame is often a skip macroblock, the added code amount is small.

  For example, when the object moves from the base view to the extended view, an image of the past base view can be referred to, and the encoding efficiency can be improved. In particular, since the P picture of the extended view can be referred to only in the past direction, the past picture of the base view can be referred to, thereby improving the coding efficiency. In addition, since the encoding efficiency is improved, the encoding apparatus can extend the shooting time.

  In addition, the encoding apparatus may set the number of dummy frames generated to be the number of consecutive B pictures + 1. As a result, the P picture of the extended view refers to the I picture or P picture of the base view. When the base view I-picture or P-picture is encoded with high image quality, a high-quality image can be referred to, and thus a higher-quality image can be provided to the user.

  Also, the encoding device may not generate a dummy frame when a dummy frame has been set. This is because if the dummy frame has already been set, the encoding device refers to the past base view image when encoding the extended view.

  Also, the encoding device may generate dummy frames for the number of sheets that can be referred to the past and set the dummy frames in the extended view. Thereby, the image of the future base view can be referred to when the extended view is encoded. For example, when an object moves from the extended view to the base view, the P picture of the extended view can refer to the past image in the extended view and the future image in the base view, and the future image is more If it is close to the current image, the encoding efficiency can be improved.

  Also, the encoding device may detect the vertical and horizontal directions of its own device and change the extraction position of the base view and the extended view. Accordingly, the encoding device can capture an image in a display area suitable for a horizontally long object when detecting a landscape, and can capture an image in a display area suitable for a vertically long object when detecting a portrait.

  Further, the encoding apparatus may encode the image in the extended view based on the image in the base view. For example, it is assumed that the encoding apparatus simply has specifications for encoding an image in the base view and an image in the extended view without setting a dummy frame. At this time, the time at which the image in the base view is encoded and the time at which the image in the extended view is encoded are almost the same, so there is a possibility that the image in the base view remains in the memory. In this case, the encoding apparatus may encode the image in the base view as it is without using the encoded frame obtained by encoding the image in the base view.

  In addition, according to the decoding device, the moving image processing system, the decoding method, and the decoding program, a series of encoded frames of the first and second regions and identification information indicating the region assigned to the encoded frames are received, The encoded frame in the first area is decoded from the identification information. Thus, since the decoding apparatus can limit the decoding target to the encoded frame for the base view from the identification information, the decoding amount can be reduced.

  Further, the decoding apparatus outputs a decoded image corresponding to the encoded frame other than the output target from the information indicating the number and position of the encoded frames that are not the output target. Thereby, the decoding apparatus can output a moving image that does not include a dummy frame.

  In addition, when the display area is designated, the decoding device decodes each of the encoded frames in the base view of the series of images and the encoded frames in the extended view of the series of images, and extracts them to the display area. , May be combined and output. As a result, the decoding apparatus can respond to the change of the display area, and decodes the encoded frame in the extended view only when designated, so that the image in the extended view is always decoded. In comparison, the processing efficiency can be improved.

  In addition, when the display area is specified and the number and position of the encoded frames that are not the output target are received, the decoding apparatus receives the partial image in the base view and the extended view corresponding to the encoded frames other than the output target. Partial images may be extracted and combined. As a result, the decoding apparatus can perform reproduction in the same manner as the method of extracting the display area designated from 4k1k, and the encoding efficiency is improved, so that power consumption is reduced and reproduction time can be extended. . Further, the process of removing images other than the output target may be performed after the extraction process.

  In addition, the decoding device may convert the designated display area by receiving information about an angle when the device that has received the series of images receives the series of images. Thereby, the decoding apparatus can reproduce the image in the same direction as when a series of images is received.

  Also, the encoding apparatus may specify a base view area and an extended view area. For example, it is assumed that the shooting target object a exists in the center of the shooting range at the start of shooting, and the shooting target object a moves to the right side in the shooting range in the middle of shooting. In such a case, in the present embodiment, the display area is specified as an example specified by the user of the decoding device. However, at the time of encoding, the base view is displayed by the user operating the encoding device. And an extended view area may be specified.

  Specifically, it is assumed that the base view area is a rectangular area of 1k [pixel] in length and 2k [pixel] in width, in accordance with the user's instruction, in which the upper left coordinates are (1.2k, 0.5k). Upon receiving the user instruction, the encoding apparatus extracts an image in the base view in the designated base view area. In the encoding apparatus, the upper left coordinates are (0k, 0.5k), the area is 1k [pixel] in the vertical direction and 1.2k [pixel] in the horizontal direction, and the upper left coordinates are (3.2k, 0. 5k), and an area of 1k [pixel] in length and 0.8k [pixel] in width is extracted as an extended view area.

  Subsequently, the encoding apparatus encodes the extracted image, assigns the designated area as information, and outputs the information to the stream 121. At this time, in the extended view vector search range determination process, the search range image in the base view is acquired at a location where the abscissa of the current MB is adjacent to 1.2k.

  Next, the decoding device decodes the image of the base view when there is no operation by the user of the decoding device and the display area is not designated. Thereby, even when the imaging target a moves, the decoding target can be limited to the encoded frame for the base view, so that the decoding amount can be reduced.

  H. H.264 MVC makes it possible to reproduce a base view without re-encoding and decoding. Therefore, Full HD H.264 H.264 AVC can be played back by an existing decoding device.

  Note that the encoding method and decoding method described in this embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. The encoding program and the decoding program are recorded on a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, and a DVD, and are executed by being read from the recording medium by the computer. The encoding program and the decoding program may be distributed via a network such as the Internet.

  In addition, the image processing unit 310 and the MVC encoding / decoding processing unit 311 described in the present embodiment are referred to simply as “ASIC” for specific applications such as standard cells and structured ASIC (Application Specific Integrated Circuit). Or PLD (Programmable Logic Device) such as FPGA. The same applies to the image processing unit 407 and the MVC decoding processing unit 408. As an example, for example, the image processing unit 310 can be manufactured by defining the function of the above-described extraction unit 504 by HDL description, logically synthesizing the HDL description, and giving the ASIC or PLD.

  The following additional notes are disclosed with respect to the embodiment described above.

(Additional remark 1) The extraction part which extracts the image in a 1st area | region and the image in a 2nd area | region adjacent to the said 1st area | region from any image of a series of images,
A first encoding unit that encodes an image in the first region into an encoded frame for the first region;
When the first encoding unit encodes any one of the images, a dummy frame of an encoding frame for the first region of the previous image of the any image is set as the dummy frame of the previous image. Based on the image in the first region, a generation unit that generates a number equal to or less than the number of frames included in the appearance interval of a specific encoded frame that can be reproduced with the self-encoded frame;
A setting unit configured to set the dummy frames generated by the generation unit by the number of generations as the next encoded frame of the encoded frame for the first region of the previous image;
The preceding encoded frame group preceding the encoded frame for the first region of any of the images is preceded by the number of generations of the dummy frame from the encoded frame for the first region of any of the images. An image in the second region of any one of the images based on the image by the preceding encoded frame and the image in the second region of any one of the images. A second encoding unit that encodes an encoded frame for the second region;
An output unit for outputting the dummy frames generated for the generation number and information indicating the generation number and setting position of the dummy frames;
An encoding device comprising:

(Appendix 2) The generation unit
The dummy frames are generated by a number one more than the number of encoded frames having a plurality of reference destinations in the encoded frame group encoded by the first encoding unit. The encoding device according to appendix 1.

(Supplementary Note 3) The second encoding unit includes:
If the dummy frame has been set, the preceding preceding the number of generations of the dummy frame from the encoded frame in the preceding encoded frame group preceding the encoded frame for the first region of any one of the images Based on the image by the encoding frame and the image in the second region of any one of the images, the image in the second region of any one of the images is changed to the second of the one of the images. The encoding apparatus according to appendix 1 or 2, wherein encoding is performed on an encoding frame corresponding to a region of 1.

(Supplementary Note 4) The generation unit
When the second encoding unit encodes any one of the images, a dummy frame of an encoded frame for the second region of the previous image of the any image is set as the dummy frame of the previous image. Based on the image in the second region, the number of frames included in the appearance interval of a specific encoded frame that can be reproduced with the self-encoded frame is generated,
The setting unit
The dummy frames generated by the number of generations by the generation unit are set as the next encoded frame of the encoded frame for the second region of the previous image,
The second encoding unit includes:
The number of generations of the dummy frames follows from the encoded frame for the first region of the image in the subsequent encoded frame group following the encoded frame for the first region of the image. Based on the image in the subsequent encoded frame and the image in the second region of any one of the images, the image in the second region of any one of the images is Encode into an encoded frame for region 2;
The output unit is
Furthermore, the encoding apparatus according to appendix 1, wherein the number of the generated dummy frames and the information indicating the number of generated dummy frames and the set position are output.

(Additional remark 5) The acquisition part which acquires the information regarding the angle of the own apparatus when the said series of images are received,
A determination unit that determines position information of the first and second regions according to information on the angle of the device acquired by the acquisition unit;
The extraction unit includes:
Supplementary notes 1-4, wherein an image in the first area according to the position information and an image in the second area according to the position information are extracted from any of the images. The encoding apparatus as described in any one.

(Appendix 6) The output unit
6. The encoding apparatus according to appendix 5, wherein information relating to the angle of the own apparatus is output.

(Supplementary note 7) A series of encoded frames for a series of partial images in a first area in a series of images, and a series of parts in a second area adjacent to the first area in the series of images. A series of encoded frames for an image, identification information corresponding to the one given to at least one of a series of encoded frames for a series of partial images in the first or second region, and A reception unit that receives information indicating the number and position of encoded frames that are not to be output among a series of encoded frames for a series of partial images in the first or second region;
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas based on the identification information And
A decoding unit for decoding a series of encoded frames selected by the selection unit into a series of partial images;
Of the series of partial images decoded by the decoding unit, other parts other than the partial images corresponding to the encoded frames that are not the output target based on the information indicating the number and position of the encoded frames that are not the output target An output unit for outputting an image;
A decoding device comprising:

(Supplementary note 8) A series of encoded frames for a series of partial images in a first area of a series of images, and a series of parts in a second area adjacent to the first area of the series of images. A series of encoded frames for the image, and identification information corresponding to the one given to at least one of the series of encoded frames for the series of partial images in the first or second region, A reception part to accept,
A designating unit for designating a display area from an area obtained by combining the first area and the second area;
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas based on the identification information And
A series of encoded frames for a series of partial images in the first area selected by the selection unit are decoded into a series of partial images in the first area, and the display area is designated by the designation unit A decoding unit that decodes a series of encoded frames for the series of partial images in the second region into a series of partial images in the second region;
A first image sequence in the display area is extracted from a series of partial images in the first area, and a second image sequence in the display area is extracted from a series of partial images in the second area. An extractor for extracting;
A combining unit that combines the images of the first image sequence extracted by the extraction unit and the images of the second image sequence with the same images in the series of images;
An output unit that outputs an image sequence of the display area obtained by combining the combining units;
A decoding device comprising:

(Supplementary note 9)
Receiving information about the angle of the device when the device receiving the series of images receives the series of images;
A conversion unit that converts the display area according to the information on the angle;
The extraction unit includes:
The first image sequence in the display area after conversion by the conversion unit is extracted from the series of partial images in the first area, and the display after conversion from the series of partial images in the second area Extracting a second image sequence in the region;
The output unit is
The decoding apparatus according to appendix 8, wherein an image sequence of the converted display area obtained by combining the combining units is output.

(Supplementary note 10) A moving image processing system having an encoding device for encoding a series of images and a decoding device for decoding the encoded series of images,
The encoding device includes:
An extraction unit that extracts an image in a first area and an image in a second area adjacent to the first area from any one of the series of images;
A first encoding unit that encodes an image in the first region into an encoded frame for the first region;
An encoded frame for the first region is decoded, and based on the decoded image and the image in the second region, the image in the second region is encoded into the second region. A second encoding unit for encoding
First, identification information indicating at least one of the first and second regions is assigned to a corresponding encoded frame, and each encoded frame for the first and second regions is output to the decoding device. 1 output unit,
The decoding device
A reception unit that receives each encoded frame for the first and second regions to which the identification information is assigned;
A selection unit that selects an encoded frame for the first region based on the identification information from each encoded frame for the first and second regions;
A decoding unit that decodes an encoded frame for the first region selected by the selection unit into a decoded image;
A second output unit that outputs a decoded image decoded by the decoding unit;
A moving image processing system comprising:

(Appendix 11) The computer
With reference to a storage unit that stores a series of images, an image in a first area and an image in a second area adjacent to the first area are selected from any one of the series of images. Extract and
Encoding an image in the first region into an encoded frame for the first region;
When encoding any one of the images, a dummy frame of the encoding frame for the first region of the image before the one of the images is changed to an image in the first region of the previous image. Based on the number of frames included in the appearance interval of a specific encoded frame that can be reproduced in the self-encoded frame,
The number of generated dummy frames is set to the next encoded frame of the encoded frame for the first region of the previous image,
The preceding encoded frame group preceding the encoded frame for the first region of any of the images is preceded by the number of generations of the dummy frame from the encoded frame for the first region of any of the images. An image in the second region of any one of the images based on the image by the preceding encoded frame and the image in the second region of any one of the images. Encoding into the encoded frame for the second region;
Outputting each encoded frame for the first and second regions, the dummy frames generated for the generation number, and information indicating the generation number and setting position of the dummy frames;
An encoding method characterized by executing processing.

(Supplementary note 12)
A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. An encoded frame, identification information corresponding to one of the series of encoded frames for the series of partial images in the first or second region, and the first or second Receiving information indicating the number and position of non-output encoded frames among a series of encoded frames for a series of partial images in the area of 2,
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas, based on the identification information;
Decoding a selected series of encoded frames into a series of partial images;
Based on the information indicating the number and position of the encoded frames that are not the output target, the other partial images other than the partial images corresponding to the encoded frames that are not the output target are output from the series of decoded partial images. ,
A decoding method characterized by executing processing.

(Supplementary note 13)
A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. Receiving an encoded frame and identification information corresponding to at least one of a series of encoded frames for a series of partial images in the first or second region;
A display area is designated from an area obtained by combining the first area and the second area,
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas, based on the identification information;
When a series of encoded frames for a series of partial images in the selected first area is decoded into a series of partial images in the first area, and when the display area is designated, Decoding a series of encoded frames for a series of partial images in a region into a series of partial images in the second region;
A first image sequence in the display area is extracted from a series of partial images in the first area, and a second image sequence in the display area is extracted from a series of partial images in the second area. Extract and
Combining the extracted images of the first image sequence and the images of the second image sequence with the same image in the series of images,
Outputting an image sequence of the display area obtained by combining,
A decoding method characterized by executing processing.

(Supplementary note 14)
With reference to a storage unit that stores a series of images, an image in a first area and an image in a second area adjacent to the first area are selected from any one of the series of images. Extract and
Encoding an image in the first region into an encoded frame for the first region;
When encoding any one of the images, a dummy frame of the encoding frame for the first region of the image before the one of the images is changed to an image in the first region of the previous image. Based on the number of frames included in the appearance interval of a specific encoded frame that can be reproduced in the self-encoded frame,
The number of generated dummy frames is set to the next encoded frame of the encoded frame for the first region of the previous image,
The preceding encoded frame group preceding the encoded frame for the first region of any of the images is preceded by the number of generations of the dummy frame from the encoded frame for the first region of any of the images. An image in the second region of any one of the images based on the image by the preceding encoded frame and the image in the second region of any one of the images. Encoding into the encoded frame for the second region;
Outputting each encoded frame for the first and second regions, the dummy frames generated for the generation number, and information indicating the generation number and setting position of the dummy frames;
An encoding program for executing a process.

(Supplementary note 15)
A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. An encoded frame, identification information corresponding to one of the series of encoded frames for the series of partial images in the first or second region, and the first or second Receiving information indicating the number and position of non-output encoded frames among a series of encoded frames for a series of partial images in the area of 2,
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas, based on the identification information;
Decoding a selected series of encoded frames into a series of partial images;
Based on the information indicating the number and position of the encoded frames that are not the output target, the other partial images other than the partial images corresponding to the encoded frames that are not the output target are output from the series of decoded partial images. ,
A decryption program for executing a process.

(Supplementary note 16)
A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. Receiving an encoded frame and identification information corresponding to at least one of a series of encoded frames for a series of partial images in the first or second region;
A display area is designated from an area obtained by combining the first area and the second area,
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas, based on the identification information;
When a series of encoded frames for a series of partial images in the selected first area is decoded into a series of partial images in the first area, and when the display area is designated, Decoding a series of encoded frames for a series of partial images in a region into a series of partial images in the second region;
A first image sequence in the display area is extracted from a series of partial images in the first area, and a second image sequence in the display area is extracted from a series of partial images in the second area. Extract and
Combining the extracted images of the first image sequence and the images of the second image sequence with the same image in the series of images,
Outputting an image sequence of the display area obtained by combining,
A decryption program for executing a process.

DESCRIPTION OF SYMBOLS 101 Coding apparatus 400 Decoding apparatus 501 Reception part 502 Acquisition part 503 Determination part 504 Extraction part 505 1st encoding part 506 2nd encoding part 507 Generation part 508 Setting part 509 Output part 521 Reception part 522 Selection part 523 Specification part 524 Conversion unit 525 Decoding unit 526 Extraction unit 527 Coupling unit 528 Output unit

Claims (14)

An extraction unit that extracts an image in the first area and an image in the second area adjacent to the first area from any one of the series of images;
A first encoding unit that encodes an image in the first region into an encoded frame for the first region;
When the first encoding unit encodes any one of the images, a dummy frame of an encoding frame for the first region of the previous image of the any image is set as the dummy frame of the previous image. Based on the image in the first region, a generation unit that generates a number equal to or less than the number of frames included in the appearance interval of a specific encoded frame that can be reproduced with the self-encoded frame;
A setting unit configured to set the dummy frames generated by the generation unit by the number of generations as the next encoded frame of the encoded frame for the first region of the previous image;
The preceding encoded frame group preceding the encoded frame for the first region of any of the images is preceded by the number of generations of the dummy frame from the encoded frame for the first region of any of the images. An image in the second region of any one of the images based on the image by the preceding encoded frame and the image in the second region of any one of the images. A second encoding unit that encodes an encoded frame for the second region;
An output unit for outputting each encoded frame for the first and second regions, the dummy frames generated for the generation number, and information indicating the generation number and setting position of the dummy frames;
An encoding device comprising: The generator is
The dummy frames are generated by a number one more than the number of encoded frames having a plurality of reference destinations in the encoded frame group encoded by the first encoding unit. The encoding device according to claim 1. The generator is
When the second encoding unit encodes any one of the images, a dummy frame of an encoded frame for the second region of the previous image of the any image is set as the dummy frame of the previous image. Based on the image in the second region, the number of frames included in the appearance interval of a specific encoded frame that can be reproduced with the self-encoded frame is generated,
The setting unit
The dummy frames generated by the number of generations by the generation unit are set as the next encoded frame of the encoded frame for the second region of the previous image,
The second encoding unit includes:
The number of generations of the dummy frames follows from the encoded frame for the first region of the image in the subsequent encoded frame group following the encoded frame for the first region of the image. Based on the image in the subsequent encoded frame and the image in the second region of any one of the images, the image in the second region of any one of the images is Encode into an encoded frame for region 2;
The output unit is
2. The encoding apparatus according to claim 1, further comprising: outputting the dummy frames generated for the number of generations and information indicating the generation numbers and setting positions of the dummy frames. An acquisition unit that acquires information about the angle of the device when receiving the series of images;
A determination unit that determines position information of the first and second regions according to information on the angle of the device acquired by the acquisition unit;
The extraction unit includes:
The image in the first area according to the position information and the image in the second area according to the position information are extracted from any one of the images. The encoding device according to any one of the above. A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. An encoded frame, identification information corresponding to one of the series of encoded frames for the series of partial images in the first or second region, and the first or second A reception unit that receives information indicating the number and position of encoded frames that are not to be output among a series of encoded frames for a series of partial images in the region of 2;
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas based on the identification information And
A decoding unit for decoding a series of encoded frames selected by the selection unit into a series of partial images;
Of the series of partial images decoded by the decoding unit, other parts other than the partial images corresponding to the encoded frames that are not the output target based on the information indicating the number and position of the encoded frames that are not the output target An output unit for outputting an image;
A decoding device comprising: A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. A receiving unit that receives the encoded frame and identification information corresponding to at least one of the series of encoded frames for the series of partial images in the first or second region; ,
A designating unit for designating a display area from an area obtained by combining the first area and the second area;
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas based on the identification information And
A series of encoded frames for a series of partial images in the first area selected by the selection unit are decoded into a series of partial images in the first area, and the display area is designated by the designation unit A decoding unit that decodes a series of encoded frames for the series of partial images in the second region into a series of partial images in the second region;
A first image sequence in the display area is extracted from a series of partial images in the first area, and a second image sequence in the display area is extracted from a series of partial images in the second area. An extractor for extracting;
A combining unit that combines the images of the first image sequence extracted by the extraction unit and the images of the second image sequence with the same images in the series of images;
An output unit that outputs an image sequence of the display area obtained by combining the combining units;
A decoding device comprising: The reception unit
Receiving information about the angle of the device when the device receiving the series of images receives the series of images;
A conversion unit that converts the display area according to the information on the angle;
The extraction unit includes:
The first image sequence in the display area after conversion by the conversion unit is extracted from the series of partial images in the first area, and the display after conversion from the series of partial images in the second area Extracting a second image sequence in the region;
The output unit is
The decoding apparatus according to claim 6, wherein the image sequence of the display area after conversion obtained by combining the combining units is output. A video processing system having an encoding device for encoding a series of images and a decoding device for decoding the encoded series of images,
The encoding device includes:
An extraction unit that extracts an image in a first area and an image in a second area adjacent to the first area from any one of the series of images;
A first encoding unit that encodes an image in the first region into an encoded frame for the first region;
A second encoding unit that encodes an image in the second region into an encoded frame for the second region based on an image of the encoded frame for the first region and an image in the second region; An encoding unit;
First, identification information indicating at least one of the first and second regions is assigned to a corresponding encoded frame, and each encoded frame for the first and second regions is output to the decoding device. 1 output unit,
The decoding device
A reception unit that receives each encoded frame for the first and second regions to which the identification information is assigned;
A selection unit that selects an encoded frame for the first region based on the identification information from each encoded frame for the first and second regions;
A decoding unit that decodes an encoded frame for the first region selected by the selection unit into a decoded image;
A second output unit that outputs a decoded image decoded by the decoding unit;
A moving image processing system comprising: Computer
With reference to a storage unit that stores a series of images, an image in a first area and an image in a second area adjacent to the first area are selected from any one of the series of images. Extract and
Encoding an image in the first region into an encoded frame for the first region;
When encoding any one of the images, a dummy frame of the encoding frame for the first region of the image before the one of the images is changed to an image in the first region of the previous image. Based on the number of frames included in the appearance interval of a specific encoded frame that can be reproduced in the self-encoded frame,
The number of generated dummy frames is set to the next encoded frame of the encoded frame for the first region of the previous image,
The preceding encoded frame group preceding the encoded frame for the first region of any of the images is preceded by the number of generations of the dummy frame from the encoded frame for the first region of any of the images. An image in the second region of any one of the images based on the image by the preceding encoded frame and the image in the second region of any one of the images. Encoding into the encoded frame for the second region;
Outputting each encoded frame for the first and second regions, the dummy frames generated for the generation number, and information indicating the generation number and setting position of the dummy frames;
An encoding method characterized by executing processing. Computer
A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. An encoded frame, identification information corresponding to one of the series of encoded frames for the series of partial images in the first or second region, and the first or second Receiving information indicating the number and position of non-output encoded frames among a series of encoded frames for a series of partial images in the area of 2,
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas, based on the identification information;
Decoding a selected series of encoded frames into a series of partial images;
Based on the information indicating the number and position of the encoded frames that are not the output target, the other partial images other than the partial images corresponding to the encoded frames that are not the output target are output from the series of decoded partial images. ,
A decoding method characterized by executing processing. Computer
A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. Receiving an encoded frame and identification information corresponding to at least one of a series of encoded frames for a series of partial images in the first or second region;
A display area is designated from an area obtained by combining the first area and the second area,
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas, based on the identification information;
When a series of encoded frames for a series of partial images in the selected first area is decoded into a series of partial images in the first area, and when the display area is designated, Decoding a series of encoded frames for a series of partial images in a region into a series of partial images in the second region;
A first image sequence in the display area is extracted from a series of partial images in the first area, and a second image sequence in the display area is extracted from a series of partial images in the second area. Extract and
Combining the extracted images of the first image sequence and the images of the second image sequence with the same image in the series of images,
Outputting an image sequence of the display area obtained by combining,
A decoding method characterized by executing processing. On the computer,
With reference to a storage unit that stores a series of images, an image in a first area and an image in a second area adjacent to the first area are selected from any one of the series of images. Extract and
Encoding an image in the first region into an encoded frame for the first region;
When encoding any one of the images, a dummy frame of the encoding frame for the first region of the image before the one of the images is changed to an image in the first region of the previous image. Based on the number of frames included in the appearance interval of a specific encoded frame that can be reproduced in the self-encoded frame,
The number of generated dummy frames is set to the next encoded frame of the encoded frame for the first region of the previous image,
The preceding encoded frame group preceding the encoded frame for the first region of any of the images is preceded by the number of generations of the dummy frame from the encoded frame for the first region of any of the images. An image in the second region of any one of the images based on the image by the preceding encoded frame and the image in the second region of any one of the images. Encoding into the encoded frame for the second region;
Outputting each encoded frame for the first and second regions, the dummy frames generated for the generation number, and information indicating the generation number and setting position of the dummy frames;
An encoding program for executing a process. On the computer,
A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. An encoded frame, identification information corresponding to one of the series of encoded frames for the series of partial images in the first or second region, and the first or second Receiving information indicating the number and position of non-output encoded frames among a series of encoded frames for a series of partial images in the area of 2,
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas, based on the identification information;
Decoding a selected series of encoded frames into a series of partial images;
Based on the information indicating the number and position of the encoded frames that are not the output target, the other partial images other than the partial images corresponding to the encoded frames that are not the output target are output from the series of decoded partial images. ,
A decryption program for executing a process. On the computer,
A series of encoded frames for a series of partial images in a first region of the series of images, and a series of sequences for a series of partial images in a second region adjacent to the first region of the series of images. Receiving an encoded frame and identification information corresponding to at least one of a series of encoded frames for a series of partial images in the first or second region;
A display area is designated from an area obtained by combining the first area and the second area,
Selecting a series of encoded frames for a series of partial images in the first area from a series of encoded frames for a series of partial images in the first and second areas, based on the identification information;
When a series of encoded frames for a series of partial images in the selected first area is decoded into a series of partial images in the first area, and when the display area is designated, Decoding a series of encoded frames for a series of partial images in a region into a series of partial images in the second region;
A first image sequence in the display area is extracted from a series of partial images in the first area, and a second image sequence in the display area is extracted from a series of partial images in the second area. Extract and
Combining the extracted images of the first image sequence and the images of the second image sequence with the same image in the series of images,
Outputting an image sequence of the display area obtained by combining,
A decryption program for executing a process.

Images