JP2012050005A - Image processing device, image processing method and image processing program - Google Patents

Abstract

An image processing apparatus, an image processing method, and an image processing program capable of correctly generating interpolated image data for interpolating between image data for stereoscopic video are provided.
An image processing apparatus according to the present invention generates processed image data indicating an image including a blank image of one block or more between a first image for displaying a stereoscopic image and a second image from the stereoscopic image data. The motion vectors of the plurality of second blocks within a predetermined range from the first block included in the image indicated by the data processing unit that performs processing and the processing interpolation image data that interpolates between the plurality of processing image data. A candidate vector determination unit that determines a candidate vector of a motion vector, a motion vector determination unit that determines a motion vector of the first block from the candidate vector, an image interpolation unit that generates processed interpolation image data based on the motion vector, An output unit is provided that outputs interpolated image data for interpolating between a plurality of stereoscopic image data based on the processed interpolated image data.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program, and more particularly to a technique for processing image data for stereoscopic video.

  In recent years, televisions have been shifted from CRTs to flat panel displays such as liquid crystal displays, plasma displays, and organic EL displays. In the field of home video equipment, flat panel displays are adopted for terrestrial digital broadcasting-compatible televisions. And IP (Interlace / Progressive) conversion technology has become indispensable for the display specification of a flat panel display.

  Here, a motion vector detection method in the IP conversion process disclosed in Patent Literature 1 will be described with reference to FIGS. FIG. 10 is a conceptual diagram of motion vector detection according to Patent Document 1. FIG. 11 is a configuration diagram of a motion-compensated IP conversion processing device according to Patent Document 1. FIG. 12 is an explanatory diagram of blocks used as candidate vectors according to Patent Document 1. In FIG.

  As shown in FIG. 10, in the IP conversion processing according to Patent Document 1, a field n-1 image 1001 is placed between a field n image 1002 and a field n-2 image 1000 that is two fields away from the field n image 1002. Interpolate. Note that the field n image 1002, the field n-2 image 1000, and the field n-1 image 1001 are interlaced images. The IP conversion process uses a motion vector when generating the interpolation field image 1001.

  Next, a motion vector detection process in the IP conversion process according to Patent Document 1 will be described with reference to FIG. In the motion vector detection process according to Patent Document 1, one field screen is divided into M × N pixel blocks 1007 in order to generate an interpolation field image 1001. And the motion vector detection process concerning patent document 1 is processed per block from the upper left of a screen to the lower right like a normal scanning line system.

  In the motion vector detection process according to Patent Document 1, in the process of generating an interpolation field image (field n-1 image) 1001, an image one field before that is referred to by a candidate vector 1006 with respect to a block 1004 currently being processed. The difference between the reference block 1 (1003) of the (field n-2 image) 1000 and the reference block 2 (1005) of the image (field n image) 1002 after one field is calculated. This difference is calculated as the sum of absolute differences of the pixels of the reference block 1 (1003) and the reference block 2 (1005) (hereinafter referred to as “SAD (Sum Of Absolute Difference)”). In the motion vector detection processing according to Patent Document 1, SAD is calculated for all candidate vectors, and a candidate vector having a minimum value for SAD is determined as a motion vector of the current processing block 1004.

  Next, a motion vector detection process according to Patent Document 1 will be described with reference to FIG. The SAD calculation unit 900 receives image data 910 one field before the interpolation field image and image data 911 one field after the interpolation field image. The SAD calculation unit 900 calculates the SAD between the reference block 1 and the reference block 2 indicated by the candidate vector 913 output from the candidate vector generation unit 902. The SAD operation unit 900 performs the same operation on all candidate vectors. The SAD minimum value determination unit 901 determines the minimum value of SAD from the calculation result 912 by the SAD calculation unit 900. The motion vector determination unit 903 determines the motion vector 915 of the block currently being processed from the candidate vectors 913 based on the determination result 914 by the SAD minimum value determination unit 901.

  Next, the motion vector detection process will be described with reference to FIG. As shown in FIG. 12, a peripheral block 1103 having a high correlation with the current processing block 1102 exists around the current processing block 1102 in the 1-field screen 1100. FIG. 12 shows a state where the process of generating the interpolation field image has progressed to some extent. As shown in FIG. 11, the motion vector 915 is stored in the memory 904. The already detected vector 916 stored in the memory 904 is used as a candidate vector in processing in the next processing block. Here, blocks 1104, 1105, 1106, and 1107 of the peripheral block 1103 are blocks that have a high correlation with the current processing block 1102 and have already detected a motion vector. That is, the detected motion vectors in the blocks 1104, 1105, 1106, and 1107 are used as candidate vectors. According to this, it is possible to reduce the amount of calculation processing for motion vector detection compared to the full search method.

  On the other hand, the establishment of a stereoscopic video standard in the field of home video equipment has led to a demand for mounting a television that satisfies the standard. Accordingly, it has become necessary to perform IP conversion on stereoscopic image data. However, the technique disclosed in Patent Document 1 is not an IP conversion method corresponding to image data for stereoscopic video. Therefore, when IP conversion is performed on image data for stereoscopic video, there is a problem that erroneous conversion occurs.

  The problem will be specifically described below. As described above, the IP conversion process according to Patent Document 1 is based on the premise that all the image data input to the motion-compensated IP conversion processing apparatus displays the same type of image. IP conversion is performed using the data and image data after one field. Further, the IP conversion processing according to Patent Document 1 is currently being processed on the assumption that the image displayed by the image data input to the motion compensation IP conversion processing device includes only the same type of image. The IP conversion is performed using the detected motion vector in the peripheral block having a high correlation with the block as a candidate vector.

  On the other hand, as shown in FIG. 13, in the FrameSequential format, which is one of the data configuration methods of stereoscopic video, an image for left eye (hereinafter referred to as “L image”) and an image for right eye (hereinafter referred to as “R image”). ”) Alternately exist along the time series. That is, the FrameSequential format is a format in which L images and R images are alternately input for each field. As shown in FIGS. 14 and 15, in the SideBySide format and TopAndBottom format images, an L image and an R image are arranged side by side on the left and right or top and bottom. That is, the SideBySide format and the TopAndBottom format are formats in which an L image and an R image are input simultaneously in the same field.

  For this reason, when the image data for stereoscopic video is converted by the IP conversion processing according to Patent Document 1, the following problems occur.

  With reference to FIG. 16, a case where image data for stereoscopic video in the FrameSequential format is input to the motion compensated IP conversion processing device according to Patent Document 1 will be described.

  The motion compensation IP conversion processing apparatus divides one field screen into M × N pixel blocks 1307 in order to generate an interpolation field image (field n−1 image) 1301. The motion-compensated IP conversion processing apparatus performs processing in units of blocks from the upper left to the lower right of the screen as in a normal scanning line method.

  In the process of generating the interpolation field image (field n−1 image) 1301, the motion compensated IP conversion processing device, with respect to the current processing block 1304, the image (field n− The difference between the reference block 1 (1303) of (2 images) 1300 and the reference block 2 (1305) of the image (field n image) 1302 after one field is calculated. Here, the image data input to the motion-compensated IP conversion processing device is image data for stereoscopic video in the FrameSequential format. Therefore, when the image 1300 one field before is an L image, the reference block 1 (1303) of the image 1300 one field before the L image in the process for the current processing block 1304 of the interpolation field image 1301 which is the R image, In addition, the reference block 2 (1305) of the image 1302 after one field, which is an L image, is used. As a result, there is a problem that IP conversion is performed by mixing the L image and the R image, and an incorrect IP conversion result is generated.

  Next, with reference to FIG. 17, a case will be described in which stereoscopic video image data in SideBySide format is input to the motion compensated IP conversion processing device according to Patent Document 1.

  As shown in FIG. 14, an L image is arranged on the left side of the one-field screen and an R image is arranged on the right side of the image displayed by the stereoscopic video image data in the SydeBySide format. In FIG. 17, the boundary between the L image and the R image is shown as an image boundary 1408. The image boundary 1408 between the L image and the R image normally exists in the center of the one-field screen 1400. In FIG. 17, the current processing block 1402 and the block 1406 for which a motion vector used as a candidate vector has already been detected exist across the image boundary 1408. In other words, the current processing block 1402 exists at the right end of the L image, and the block 1406 for which a motion vector has been detected exists at the left end of the R image.

  As shown in FIG. 17, the motion compensation IP conversion processing device divides a one-field screen 1400 into M × N blocks 1401. In addition, a peripheral block 1403 having a high correlation with the current processing block 1402 exists around the current processing block 1402. FIG. 17 shows a state in which the process of generating the interpolation field image has progressed to some extent. In this case, the motion vectors of the blocks 1404, 1405, 1406, and 1407 in which motion vectors have been detected among the peripheral blocks 1403 having a high correlation with the current processing block 1402 are used as candidate vectors. That is, in the process of detecting the motion vector of the current processing block 1402 included in the L image, not only the blocks 1404, 1405, and 1407 included in the same L image but also the block 1406 included in the R image is used. As described above, when the motion vector of the current processing block 1402 existing in the L image is detected, the motion vector of the block 1406 existing in the R image is mixed into the candidate vector.

  Therefore, there is a problem that an incorrect motion vector is detected as the motion vector of the current processing block 1402, and an incorrect IP conversion result is generated. For example, even if the orientation of the candidate vector of the block 1406 is different from the original motion vector in the current processing block 1402, the SAD of the candidate vector of the block 1406 out of the blocks 1404 to 1407 happens to be the minimum value. There is a case. In this case, the motion vector of the current processing block 1402 is not correctly detected, and erroneous conversion occurs in IP conversion.

  Even when the current processing block moves to the left end of the R image, the motion vector of the right end block of the L image is mixed in the candidate vector. Therefore, even in this case, there is a problem that an incorrect IP conversion result is generated.

  As described above, in the technique disclosed in Patent Document 1, when the 3D image data in SideBySide format is IP-converted, the L image refers to the R image data at the boundary portion between the L image and the R image, and the R image is IP conversion is performed with reference to L image data. Therefore, there is a problem that an incorrect IP conversion result is generated.

  Further, although illustration and detailed description are omitted, even when the TopAndBottom format stereoscopic image data shown in FIG. 15 is IP-converted, at the boundary between the lower end of the L image and the upper end of the R image, for stereoscopic video in the SideBySide format It is clear that there is a problem that an incorrect IP conversion result is generated as in the case of image data.

  In other words, the technique disclosed in Patent Document 1 cannot correctly detect a motion vector at the boundary between the L image and the R image, and therefore correctly generates interpolated image data that interpolates between image data for stereoscopic video. There is a problem that cannot be done.

JP 2006-217486 A

  As described in the background art, the technique disclosed in Patent Document 1 has a problem that interpolation image data for interpolating between image data for stereoscopic video cannot be generated correctly.

  An image processing apparatus according to a first aspect of the present invention includes: a first image that displays a stereoscopic image; a stereoscopic image data that indicates an image including a second image; the first image and the second image; A first data processing unit that generates processed image data indicating an image including a blank image of one block or more and a processed interpolation image indicated by the processed interpolation image data that interpolates between the plurality of processed image data. A candidate vector determining unit that determines motion vectors of a plurality of second blocks that are included in a predetermined range from the block of which the motion vectors have been detected as candidate vectors of motion vectors of the first block; Based on the determined motion vector, a motion vector determination unit that determines a motion vector of the first block from the determined plurality of candidate vectors. An image interpolation unit that interpolates an image in the lock and generates the processed interpolated image data, and a plurality of stereoscopic image data used to generate the plurality of processed image data based on the generated processed interpolated image data And an output unit that outputs stereoscopic interpolation image data for interpolating between them.

  The image processing method according to the second aspect of the present invention includes a first image that displays a stereoscopic image and a stereoscopic image data that indicates an image including the second image, and the first image and the second image. Processed image data indicating an image including a blank image of one block or more is generated between the first blocks included in the processed interpolation image indicated by the processed interpolation image data for interpolating between the plurality of processed image data. The motion vectors of a plurality of second blocks that have been detected within the range of the first block are determined as candidate vectors of motion vector candidates of the first block, and from the determined plurality of candidate vectors , Determining a motion vector of the first block, and interpolating an image in the first block based on the determined motion vector to generate the processed interpolated image data Based on said generated processed interpolated image data, and outputs the three-dimensional interpolation image data to a plurality of interpolating between stereoscopic image data used to generate the plurality of processed image data.

  An image processing program according to a third aspect of the present invention includes: a first image that displays a stereoscopic image; a stereoscopic image data indicating an image including a second image; the first image and the second image; Processing for generating processed image data indicating an image including one or more blank images in between, and a first block included in the processed interpolation image indicated by the processed interpolation image data for interpolating between the plurality of processed image data A process of determining motion vectors of a plurality of second blocks included in a predetermined range from which motion vectors have been detected as candidate vectors of motion vector candidates of the first block; A process for determining a motion vector of the first block from the candidate vectors, and interpolating an image in the first block based on the determined motion vector Processing for generating the processed interpolated image data, and output of the interpolated image data for interpolating between the plurality of stereoscopic image data used for generating the plurality of processed image data based on the generated processed interpolated image data Processing to be executed by a computer.

  According to each aspect of the present invention described above, since the blank image is included between the first image and the second image, the first image is near the boundary between the first image and the second image. When determining a motion vector of a block, the second block included in an image that does not include the first block among the first image and the second image is prevented from being determined as a motion vector. Can do.

  According to each aspect of the present invention described above, it is possible to provide an image processing apparatus, an image processing method, and an image processing program that can correctly generate interpolated image data for interpolating between image data for stereoscopic video.

1 is a configuration diagram of a stereoscopic video display system according to a first exemplary embodiment of the present invention. It is an operation | movement flowchart which shows the process of the data processing part concerning Embodiment 1 of this invention. It is an operation | movement flowchart which shows the blank data synthetic | combination process in the case of processing the image data of SideBySide format. It is a block diagram of the image displayed by the image data after the blank data synthetic | combination process with respect to the image data of SideBySide format. It is an operation | movement flowchart which shows the blank data synthetic | combination process in the case of processing the image data of a TopAndBottom format. It is a block diagram of the image displayed by the image data after the blank data synthetic | combination process with respect to the image data of TopAndBottom format. It is a conceptual diagram of the motion vector detection concerning Embodiment 1 of this invention. It is explanatory drawing of the block made into the candidate vector concerning Embodiment 1 of this invention. It is a block diagram of the three-dimensional video display system concerning Embodiment 2 of this invention. It is a conceptual diagram of the motion vector detection method concerning patent document 1. FIG. 1 is a configuration diagram of a motion compensated IP conversion processing device according to Patent Document 1. FIG. It is explanatory drawing of the block made into the candidate vector concerning patent document 1. FIG. It is a figure which shows the image of FrameSequential format. It is a figure which shows the image of SideBySide format. It is a figure which shows the image of TopAndBottom format. It is a figure for demonstrating the problem at the time of using the image data for stereoscopic images for the technique concerning patent document 1. FIG. It is a figure for demonstrating the problem at the time of using the image data for stereoscopic images for the technique concerning patent document 1. FIG.

Embodiment 1 of the present invention.
With reference to FIG. 1, the structure of the three-dimensional-video display system concerning Embodiment 1 of this invention is demonstrated. FIG. 1 is a configuration diagram of a stereoscopic video display system according to Embodiment 1 of the present invention.

  The stereoscopic video display system 10 includes an image generation unit 101, a data processing information storage unit 102, a data processing unit 103, an IP conversion processing unit 104, and an image output unit 105.

  The image generation unit 101 generates stereoscopic video image data (hereinafter referred to as “image data”) indicating stereoscopic video from input data 106 input from the outside. The image generation unit 101 generates image data based on information acquired from the data processing information storage unit 102.

  The data processing information storage unit 102 stores format information, image size, blank data information, and the like. The format information is information indicating the format of image data output from the image generation unit 101, for example. The image size is, for example, information indicating the overall size of the image displayed by the image data output from the image generation unit 101, the size of each of the L image and the R image, the size of the blank data image to be added to the image, and the like. It is. In other words, the image size is information indicating the overall size of the image data and the sizes of the L image data portion, the R image data portion, and the blank data portion of the image data. Blank data information is information indicating blank data to be added to image data. Blank data is information indicating a pixel value of a blank data image. The information stored in the data processing information storage unit 102 is input with arbitrary contents by an user via an input device (not shown), for example. The input device is, for example, a remote controller, a mouse, a keyboard, or the like. Information stored in the data processing information storage unit 102 may also be extracted by analyzing the input data 106. In this case, information stored in the data processing information storage unit 102 is input from an input data analysis unit (not shown), the image generation unit 101, or the like.

  The data processing unit 103 adds blank data to the image data generated by the image generation unit 101 using an internal buffer (not shown). The internal buffer is a storage device such as a register and a memory. The data processing unit 103 adds blank data to the image data based on the information acquired from the data processing information storage unit 102.

  The IP conversion processing unit 104 performs IP conversion on the image data to which the blank data is added by the data processing unit 103. The IP conversion processing unit 104 has an internal memory (not shown) in which image data for a plurality of fields necessary for performing IP conversion processing is stored. The IP conversion processing unit 104 functions as a vector determination unit and an image interpolation unit.

  The image output unit 105 displays the L image and the R image as the output image 107 based on the image data of the L image data unit and the R image data unit among the image data IP converted by the IP conversion processing unit 104 ( (Not shown). The image output unit 105 displays the image data on the display device by, for example, outputting image data for displaying the output image 107 to the display device. The display device is, for example, a flat panel display such as a liquid crystal display, a plasma display, and an organic EL display. The image output unit 105 functions as an output unit.

  Then, with reference to FIG. 1, the process of the three-dimensional-video display system 10 concerning Embodiment 1 of this invention is demonstrated.

  The image generation unit 101 generates image data from input data 106 input from the outside. Here, the input data 106 is, for example, terrestrial digital broadcast signal data, image signal data from an external system based on HDMI (High Definition Multimedia Interface), or the like. The input data 106 includes image data encoded by an arbitrary codec. The image data is encoded by, for example, MPEG (Moving Picture Expert Group) format. The image generation unit 101 decodes the image data included in the input data 106 based on the encoding information acquired from the data processing information storage unit 102. The encoding information is information including a calculation formula for encoding, for example. The image generation unit 101 outputs the image data generated by decoding to the data processing unit 103. Note that the image data generated by the image generation unit 101 is image data for displaying an interlaced image.

  The data processing unit 103 is one block that is a processing unit when the IP conversion processing unit 104 performs processing between the L image and the R image included in the image displayed by the image data output from the image generation unit 101. Image data for displaying an image to which a blank data image having a width of one minute is added is generated. A block is a unit for detecting a motion vector when generating interpolated image data for interpolating between a plurality of image data, as will be described later. The data processing unit 103 adds blank data to the image data based on the format information, image size, and blank data information acquired from the data processing information storage unit 102. The data processing unit 103 adds blank data so that the pixel value differences of all blocks included in the blank data image are the same.

  The data processing unit 103 outputs the image data with the blank data added to the IP conversion processing unit 104. The data processing unit 103 stores the image size information for specifying the overall size of the image after adding the blank data image and the positions of the L image and the R image excluding the blank data image of the image. (Not shown). In other words, the image position information is information for specifying the overall size of the image data to which blank data is added and the positions of the L image data portion and the R image data portion excluding the blank data portion of the image data. . The image position information storage unit is a storage device such as a register, a memory, and a hard disk.

  The IP conversion processing unit 104 stores the image data output from the data processing unit 103 in an internal memory. The IP conversion processing unit 104 identifies image data for each field based on the image size stored in the image information storage unit. The IP conversion processing unit 104 stores image data for the number of fields necessary for IP conversion in the internal memory, and performs IP conversion in order from the field where IP conversion is possible. The IP conversion processing unit 104 outputs the image data after IP conversion to the image output unit 105.

  The image output unit 105 displays an L image and an R image based on the image data of the L image data unit and the R image data unit, excluding the blank data unit, of the image data output from the IP conversion processing unit 104. Output to. Note that the image output unit 105 determines the positions of the L image data unit and the R image data unit obtained by removing the blank data unit from the image data based on the image position specifying information acquired from the data processing information storage unit 102 and the image information storage unit. Is identified. Further, the image output unit 105 may specify the positions of the L image data unit and the R image data unit obtained by removing the blank data unit from the image data based on the image size acquired from the data processing information storage unit 102. Good.

  Next, the processing of the data processing unit 103 will be described with reference to FIG. FIG. 2 is an operation flowchart showing processing of the data processing unit 103 according to the first embodiment of the present invention.

  The data processing unit 103 acquires format information, image size, blank data information, and the like from the data processing information storage unit 102 (S001). The data processing unit 103 acquires the image data output from the image generation unit 101 (S002). The data processing unit 103 adds blank data to the image data based on the format information, image size, and blank data information acquired from the data processing information storage unit 102 (S003). For example, the data processing unit 103 specifies the format of the input image data based on the format information. Then, the data processing unit 103 executes blank data synthesis processing according to the specified format.

  Here, with reference to FIGS. 3 to 6, the blank data synthesis process in step S <b> 003 will be described in detail. First, with reference to FIG. 3, a blank data composition process in the case of processing image data in SideBySide format will be described. FIG. 3 is an operation flowchart showing blank data synthesis processing when processing image data in SideBySide format.

  When processing the image data in SideBySide format, the data processing unit 103 performs IP conversion processing on the first line data in the L image data portion of the image data based on the image size acquired from the data processing information storage unit 102. The data is output to the unit 104 (S201). That is, the image size stored in the data processing information storage unit 102 includes information that can specify the widths of the L image data unit, the R image data unit, and the blank data unit in one line. Next, the data processing unit 103 outputs blank data for the first line to the IP conversion processing unit 104 based on the image size acquired from the data processing information storage unit 102 (S202). Here, one block of blank data is output for one line. Next, the data processing unit 103 outputs the data of the first line in the R image data portion of the image data to the IP conversion processing unit 104 (S203).

  Next, the data processing unit 103 determines whether or not all lines of image data have been output (S204). Here, assuming that there are three or more lines of image data, the data processing unit 103 determines that all lines of the image data have not been output (S204: No). If so, the data processing unit 103 repeats the same processing as steps S201 to S203 until the image data of all lines is output in the order of the second line and the third line. The IP conversion processing unit 104 stores the image data input from the data processing unit 103 in the internal memory.

  If the data processing unit 103 determines that all the lines of the image data have been output (S204: Yes), the process for the image data ends.

  Here, FIG. 4 shows an image displayed by the image data output to the IP conversion processing unit 104 by the processing shown in FIG. As shown in FIG. 4, the image data in the SideBySide format is such that the image displayed by the image data has a blank data image 303 having a width of 1 block added between the L image 301 and the R image 302. To be processed.

  Next, with reference to FIG. 5, a blank data composition process in the case of processing image data in the TopAndBottom format will be described. FIG. 5 is an operation flowchart showing blank data synthesis processing when processing image data in the TopAndBottom format.

  When processing the image data in the TopAndBottom format, the data processing unit 103 outputs all lines of the L image data portion of the image data to the IP conversion processing unit 104 (S401). The data processing unit 103 recognizes the sizes of all lines in the L image data unit based on the image size acquired from the data processing information storage unit 102. Next, the data processing unit 103 outputs blank data of the number of lines corresponding to one block width to the IP conversion processing unit 104 (S402). Next, the data processing unit 103 outputs all lines of the R image data unit to the IP conversion processing unit 104 (S403). The data processing unit 103 recognizes all lines in the R image data unit based on the image size acquired from the data processing information storage unit 102.

  Here, FIG. 6 shows an image displayed by the image data input to the IP conversion processing unit 104 by the processing shown in FIG. As shown in FIG. 6, the image data in the TopAndBottom format is processed so that the image displayed by the image data has a shape in which a blank data image 503 is added between the L image 501 and the R image 502. .

Next, a blank data composition process when processing image data in the FrameSequential format will be described.
When processing the image data in the FrameSequential format, the data processing unit 103 does not process the image data for displaying the L image and the image data for displaying the R image as one image data, except that FIG. 3 or FIG. The same processing as that described with reference to FIG. That is, the data processing unit 103 generates image data for displaying one image in which each of the L images 301 and 501 and the R images 302 and 502 in FIGS. 4 and 6 has a size corresponding to one interlaced image.

  Specifically, the following processing may be performed. When image data for displaying an L image is input from the image generation unit 101, the data processing unit 103 stores the image data in an internal buffer. Thereafter, when input of image data for displaying an R image is started, the processing in FIG. 3 or FIG. 5 is started. Note that it is possible to arbitrarily select which of the processes shown in FIG. 3 or FIG. 5 is to be performed.

  Next, processing in the IP conversion processing unit 104 will be described with reference to FIGS. 7 and 8. First, a motion vector detection process when blank data is added to the image data in the FrameSequential format by the process shown in FIG. 3 will be described with reference to FIG. FIG. 7 is a conceptual diagram of motion vector detection according to the first exemplary embodiment of the present invention.

  Here, in the FrameSequential format in the present embodiment, it is assumed that two lines of odd-numbered interlaced images and even-numbered interlaced images are alternately arranged in time series. In this case, when an odd line L image, an odd line R image, an even line L image, an even line R image, an odd line L image, and an odd line R image are arranged in the order of fields m to m + 5. (M is an arbitrary positive integer).

  In this case, as shown in FIG. 7, the data processing unit 103 generates image data in which a blank data image is added between the L image and the R image. That is, FIG. 7 shows an image 600 displayed by the image data generated from the image data of the field m and the field m + 1 and an image 602 displayed by the image data generated from the image data of the field m + 4 and the field m + 5. ing. Note that FIG. 7 illustrates the case where the image data is converted into one image data as shown in FIG. 4 by the process shown in FIG. For example, when the IP conversion processing unit 104 interpolates the image data that displays the odd-line L image in the field m + 2 and the image data that displays the odd-line R image in the field m + 3, the image data and the image of the image 600 Based on the image data 602, the interpolated image data of the interpolated image 601 is generated.

  When generating image data for displaying the interpolated image 601, the IP conversion processing unit 104 divides the image displayed by the image data into a plurality of blocks 607 each having M × N pixels. Here, each of M and N is an arbitrary positive integer. Then, the IP conversion processing unit 104 performs processing in block units from the upper left to the lower right of the screen as in the normal scanning line method.

  In the process of generating image data for displaying the interpolated image 601, the IP conversion processing unit 104 compares the current processing block 604 with reference block 1 (603) of the image 600 referred to by the candidate vector 606 and the image 602. SAD with reference block 2 (605) is calculated.

  At this time, the previous image 600 and the next image 602 from the interpolation image 601 on the time axis have the same configuration as the interpolation image 601 as shown in FIG. That is, an arbitrary block of the previous image 600 and a block at the same position as the arbitrary block of the subsequent image 602 are included in the same image of the L image and the R image. Therefore, even if the reference block 1 (603) of the previous image 600 and the reference block 2 (605) of the next image 602 are used, the L image and the R image are mixed and IP conversion is performed. Nothing will happen. As a result, IP conversion can be performed correctly.

  Also, when blank data is added to the image data in the FrameSequential format by the process shown in FIG. 5, as described above, an arbitrary block of the previous image and an arbitrary block of the subsequent image It is obvious that the block at the same position is included in the same image of the L image and the R image. Therefore, detailed description is omitted. In other words, even when image data with blank data added as shown in FIG. 6 is generated by the processing shown in FIG. 5, the L image and the R image are not mixed to perform IP conversion. IP conversion can be performed correctly.

  Next, a motion vector detection process when blank data is added to the image data in SideBySide format by the process shown in FIG. 3 will be described with reference to FIG. FIG. 8 is an explanatory diagram of blocks as candidate vectors according to the first embodiment of the present invention.

  Here, the interpolated image data for displaying the interlaced image in the field n-1 based on the image data for displaying the interlaced image in the field n-2 and the image data for displaying the interlaced image in the field n is shown here. The case of interpolation will be described (n is an arbitrary positive integer). That is, a case will be described in which interpolated image data for displaying an interlaced image combined with an interlaced image in field n-1 is generated in order to generate a progressive image in field n-1. For example, an interlaced image in fields n and n-2 is an image displayed on odd lines, and an interlaced image in field n-1 is an image displayed on even lines.

  The IP conversion processing unit 104 divides the image 700 in one field into a plurality of blocks 701 of M × N pixels in order to generate interpolation image data for displaying the interpolation field image (field n−1 image) 700. Further, eight peripheral blocks 703 having a high correlation with the current processing block 702 exist around the current processing block 702. FIG. 8 shows a state where the process of generating the interpolation field image 700 has progressed to some extent. In this case, the IP conversion processing unit 104 sets the motion vectors of the detected blocks 704, 705, 706, and 707 in the peripheral block 703 having a high correlation with the current processing block 702 as candidate vectors.

  That is, as shown in FIG. 8, the motion vector of the block included in the R image is not used as the candidate vector, and the motion vector of the block 706 existing on the blank data is used as the candidate vector. Here, the block 706 existing on the blank data in the interpolation field image 700, the block at the same position as the block 706 in the image before one field (field n-2 image) and the image after one field (field n image), The pixel values included in these blocks are always the same value. Therefore, no motion vector is detected at block 706. Therefore, the candidate vector in block 706 does not adversely affect the IP conversion process in current processing block 702.

  Next, the reason will be specifically described. As described above, the candidate vector in block 706 is zero. Therefore, for example, when the current processing block 702 originally has a motion vector, the SAD calculated using the candidate vector in the block 706 does not become the minimum value. Therefore, in this case, not the candidate vector of the block 706 but any of the candidate vectors of the blocks 704, 705, and 707 is detected as the motion vector of the current processing block 702.

  That is, one of the candidate vectors of the block included in the L image is detected as the motion vector of the current processing block 702. Therefore, as described in the background art, the block candidate vectors included in the R image are not detected as motion vectors. As a result, as a motion vector of the current processing block 702, the SAD accidentally has a minimum value, but a candidate vector in a direction different from the original is not determined as a motion vector. That is, it is possible to prevent an erroneous IP conversion result from being generated at the boundary between the right end of the L image and the left end of the R image.

  On the other hand, it is also conceivable that the SAD calculated based on the candidate vector of the block 706 becomes the minimum value. However, in this case, the candidate vector of block 706 is zero. Therefore, the candidate vector of the block 706 does not become a motion vector in the direction different from the original although the SAD accidentally becomes the minimum value. Therefore, since the candidate vector of the block 706 in this case is a valid motion vector, there is no problem even if it is detected as a motion vector. In other words, in this case, it is considered that the motion vectors in the blocks 704, 705, and 707 having a high correlation with the current processing block 702 are also close to zero.

  The IP conversion processing unit 104 determines a candidate vector having a minimum SAD as a motion vector of the current processing block 702 from among the candidate vectors. The IP conversion processing unit 104 calculates the pixel value of the current processing block 702 based on the determined motion vector, and interpolates the image in the current processing block 702. In this way, the IP conversion processing unit 104 interpolates the images in all the blocks included in the interpolation field image 700, and generates interpolation image data for displaying the interpolation field image 700. The IP conversion processing unit 104 combines the generated interpolated image data with the image data in the same field as the interpolated image data, and generates image data for displaying a progressive image. Here, for example, the IP conversion processing unit 104 acquires, from the data processing unit 103, image data for displaying an interlaced image interpolated by an interlaced image displayed by the interpolated image data. Then, the IP conversion processing unit 104 outputs the image data of the progressive image to the image output unit 105.

  The image output unit 105 outputs, as one image, an R image and an L image excluding a blank image among images displayed by the image data output from the IP conversion processing unit 104 to the display device. As a result, among the interlaced image displayed by the interpolated image data and the interlaced image interpolated by the interlaced image, the R image and the L image excluding the blank image are output as one progressive image.

  According to the above-described embodiment, even when blank data shown in FIG. 5 is added to the image data in the TopAndBottom format, an incorrect IP conversion result is generated at the boundary between the lower end of the L image and the upper end of the R image. Obviously not. Therefore, detailed description is omitted.

  As described above, in the first embodiment, the data processing unit 103 includes an L image and an R image that display a stereoscopic image when generating interpolated image data that interpolates between a plurality of stereoscopic image data. Processed image data indicating an image including one or more blank data images between the L image and the R image is generated from the stereoscopic image data indicating the image. The IP conversion processing unit 104 includes motion vectors of a plurality of blocks that are included in a predetermined range from the current processing block included in the image indicated by the processed interpolated image data that interpolates between the plurality of processed image data, and for which the motion vectors have been detected. Thus, it is determined as a candidate vector for the motion vector of the current processing block. The IP conversion processing unit 104 interpolates the image in the current processing block based on the determined motion vector so as to generate processed interpolation image data. Then, the image output unit 105 outputs interpolated image data for interpolating between a plurality of image data used to generate a plurality of processed image data based on the processed interpolated image data.

  According to this, since the blank data image is included between the L image and the R image, the L image and the R image are determined when determining the motion vector of the current processing block in the vicinity of the boundary between the L image and the R image. It is possible to prevent a candidate block included in an image that does not currently include a processing block from being determined as a motion vector. Therefore, it is possible to correctly generate interpolated image data that interpolates between image data for stereoscopic video.

  In the first embodiment, the data processing unit 103 indicates image data indicating the L image in the first field and the R image in the second field subsequent to the first field, as in the FrameSequential format. Processed image data indicating one image is generated from the image data.

  According to this, on the time axis, from the processed interpolation image data, an arbitrary block in the image indicated by the previous processed image data and the same position as the arbitrary block in the image indicated by the subsequent processed image data These blocks are included in the same image of the L image and the R image. Therefore, even when the L image and the R image exist alternately along the time series as in the FrameSequential format, the L image and the R image are not mixed to interpolate the image of the current processing block. . That is, the interpolation image data can be generated correctly.

Embodiment 2 of the present invention.
With reference to FIG. 9, the structure of the three-dimensional-video display system concerning Embodiment 2 of this invention is demonstrated. FIG. 9 is a configuration diagram of a stereoscopic video display system according to the second embodiment of the present invention.

  The stereoscopic video display system 80 includes an image generation unit 801, a data processing information storage unit 802, a data processing unit 803, an IP conversion processing unit 804, an image output unit 805, and an image data memory 806.

The point that the image generation unit 801 stores the generated image data in the image data memory 806 and the point that the image generation completion notification is output to the data processing unit 803 when the image data is generated are described in the first embodiment. Different from the image generation unit 101.
The data processing information storage unit 802 functions in the same manner as the data processing information storage unit 102 according to the first embodiment.

  The data processing unit 803 acquires image data from the image data memory 806, adds blank data to the acquired image data, stores image data with the blank data added thereto in the data processing information storage unit 802, and The difference from the data processing unit 103 according to the first embodiment is that a data processing completion notification is output to the IP conversion processing unit 804 when image data with blank data added is generated.

Embodiment 1 is that the IP conversion processing unit 804 acquires image data with blank data added from the image data memory 806 and stores the image data obtained by IP conversion of the acquired image data in the image data memory 806. Different from the IP conversion processing unit 104 according to the above.
The image output unit 805 is different from the image output unit 105 according to the first embodiment in that the IP-converted image data is acquired from the image data memory 806.

  The image data memory 806 stores image data. The image data memory 806 is not limited to a memory. For example, the image data memory 806 may be another storage device such as a memory and a hard disk. In the second embodiment, since the image data is stored in the image data memory 806, the internal buffer of the data processing unit 803 and the internal memory of the IP conversion processing unit 804 are not necessary.

  Next, with reference to FIG. 9, processing of the stereoscopic video display system 80 according to the second embodiment of the present invention will be described.

  The image generation unit 801 generates image data from input data 106 input from the outside. The image generation unit 801 stores the generated image data in the image data memory 806. The image generation unit 801 stores the address where the image data on the image data memory 806 is stored in the image information storage unit. Then, the image generation unit 801 outputs an image generation completion notification to the data processing unit 803.

  The data processing unit 803 acquires the image data generated by the image generation unit 801 from the image data memory 806 in response to the image generation completion notification output from the image generation unit 801. Note that the data processing unit 803 acquires image data from the address stored in the image information storage unit by the image generation unit 801. The data processing unit 803 adds blank data to the acquired image data. The data processing unit 803 stores the image data obtained by processing the blank data in the image data memory 806. The data processing unit 803 stores the address where the image data on the image data memory 806 is stored in the image information storage unit. Then, the data processing unit 803 outputs a data processing completion notification to the IP conversion processing unit 804.

  The IP conversion processing unit 804 acquires the image data generated by the data processing unit 803 from the image data memory 806 in response to the data processing completion notification output from the data processing unit 803. Note that the IP conversion processing unit 804 acquires image data from the address stored in the image information storage unit by the data processing unit 803. The data processing unit 803 acquires image data of the number of fields necessary for IP conversion, and stores it again in a work area different from the area where the acquired image data is stored. The IP conversion processing unit 804 stores the image data of the number of fields necessary for IP conversion in the work area, and performs IP conversion in order from the field where IP conversion is possible. The IP conversion processing unit 804 stores the image data after IP conversion in the image data memory 806. The IP conversion processing unit 804 stores the address where the image data on the image data memory 806 is stored in the image information storage unit.

  The image output unit 805 acquires the image data generated by the IP conversion processing unit 804 from the image data memory 806 at an arbitrary output timing. Note that the image output unit 805 acquires image data from the address stored in the image information storage unit by the IP conversion processing unit 804. The image output unit 805 outputs the L image and the R image as the output image 107 to the display device based on the image data of the L image data portion and the R image data portion obtained by removing the blank data portion from the acquired image data.

  Here, in home video equipment such as a television, the timing from when input data is input to when an image is output is adjusted. This timing is such that the image data after the IP conversion can be output at an arbitrary timing by temporarily storing the image data in the memory between the input of the input data and the output of the image. The method of adjustment is common.

  As described above, in the second embodiment, the IP conversion processing unit stores the image data after IP conversion in the image data memory. The image output unit acquires image data stored in the image data memory, and outputs an output image based on the acquired image data. According to this, an image after IP conversion can be output at an arbitrary timing.

Note that the present invention is not limited to the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention.
In the present embodiment, the case where image data to be interpolated in IP conversion is generated has been exemplified. However, the present invention is not limited to this as long as image data to be interpolated with stereoscopic video image data is generated. For example, the present invention can be applied to a case where image data to be simply interpolated between a plurality of image data is added without performing IP conversion.

  In this embodiment, one block of blank data image is added between the L image and the R image, but the size of the blank data to be added is not limited to this. That is, a blank data image having a size of one block or more may be added between the L image and the R image.

  In the present embodiment, candidate vectors are selected from eight blocks within the range of one block from the current processing block. However, if they are within a predetermined range in which the correlation with the current processing block is high, Not limited. For example, when the correlation with the current processing block is high from the current processing block to the blocks within the range of 2 blocks, 24 blocks within the range of 2 blocks from the current processing block may be used as candidate vectors. In this case, for example, by adding a blank data image having a size of 2 blocks or more, a candidate block included in an image that does not include the current processing block among the L image and the R image is determined as a motion vector. Can be prevented.

  In this embodiment, a motion vector is detected from a candidate vector by calculating SAD. However, a method for evaluating a candidate vector based on a difference in pixel values between reference block 1 and reference block 2 If so, it is not limited to this. For example, the candidate vector may be evaluated by a sum of squared differences (hereinafter referred to as “SSD (Sum Of Squared Difference)”).

  In the second embodiment, the image generation unit 801, the data processing unit 803, the IP conversion processing unit 804, and the image output unit 805 use the image data memory 806, but all of these use the image data memory 806. You don't have to. Specifically, in the second embodiment, the image generation unit 801 stores image data, the data processing unit 803 acquires image data, and the data processing unit 803 stores image data. The IP conversion processing unit 804 includes means B for acquiring image data, the IP conversion processing unit 804 stores image data, and the image output unit 805 includes means C for acquiring image data. Of these means A to C, any one or two means may be implemented. That is, for the processes corresponding to any one or two of the means A to C, the image data memory 806 may not be used as in the first embodiment. For example, when only the means A and C are mounted, the data processing unit 803 directly outputs the image data to the IP conversion processing unit 804 without storing the image data in the image data memory 806.

  The stereoscopic video display system according to the present invention described above includes a program for realizing the functions of the above-described embodiments, a CPU (Central Processing Unit), MPU (Micro Processing Unit), MCU (Micro Control) included in a computer or system. Unit) can be configured by executing.

  The program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)) are included. The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

  Further, the present invention is not limited to the case where the function of the above-described embodiment is realized by executing a program that realizes the function of the above-described embodiment. A case where this program realizes the functions of the above-described embodiment in cooperation with an OS (Operating System) or application software running on the computer is also included in the embodiment of the present invention. Further, when all or part of the processing of the program is performed by a function expansion board inserted into the computer or a function expansion unit connected to the computer, the functions of the above-described embodiment are realized. It is contained in embodiment of invention.

10, 80 3D image display system 101, 801 Image generation unit 102, 802 Data processing information storage unit 103, 803 Data processing unit 104, 804 IP conversion processing unit 105, 805 Image output unit 106, 806 Input data 107, 807 Output image 301, 501 L image 302, 502 R image 303, 503 Blank data image 600, 602 Image 601 Interpolated image 603, 1003, 1303 Reference block 1
604, 702, 1004, 1102, 1304, 1402 Current processing block 605, 1005, 1305 Reference block 2
606, 913, 1006, 1306 Candidate vectors 607, 701, 1101, 1401 Block 700 One field image 808 Image data memory 703, 1103, 1403 Peripheral blocks 704, 705, 706, 707, 1104, 1105, 1106, 1107, 1404, 1405, 1406, 1407 Blocks used as candidate vectors 900 SAD operation unit 901 SAD minimum value determination unit 902 Candidate vector generation unit 903 Motion vector determination unit 904 Memory 910 One-field previous image data 911 One-field subsequent image data 912 Operation result 914 Determination result 915 motion vector 916 already detected vector 1000 field n-2 image 1001 field n-1 image (interpolation)
1002 Field n image 1007, 1307 Pixel block 1100, 1400 1 field screen 1108 Processed block 1300 Image before 1 field (field n-2 image)
1301 Interpolated field image (field n-1 image)
1302 Image after one field (field n image)
1408 Image boundary between R image and L image

Claims (10)

A processed image showing an image including a blank image of one block or more between the first image and the second image from the stereoscopic image data indicating the first image and the second image displaying the stereoscopic image. A data processing unit for generating data;
Motion vectors of a plurality of second blocks included in a predetermined range from the first block included in the processed interpolation image indicated by the processed interpolation image data for interpolating between the plurality of processed image data A candidate vector determination unit that determines a candidate vector of motion vector candidates of the first block;
A motion vector determination unit that determines a motion vector of the first block from the determined plurality of candidate vectors;
An image interpolation unit that interpolates an image in the first block based on the determined motion vector and generates the processed interpolated image data;
Based on the generated processed interpolation image data, an output unit that outputs stereoscopic interpolation image data for interpolating between the plurality of stereoscopic image data used to generate the plurality of processed image data;
An image processing apparatus. The stereoscopic image data includes first image data indicating the first image in a first field and second image data indicating the second image in a second field subsequent to the first field. Including
The image processing apparatus according to claim 1, wherein the data processing unit generates processed image data indicating one image from the first image data and the second image data.   The image processing apparatus according to claim 1, wherein the stereoscopic image data is image data indicating an image including the first image and the second image in the same field. The image processing apparatus further includes an image data storage unit that stores the processed interpolation image data,
The image interpolation unit stores the generated processed interpolation image data in the image data storage unit,
The output unit acquires the processed interpolation image data stored in the image data storage unit, and outputs the stereoscopic interpolation image data based on the acquired processed interpolation image data. The image processing apparatus according to item. The image processing apparatus further includes an image information storage unit that stores position specifying information for specifying the positions of the first image and the second image in the processed interpolation image,
The image interpolation unit stores position specifying information of the generated processed interpolation image data in the image information storage unit,
The output unit outputs image data indicating the first image and the second image from the generated processed interpolation image data based on the position specifying information stored in the image information storage unit. The image processing apparatus according to claim 1, wherein the image processing apparatus outputs the interpolated image data.   The motion vector determination unit determines the motion vector based on a difference in pixel values between blocks specified based on the candidate vector and included in each of the images indicated by the plurality of processed image data. 6. The image processing apparatus according to any one of items 1 to 5.   The first image and the second image are an image for the left eye and an image for the right eye, or are an image for the right eye and an image for the left eye, respectively. Image processing apparatus. The stereoscopic image data is image data indicating an interlaced image,
8. The stereoscopic interpolated image data is image data indicating an interlaced image that is interpolated to convert the interlaced image indicated by the stereoscopic image data into a progressive image in IP (Interlace / Progressive) conversion. The image processing apparatus according to claim 1. An image including a blank image of one block or more between the first image and the second image from stereoscopic image data indicating an image including a first image and a second image displaying a stereoscopic image. Generate processed image data to show
Motion vectors of a plurality of second blocks included in a predetermined range from the first block included in the processed interpolation image indicated by the processed interpolation image data for interpolating between the plurality of processed image data As a candidate vector of motion vector candidates of the first block,
Determining a motion vector of the first block from the determined plurality of candidate vectors;
Based on the determined motion vector, an image in the first block is interpolated to generate the processed interpolated image data,
An image processing method for outputting stereoscopic interpolation image data for interpolating between a plurality of stereoscopic image data used for generating the plurality of processed image data based on the generated processed interpolation image data. An image including a blank image of one block or more between the first image and the second image from stereoscopic image data indicating an image including a first image and a second image displaying a stereoscopic image. Processing to generate processed image data to be shown;
Motion vectors of a plurality of second blocks included in a predetermined range from the first block included in the processed interpolation image indicated by the processed interpolation image data for interpolating between the plurality of processed image data Is determined as a candidate vector of motion vector candidates of the first block;
A process of determining a motion vector of the first block from the determined plurality of candidate vectors;
A process of interpolating an image in the first block based on the determined motion vector to generate the processed interpolated image data;
A process of outputting stereoscopic interpolation image data for interpolating between a plurality of stereoscopic image data used for generating the plurality of processed image data based on the generated processed interpolation image data;
An image processing program for causing a computer to execute.

Images