JP4706409B2 - Input display device and input display method - Google Patents

Description

  The present invention relates to an input display device and an input display method applicable to, for example, an EL (Electro Luminescence) display, and more particularly to an input display device and an input display method capable of capturing an input image in parallel with image display. .

  Conventionally, in a technique for displaying a three-dimensional stereoscopic image, coordinate input of the three-dimensional image and display of the three-dimensional image are realized by using different devices. For example, three-dimensional coordinate input techniques include laser scanning and CCD camera measurement techniques, stereo photography techniques used in aerial photography, techniques using a contact digitizer, and the like, and dedicated input equipment is used. Further, in order to visualize the input three-dimensional information, a display is installed in another place and displayed as information mapped on the two-dimensional display. For example, in order to express a three-dimensional image, there are a display and parallax barrier display technology using a lenticular lens that performs only display. In addition, there are technologies using integral photography, holography, and the like, which have a lot in common with imaging and display systems.

  As a full parallax image display system that enables viewpoint selection, for example, there is an IP (Integral Photography) stereoscopic image system. In IP technology, an image is taken through a lens array in which a large number of microlenses, such as insect compound eyes, are arranged, and a photographic plate on which a plurality of small subject images are formed is sandwiched by the photographic plate. Illuminate from a position opposite to the position of. Then, by reproducing the light beam in the reverse direction that follows the same locus as the light beam from the subject at the time of shooting toward the photographic plate, a three-dimensional image of the subject can be reproduced at the position where the subject is located. An IP stereoscopic image system that electronically realizes such IP technology using, for example, a digital camera or a liquid crystal display has been proposed.

  In the IP stereoscopic image system, the imaging and display processes are realized using the basic principle of IP. In the IP, information captured through the lens array is also displayed through the lens array. The IP stereoscopic image system basically includes a digital camera in which a lens array for imaging a subject is installed in front, and a liquid crystal display in which a lens array is installed on a surface for reproducing an image of the subject. At the time of imaging and reproduction, the subject at the time of imaging is observed as a three-dimensional real image at the same position based on the concept of ray reproduction in which the same optical path is traced in the opposite direction.

  Here, an example of a conventional image display system using a one-stage IP will be described with reference to FIG. When the real object 62 is photographed, an input light beam 66 is applied to an imaging plate 60 including a plurality of fly-eye lenses 61 including an imaging element (not shown). One fly-eye lens 61 includes a plurality of image sensors, and records an inverted single-eye image 63. At the time of image reproduction, when the reproduction illumination light 64 is irradiated through the imaging plate 60, the reproduction light beam 67 reaches the observer 65 and can be recognized as a three-dimensional image. However, with this one-stage IP, the viewer 65 looks from the back side of the real object 62 when observing, in other words, looks like the back side of the surface, so that it is suitable as a three-dimensional image. Is hard to say.

  An example of an image display system using a two-stage IP in which a reconstructed image is once again copied to the IP in order to solve the problem of flipping will be described with reference to FIG. In an IP stereoscopic image system using a two-stage IP, a three-dimensional object image is captured and displayed using two IPs. That is, the image information of the real object 62 is imaged by irradiating the input light beam 66a with the first IP configured by the imaging plate 60a and the fly-eye lens 61a. Then, when reproducing the image information captured with the first IP, the first IP reproduction illumination light 64a is irradiated through the first IP imaging plate 60a. The first IP reconstructed light beam 67b is focused on the real object 62 to be photographed, and then irradiated to the second IP composed of the imaging plate 60b and the fly-eye lens 61b to capture an image. Then, when reproducing the image information captured with the second IP, the second IP reproduction illumination light 64b is irradiated through the second IP imaging plate 60b. The second IP reproduction light beam 68 is focused on the real object 62 to be photographed, and then reaches the observer 65 and can be recognized as a three-dimensional image. Thus, it is considered that the observer can recognize a stereoscopic image that does not cause the problem of turning over when the second IP reproduction light beam is observed.

Patent Document 1 discloses an IP stereoscopic image display system.
JP 2002-296541 A

  Conventionally, the three-dimensional input technology has been used to measure an existing object or terrain, and is not suitable for an application of inputting while creating the shape of a three-dimensional object. That is, there is no technology in a three-dimensional space that visually confirms a figure as it is input, like a two-dimensional pen-input tablet. Also, in the integral photography and holography technologies, the filming system uses a film or a CCD camera, and the configuration is quite different from the display system, and there is no common shooting / display. .

  For example, when creating three-dimensional object information using conventional technology, after creating an object image using clay or the like, this object is scanned using existing three-dimensional input technology, and the coordinate information of the object, etc. Must be read. This is a two-dimensional replacement. After writing a figure on paper, the scanner performs the same operation as reading the figure information. Compared to an input device such as a pen-input tablet, this is just an alternative. It can be said that it is not too much. Not only that, but a very large system such as a three-dimensional scanner is required, so it is not very convenient.

  Also, in order to realize the IP stereoscopic image system, it is necessary to match the trajectory of the light beam from the subject at the time of photographing and the light beam in the reverse direction at the time of reproduction. It is required to accurately reproduce the positional relationship between the lens array and the lens array. In the conventional IP for the purpose of photography, such a two-step photographing process is performed, so that problems such as lens distortion occur twice, which is very difficult in terms of work accuracy. It is not realized according to the principle. Since then, various attempts have been made, such as a film incorporating a lens array. However, due to problems with work accuracy and manufacturing costs, the film has not yet been proposed.

  Therefore, in order to realize a three-dimensional imaging display system that satisfies such requirements, there are considerable restrictions in terms of technology and cost. In addition, since it is necessary to prepare a configuration of an imaging system composed of a digital camera and a lens array and a configuration of a reproduction system composed of a display and a lens array, not only costs are increased, but the system configuration becomes relatively large. End up.

  The present invention has been made in view of such circumstances, and an object thereof is to easily perform an input display operation in a three-dimensional space.

The present invention performs input and display using a display having a lens array layer in which a plurality of microlenses are arranged in a plane, and a pixel layer in which a plurality of pixels are arranged in a plane for each microlens to capture and display light rays. In this case, a light-receiving image is obtained from a signal captured by the pixel layer of the display, and an image obtained by the light-receiving processing unit is generated as a single-eye image for each microlens. Generate and extract a characteristic shape from a multi-viewpoint two-dimensional image, collate the characteristic shape with a template for input image recognition, select a feature point, and select multiple single-eye images from a single-eye image and a table of pixels by calculating the 3-dimensional coordinates of the feature points from a plurality of single-eye image selected, subjected to a process for calculating the three-dimensional motion amount near the surface of the display, the display of the pixel layer Driving gastric row, and stores the template for the input image recognition in the storage unit.

  As a result, a three-dimensional image that can be input and displayed simultaneously can be reproduced in a three-dimensional space on the display.

  According to the present invention, it is possible to simultaneously input and display a stereoscopic image on the display using a display capable of displaying a three-dimensional stereoscopic image in which information input and display are integrated. There is an effect of improving the operability of a three-dimensional image in a three-dimensional space.

  Hereinafter, an embodiment of the present invention will be described with reference to FIGS. In the present embodiment, a three-dimensional input display device capable of inputting and displaying a three-dimensional stereoscopic image in a three-dimensional space will be described as an example.

  First, an external configuration example of the three-dimensional input display device of this example will be described with reference to FIG. An input / output display 1 capable of inputting and displaying a three-dimensional stereoscopic image is a microlens array in which a number of microlenses called insect compound-eye fly-eye lenses are arrayed, and images are taken for each microlens. A display element is provided. The input / output display 1 performs image capturing and display processing by an information processing apparatus 10 that controls the system. The input light beam 4 emitted from the input pen 2a is applied to the microlens array of the input / output display 1 to record information. Then, the reproduction light beam 5 emitted from the imaging display element again converges on the locus 3 of the pen tip and then diverges. A device having such a configuration is referred to as a three-dimensional input display device 100. In the three-dimensional input display device 100 of this example, the user accumulates the coordinate information of the locus of the pen tip in time series from the input pen 2a that emits the pen tip in the space on the input / output display 1, and the accumulated coordinate information Is displayed as the locus 3 of the pen tip. In addition, it has a plurality of applications, and details will be described later.

  Next, an example of the input / output display 1 will be described with reference to FIG. A display surface 1d corresponding to the surface portion of the input / output display 1 has a three-layer structure of a microlens layer, a partition layer, and a pixel layer. On the surface of the input / output display 1 facing the subject, a microlens layer configured as a lens array is provided by arranging a large number of microlenses 1a in a planar shape. Further, the imaging display pixel 1c on which the subject image is formed is a pixel layer, and between the microlens layers, the light beam condensed by each microlens and the light beam condensed by the adjacent microlens are present. A partition wall 1b is provided as a partition layer so as not to interfere.

  With such a configuration, in the input / output display 1, a single-eye image is obtained as an image of a subject viewed through each of a large number of microlenses. The size of the input / output display panel 1 is arbitrary. The higher the lens array density and the panel pixel density, the higher the coordinate detection accuracy and the higher the resolution of the image.

  Next, an internal configuration example of the information processing apparatus 10 provided in the input / output display 1 will be described with reference to the block diagram of FIG. Note that the information processing apparatus 10 may be formed integrally with the input / output display 1 or may be formed as a separate housing.

  A CPU (Central Processing Unit) 13 that controls each unit of the information processing apparatus 10 and input display processing of the input / output display 1 is a program stored in a readable ROM (Read Only Memory) 11 or a mass storage device. Various processes can be executed in accordance with a program loaded in a RAM (Random Access Memory) 12 which can be written from the storage unit 16 constituted by a hard disk drive or the like. The RAM 12 appropriately stores data, parameters, and the like necessary for the CPU 13 to execute various processes.

  The ROM 11, RAM 12, and CPU 13 are connected to each other via a bus 25 and perform data transmission. In addition, the bus 25 is connected to the input / output display 1, the keyboard, the mouse, the input pen 2 a and the like input unit 14, the speaker and the like output unit 15, the storage unit 16, and an external terminal (not shown). It is also connected to a communication unit 17 that performs communication as a communication interface. Further, if necessary, a drive 50 is connected to the bus 25, and a removable storage medium 19 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a program read from the storage medium 19 is read. Are copied and installed in the storage unit 16 as necessary.

  At the time of image capturing, the subject is imaged by the light receiving drive unit 21 that performs image capturing processing. This imaging information is basically 30 or more pieces of moving image information per second. The light receiving drive unit 21 includes an imaging region setting unit 22 that sets an imaging region. A voltage in the vicinity of, for example, 0 V is applied to a pixel to be imaged, and the imaging region is formed by the pixel. Here, an area composed of a predetermined number of pixels on which one image of a subject performing an imaging operation is formed is referred to as an “imaging area”. And the light reception drive part 21 detects the output from the pixel which is performing imaging operation among the imaging display pixels of the input-output display 1, and is whether each pixel is irradiated with the light from the outside The presence or absence of such irradiation and the amount of light are detected. The detection result (pixel value) by the detection function of the light receiving drive unit 21 is output to the single eye image generation unit 23.

  Based on the detection result supplied from the light receiving drive unit 21, a single-eye image generated by the single-eye image generation unit 23 that combines the pixel values of the pixels to generate a single-eye image as an image for each microlens. The data is supplied to an image processing unit 40 that performs image processing of each individual eye image.

  For example, when the captured single-eye image is displayed as it is, the image processing unit 40 converts the reproduced stereoscopic image into an erect image because its depth, that is, an image with the concavities and convexities reversed. I do. The reason why the image is reversed is that the image observed from the right side of the subject is observed by the observer from the left side. Details of processing for each application executed by the image processing unit 40 will be described later.

  At the time of image display, the processed image data is supplied from the image processing unit 40 to the display driving unit 31 that displays the single-eye image on the pixel performing the display operation based on the single-eye image data. Here, the area where the single-eye image is displayed is an area at a position corresponding to the area of the input / output display 1 where the single-eye image is captured. Here, an area composed of a predetermined number of pixels for displaying one image of the subject performing the display operation is referred to as a “display area”. A display area setting unit 32 that sets a display area in the display drive unit 31 controls driving of the pixels of the input / output display 1, applies a positive voltage to the pixels to be displayed, and displays by the pixels. Form a region. Then, an image is displayed for the predetermined number of lens arrays.

  When the captured image data is transmitted via a communication line via the communication unit 17, a predetermined compression process is performed to reduce the amount of information. In this case, since each individual eye image is very similar, for example, 5 × 5 images acquired at surrounding positions around a single eye image serving as a reference acquired at a certain position. A predetermined number of single-eye images such as. For the single eye image serving as the reference, image data of one entire image compressed by JPEG (Joint Photographic Expert Group) method or the like is transmitted, and for the other 24 single eye images, the single eye image serving as the reference. The image data is compressed by transmitting only the difference data from the image. The captured image data, a plurality of single-eye images, a template for image recognition, and the like can be stored in the storage unit 16 or the storage medium 19 by performing the same processing as the compression processing at the time of transmission, and can be taken out at any time. .

  Next, a configuration example of the imaging display pixels constituting the input / output display 1 and an example of input display processing will be described with reference to FIGS. 4 and 5 show a configuration example of the imaging display pixel 1c constituting the input / output display 1. FIG. The input / output display 1 of this example is provided with a TFT (Thin Film Transistor) 41 as a switching means and an EL (Electro Luminescence) element 44 as a light emitting element in each pixel, and is applied to a self-luminous EL display. As an example.

  First, a configuration example of the imaging display pixel 1c will be described with reference to FIG. The imaging display pixel includes a TFT 41 and an EL element 44. A source line 42 that supplies a source current is connected to the source electrode S of the TFT 41, and a gate line 43 that supplies a gate current is connected to the gate electrode G of the TFT 41. An EL element 44 is connected to the drain electrode D of the TFT 41, and a counter electrode 45 is connected to the EL element 44.

  A processing example at the time of image input of the imaging display pixel 1c configured as described above will be described with reference to FIG. When a voltage in the vicinity of 0 V or in the reverse direction is applied to the gate electrode G of the TFT 41, no current flows through the channel even when a voltage is applied to the source electrode S. In this state, when the input light beam 47a collected from the outside by the microlens irradiates the TFT 41, a leak current (off) from the drain electrode D to the source electrode S is traced due to the photoconductivity of the channel of the TFT 41. Current) 46a is generated. Similarly, a leak current also occurs in the EL element 44. Therefore, by amplifying the leakage current 46a generated by the pixel (TFT 41, EL element 44) applied with a voltage in the vicinity of 0V or in the reverse direction and detecting the presence or absence, light is externally applied to the pixel. It becomes possible to identify whether or not it is irradiated. Further, the amount of light can be identified by the amount of leakage current. Each RGB pixel (a pixel provided with an EL element 44 that emits R, a pixel provided with an EL element 44 that emits G, and a pixel provided with an EL element 44 that emits B) provides RGB. Since the presence / absence and amount of each light are identified, it is possible to obtain a single-eye image of a color subject by combining the outputs of all the pixels constituting the imaging region.

  A processing example at the time of displaying an image of the imaging display pixel 1c will be described with reference to FIG. 5. When a positive voltage (bias voltage) is applied to the gate electrode G of the TFT 41, it is indicated by a solid line arrow 46b in FIG. Thus, according to the voltage applied to the source electrode S, a current flows in the direction from the source electrode S to the drain electrode D in the active semiconductor layer (channel) made of alpha moss silicon or polysilicon. The anode electrode of the EL element 44 is connected to the drain electrode D of the TFT 41, and the anode-cathode electrode generated when the current 46 b flows through the channel of the TFT 41 (the electrode to which the drain electrode D is connected and the counter electrode 45). The EL element 44 emits light according to the potential difference. Then, the reproduction light beam 47b from the EL element 44 is emitted to the outside of the input / output display 1 through the partition layer and the microlens layer on the input / output display 1, and an image is displayed.

  From the above, it can be seen that a display operation and an imaging operation (input operation) can be selectively performed by one pixel (TFT 41) by controlling the applied voltage. 4 and 5, for example, when light is applied from the outside in a state where a voltage of about 0 V is applied, a leak current generated by the EL element 44 is also detected in addition to the TFT 11. Therefore, it can be said that the sensitivity to external light is high.

  Next, measurement examples of the value of the drain current Id of the TFT 41 when light is irradiated and when not irradiated will be described with reference to FIG. FIG. 6 is an explanatory diagram illustrating an example of measured values of current generated in the pixels illustrated in FIGS. 4 and 5. The vertical axis represents the current in the pixel, and the horizontal axis represents the voltage applied to the gate electrode G.

  A line L1 representing the measurement result is a current detected from the pixel when a light is applied with a positive voltage applied (current flowing in the channel of the TFT 41 and current flowing in the EL element 44). A line L2 indicates a value of a current generated in the pixel when light is not irradiated in a state where a positive voltage is applied. When a positive voltage is applied by the lines L1 and L2, there is no difference in the detected current value regardless of the presence or absence of light from the outside.

  On the other hand, the line L3 representing the measurement result represents the value of the current generated by the pixel when light is irradiated from the outside in a state where a reverse voltage is applied, and the line L4 is the reverse direction. This represents the value of the current generated by the pixel when light is not irradiated from the outside in the state in which the voltage is applied. As can be seen by comparing the line L3 and the line L4, when a reverse voltage is applied, there is a difference in the current generated by the pixel when the light is irradiated from the outside and when the light is not irradiated. Has occurred. For example, when a predetermined amount of light is irradiated from the outside in a state where a voltage of about −5 volts (reverse voltage) is applied, a current of about “1E-8 (A)” (the activity of the TFT) Current generated in the semiconductor layer and current generated by the EL element).

  In FIG. 6, a line L4 indicates that a minute current of “1E-10 (A)” is generated even when no light is irradiated from the outside. This is due to noise during measurement. In addition, even if it is EL element which light-emits any color among RGB, the measurement result substantially the same as what is shown in FIG. 6 is obtained. Therefore, even when a voltage in the vicinity of 0 V is applied, it is possible to detect this difference, that is, whether or not light is irradiated, by amplifying the generated current.

  Therefore, it is possible to cause a certain pixel to perform an imaging operation by controlling the gate voltage to be a value in the vicinity of 0 V without intentionally applying a reverse voltage. Then, by controlling the gate voltage to be a value near 0V and performing the imaging operation, compared to the case where the operation is performed by applying the reverse voltage, the amount of the reverse voltage is consumed. Power can be reduced.

  Further, since the number of voltages to be controlled is reduced, the control and the system configuration are facilitated. In other words, the control to control the gate voltage so that the positive voltage is applied since the control so that the gate voltage becomes a value near 0V is also the control so that the positive voltage is not applied. It can be realized with only a line and a power supply circuit (there is no need to provide a separate control line for controlling the gate voltage so that a reverse voltage is applied).

  As a result, it is possible to simplify the configuration of the power supply circuit from the drive board or system board of the input / output display 1 and realize low power consumption as well as the limited space on those boards. Efficient use can also be realized.

  Further, by not applying the reverse voltage, it is possible to avoid the destruction of the TFT 41 and the EL element 44 that may occur when the reverse voltage is applied. For example, the withstand voltage of the TFT 41 can be improved by increasing the channel length (L length), but in this case, the current at the time of on (conduction) decreases, and in order to secure a sufficient current The channel width (W length) needs to be increased. As a result, in order to increase the withstand voltage without changing the current value flowing through the TFT 41, it is necessary to increase the size of one TFT 41, and to each pixel of a high-definition display with a small size of each pixel, the TFT 41 It becomes difficult to arrange.

  Therefore, by eliminating the reverse voltage, the withstand voltage design of the TFT 41 and EL element 44 can be facilitated, and the size of the TFT 41 and EL element 44 itself can be reduced, thereby realizing a high-definition display. It becomes possible to do.

  As described above, when at least the TFT 41 is provided in each pixel constituting the input / output display 1, it is possible to perform imaging using the pixel by applying a voltage in the vicinity of 0V or in the opposite direction. become.

  For example, by causing all of the pixels of the input / output display 1 to perform an imaging operation, an image of the subject can be obtained with the number of microlenses (the number of imaging regions) and the respective positions as viewpoints.

  Next, a display example of one area of the input / output display 1 will be described with reference to FIG. FIG. 7 is a plan view showing an example of one region formed by a plurality of imaging display pixels 1c surrounded by the partition wall 1b of FIG. This one area is composed of, for example, 15 × 15 imaging display pixels 1c (pixels each including three RGB sub-pixels (15 × 3 × 15 sub-pixels)). As described above, since one microlens 1a is provided in front of one area composed of 15 × 15 imaging display pixels 1c (subject side), the 15 × 15 imaging display pixels 1c An image of the subject 50 rotated 180 degrees around the optical axis is formed in the region.

  As described above, in the input / output display 1, not only display but also imaging, that is, detection of light collected by the microlens 1a is performed by each imaging display pixel 1c. In FIG. 7, one single-eye image is captured using 15 × 15 pixels. However, the pixels are not limited to this, and imaging is performed with a larger number of pixels. Also good. Further, as the number of imaging display pixels 1c assigned to one microlens 1a is increased, a higher-resolution subject image can be obtained.

  Next, a state in which the imaging operation and the display operation of the input / output display 1 are executed will be described with reference to FIG. FIG. 8 is an explanatory diagram showing an example of the operation of each pixel.

In the input / output display 1 of this example, the display operation is performed by the entire pixels from the Y _1st row to the Y ₁ -y _1st row, and from the Y _2nd row to the Y ₂ -y _2nd row. The imaging operation is performed by all the pixels. Here, it is assumed that the target rows on which the display operation and the imaging operation are performed are a display operation row 1e and an imaging operation row 1f, respectively. The display operation line 1e and the imaging operation line 1f are sequentially switched from the top to the bottom of the input / output display 1, such as the Yth line, the Y + 1th line, the Y + 2th line,. When the switched target row reaches the lowermost row of the input / output display 1, the switching process is repeated again from the uppermost row.

  That is, in one frame, the imaging operation is performed by the entire pixels included in the imaging operation row 1f, and at the same time, the display operation is performed by the entire pixels included in the display operation row 1e different from the pixels performing the imaging operation. In addition, the operation of each pixel is controlled. Thus, when a display (imaging) period of a predetermined number of frames is seen, imaging and display are performed by all the imaging display pixels 1c constituting the input / output display 1.

  As a result, as many microlens 1a as the number of microlenses 1a can be obtained, and an image of the subject can be obtained with each position as a viewpoint. In addition, a single-eye image is displayed by the number of microlenses 1a, and a three-dimensional image is reproduced.

  Here, for example, the display operation line 1e and the imaging operation line 1f are switched from the upper end to the lower end of the input / output display 1 at a cycle of, for example, 30 Hz and 60 Hz. The imaging display pixels in the imaging operation row 1f do not display an image, but since the scanning cycle is short, the presence of pixels that perform imaging operations is not recognized by the user. In this manner, the input / output display 1 captures a subject and reproduces a stereoscopic image based on the captured image.

  Next, imaging and display processing of the information processing apparatus 10 will be described with reference to the flowcharts of FIGS. 9 and 10. Note that the imaging process of FIG. 9 and the display process of FIG. 10 are performed in parallel. First, imaging processing of the information processing apparatus 10 will be described with reference to a flowchart of FIG.

  The imaging region setting unit 22 of the light receiving drive unit 21 applies a voltage, for example, in the vicinity of 0 V to the imaging display pixel 1c of the input / output display 1 that performs the imaging operation, and sets the imaging region (step S1). Then, the light receiving drive unit 21 images the subject (step S2). That is, the output from the imaging display pixel 1c performing the imaging operation among the pixels of the input / output display 1 is detected, and whether or not the light is irradiated from the outside and the amount of the light is detected. To do. The pixel value that is the detection result is output to the single image generator 23.

  The single-eye image generation unit 23 generates a single-eye image based on the detection result supplied from the light receiving drive unit 21 (step S3). The generated single-eye image data is output to the image processing unit 40. The image processing unit 40 performs predetermined image processing on each single-eye image based on the image data supplied from the single-eye image generation unit 23 (step S4).

  By repeating the above processing and sequentially switching the pixels constituting the imaging region, a single eye image of a subject with a plurality of points on the input / output display 1 as viewpoints can be obtained.

  Next, display processing of the information processing apparatus 10 that reproduces a stereoscopic image will be described with reference to the flowchart of FIG.

  First, the display area setting unit 32 applies a voltage in the positive direction to the imaging display pixel 1c of the input / output display 1 that performs the display operation, and sets the display area (step S11). The display area is set at a position different from the imaging area set in FIG. Based on the image data stored in the RAM 12 as a buffer, the display driving unit 31 displays a single eye image on the pixels constituting the display area (step S12). When extracting the image data, it is possible to reproduce an image acquired from the communication line via the storage unit 16, the removable storage medium 19, or the communication unit 17 in addition to the RAM 12.

  The above processing is repeated, and the imaging display pixels 1c constituting the display area of the input / output display 1 are sequentially switched to reproduce the captured stereoscopic image of the subject.

  Next, examples of three types of applications that can be operated on the input / output display 1 of this example will be described with reference to FIGS. In this example, an application for a three-dimensional pointer, three-dimensional drawing, and stereoscopic CG (Computer Graphics) graphic operation will be described. In the three-dimensional input display device 100 of this example, necessary processing differs depending on the target application.

  First, a three-dimensional pointer application will be described with reference to FIG. As shown in FIG. 11, when using the three-dimensional pointer, the input / output display 1 captures a three-dimensional image drawn in the space on the input / output display 1 using the input pen 2a as a pen-shaped input unit that emits light from the tip. The captured image is captured by the input image acquisition unit 40a. In FIG. 11, since the processing blocks between the input / output display 1 and the image processing unit 40 related to imaging and display have already been described with reference to FIG. 3, description of the processing blocks is omitted.

  And the process which calculates | requires the three-dimensional coordinate of a light source from the imaged input image is performed. By the way, when obtaining the three-dimensional coordinates from the captured three-dimensional image, it is necessary to perform correction in advance because the optical axis shift of the micro lens, the optical axis tilt, the lens distortion, and the like impede accurate coordinate calculation. Therefore, the input image acquired by the input image acquisition unit 40a as an initial setting is stored as a correction value in the lens distortion table of the RAM 12. Then, the coordinate calculation is performed using the correction value of the lens distortion table in the three-dimensional coordinate calculation unit 40c that calculates the three-dimensional coordinates. The lens distortion table can be stored in the storage unit 16 so that the correction value can be read out as needed. Once the lens distortion table is created, the three-dimensional coordinates of the light source can be determined by the principle of triangulation using images obtained from a plurality of single-eye images.

  Then, predetermined image processing is performed by the output image processing unit 40d that performs contrast enhancement processing of the input image and image overlay processing in the time direction. After performing such image processing, an image is supplied to the input / output display 1 from the image output unit 40e that generates a three-dimensional pointer and a three-dimensional drawing image, and the image is reproduced on the input / output display 1. With respect to the output image, basically, there is no need to consider the optical axis shift (variation) of the lens, the optical axis tilt, the lens distortion, and the like. This is because, if the optical path at the time of input and the optical path at the time of display are the same, a virtual light source (real image) by display is generated at the same location as the light source used for input.

  The three-dimensional pointer shown in FIG. 1 is an application that displays input information (imaging information). The number of input pens 2a that emit light is not necessarily one, and a plurality of pens may be used. The input image may be displayed as it is as the output image, but the input coordinates can be easily confirmed by increasing the contrast or changing the color of the input light.

  Since a normal IP uses a film or the like, the positional relationship between the lens and the film cannot be fixed for development processing, but the positional relationship between the lens array of this example and the input / output display panel 1 is always fixed. For this reason, once an optical axis deviation or the like is measured, it can be used as an eigenvalue of the three-dimensional input display device 100, and there is no need to perform remeasurement. In addition, the imaging function of the input / output display 1 can be used for measurement, and the measurement result can also be used when generating a display image.

  Next, a three-dimensional drawing application will be described with reference to FIGS. In addition to the 3D pointer technique, the 3D rendering application of this example displays input information superimposed in the time direction, and at the same time draws on a 2D display with icons displayed in the 3D space. Is possible. In this example, a color palette icon 2b capable of selecting a plurality of colors and an eraser icon 2c capable of erasing the pen tip locus 3 are displayed on the input / output display 1 as a GUI (Graphical User Interface) in FIG. .

  As for the technique of the three-dimensional pointer, as described with reference to FIG. 11, a three-dimensional image drawn in the space on the input / output display 1 by the input pen 2a as a pen-like input unit that emits divergent light from the tip is input / output display. The image is supplied to the input image acquisition unit 40a that captures the captured image and captures the captured image. In a three-dimensional drawing application, it is easiest to input with the input pen 2a that emits divergent light from the tip. However, as long as the tip can be easily recognized, it does not necessarily emit light. Since the three-dimensional pointer imaging display technique has already been described, the details thereof will be omitted.

  In the icon GUI display process, the captured image is captured by the input image acquisition unit 40a, and the captured image is supplied to the lens distortion table calculation unit 40b, as in the three-dimensional pointer imaging process, and the correction value is stored in the lens distortion table. Store. Then, the three-dimensional coordinate calculation unit 40c performs coordinate calculation using the correction value of the lens distortion table.

  In order to display a predetermined icon on the input / output display 1 after the calculation of the three-dimensional coordinates, the calculation data is supplied to the GUI processing unit 40j that performs the GUI processing. Here, processing such as icon selection and icon movement when the user operates an icon is referred to as a command, and a change amount such as a movement amount is referred to as an operation amount. From the obtained three-dimensional coordinates, the GUI processing unit 40j performs GUI processing for displaying an icon or the like three-dimensionally, and supplies processing data to a command / operation amount determination unit 40l that determines an icon command / operation amount. .

  The command / operation amount determination unit 40l determines various parameters such as commands and operation amounts before and after a predetermined time, and then supplies data to the three-dimensional CG generation unit 40m that performs numerical processing of the three-dimensional CG. Generate numeric data. The generated numerical data is supplied to the single-eye image CG generation unit for each lens that generates CG data corresponding to the single-eye image of each lens, and the CG is obtained for the single-eye image for each microlens that performs image reproduction. Data is generated, and single-eye image CG data is supplied to the lens distortion table calculation unit 40b. The lens distortion table calculation unit 40b reads the correction value from the lens distortion table in the RAM 12 and corrects the single-eye image CG data. Then, the corrected single-eye image CG data is supplied to the CG image output unit 40o that controls the output of the CG image, and a stereoscopic image such as an icon is displayed on the input / output display 1 as the CG image.

  In this way, icons such as eraser, all erasure, and color change necessary for drawing are arranged on the three-dimensional display space with GUI display. In order to make it easier to use, a plurality of different types of pens are prepared, and the drawing color and eraser functions are assigned, so that it can be used as a more natural tool. This is possible because imaging information is used for input, and information other than coordinates cannot be input by a three-dimensional digitizer. In addition, processing for identifying different types of pens includes processing for changing the color of light emission, processing for changing the shape of the pen tip that requires processing of imaging information, and the like.

  Here, by using a natural image for the IP input image, for example, the shape of the hand can be identified, and the coordinates of the fingertip can be detected and calculated. Hereinafter, IP-specific processing for image recognition will be described with reference to FIGS.

  First, an example of two-dimensional image generation processing from multiple viewpoints will be described with reference to FIG. FIG. 13A shows an example in which a block-shaped subject 51 is photographed. FIG. 13B is an example of an IP image using a plurality of single-eye images. In this example, 20 × 15 single-eye images are captured. FIG. 13C is an example of a multi-view two-dimensional image. In this example, the two-dimensional images of the front and the other eight viewpoints are synthesized from single-eye image information of an actually simple object.

  In the three-dimensional input display device 100 of this example, when the subject is relatively close to the lens array, the entire view of the subject often does not fit in each single-eye image, depending on the angle of view of the lens. . Therefore, a two-dimensional image of the front viewpoint is created from the information of the center pixel of each individual eye image. Furthermore, a two-dimensional image of a representative viewpoint is created using information on pixels that are a fixed distance away from the center of each individual image. In this way, a multi-view two-dimensional image can be obtained.

  Next, an example of processing for performing natural image recognition and feature point coordinate detection will be described with reference to FIG. Here, the “natural image” represents an object having an entity such as a human hand. The “feature point” represents a characteristic part such as a fingertip when the object is pointed by hand.

  First, imaging processing is started, and an IP image in which a subject is imaged is acquired (step S20). The IP image 50a of this example uses a human right hand with an index finger raised as a subject, and a vertically inverted image of the subject is captured. Next, a multi-viewpoint two-dimensional image 50c is generated from the obtained IP image 50a (step S21). Then, feature quantities are extracted from the multi-viewpoint two-dimensional image 50c (step S22). In this example, the shape of the right hand is extracted as the feature amount.

  Matching the multi-viewpoint two-dimensional image 50c with the template image 50d or feature amount of the operation (hand, tool shape) to be accepted (step S23), and performing graphic / gesture recognition processing (step S24) Identify the type and content. For this matching, a technique performed on a two-dimensional image may be used.

  Next, the required three-dimensional coordinates are calculated by the recognized operation, and target feature points are selected from the two-dimensional coordinates of each viewpoint (step S25). In this example, the feature point 50′e is selected as the fingertip from the feature point image 50e representing the fingertip whose three-dimensional coordinates are calculated. Then, a plurality of single-eye images reflecting the feature points are selected based on the generated two-dimensional image (step S26). In this example, it is assumed that the selected single-eye image 50b is selected from the IP image 50a.

  Next, further feature points are detected in each individual eye image (step S27). In this example, the fingertip is detected as a feature point 50′f from a plurality of single-eye images from the single-eye image feature point image 50f. Then, target three-dimensional coordinates are calculated from the coordinates of the feature points of each individual eye image (step S28). From the calculation result of the three-dimensional coordinates, the command (operation) and the operation amount of the real object (fingertip) on the input / output display 1 can be determined as shown in the operation amount image 50g (step S29).

  In these processes, techniques used in existing two-dimensional image processing are used for image matching (feature amount matching) and feature point detection. It is also possible to first detect all feature points of each individual image and reconstruct the three-dimensional information of the subject, and then perform matching.

  Next, a 3D CG graphic operation application will be described with reference to FIGS. 11 and 15 to 17. First, a configuration example of an application processing block will be described with reference to FIG. The application of the stereoscopic CG figure operation of this example reproduces stereoscopic display information that changes according to the input coordinates. For example, it is possible to display an object such as a sphere three-dimensionally and perform operations such as moving, rotating, transforming, dividing, and combining the object by hand. In this application, graphic information and command information are interpreted from an input image (an operator's hand image), and an output image corresponding to the information is generated by computer graphics.

  First, an example of an operation process for a three-dimensional stereoscopic image 2d having an appearance of a four-wheeled vehicle virtually displayed on the input / output display 1 will be described with reference to FIG. In the example, the user performs an operation such as movement with the left hand 2e and the right hand 2f which are real objects.

  From FIG. 11, captured images of the user's left hand 2e and right hand 2f captured on the input / output display 1 are acquired by the input image acquisition unit 40a. The lens distortion table calculation unit 40b calculates a lens distortion correction value, and the multi-view two-dimensional image generation unit 40f generates a multi-view two-dimensional image of the user's left hand 2e and right hand 2f. At the same time, the three-dimensional coordinates calculated by the three-dimensional coordinate calculation unit 40c and the multi-viewpoint two-dimensional image are supplied to the natural image recognition unit 40g that recognizes that the captured image of this example is a natural image of the left hand and the right hand. The natural image recognition unit 40g performs a matching process in the image matching unit 40h that performs matching between the captured image and the template image, and performs a process of extracting a feature point of the matching image by the feature point extraction unit 40i.

  In response to the result, the gesture interpretation unit 40k that interprets the user's operation interprets the gesture, and then the command / operation amount determination unit 40l determines the user's command and operation amount. Subsequent three-dimensional CG creation processing is the same as the three-dimensional drawing processing already described, and thus the details are omitted. Also, the operations on the stereoscopic images in FIGS. 16 and 17 are the same as the processes described in FIG.

  Next, an example of operation processing for the three-dimensional stereoscopic image 2g having a dumbbell-like appearance virtually displayed on the input / output display 1 will be described with reference to FIG. The user shows an operation example in which the left hand 2e and the right hand 2f, which are entities, are deformed. Note that the operation on the stereoscopic image in FIG. 16 is the same as the processing described in the stereoscopic CG graphic operation, and thus the description thereof is omitted.

  The type of operation when operating by hand is prepared in advance as a recognizable command. For example, the action of pushing an object with a palm is “move”. The action of rubbing an object with the palm of your hand is "rotation". “Deformation” means that the fingertip is pressed perpendicularly to the object, and the operation of pinching similarly is also “deformation (pull out)”. The action of pinching and pulling an object with both hands is “dividing”, and the action of pushing two objects with two palms together is “joining”.

  Furthermore, an example of an operation process for a three-dimensional stereoscopic image 2h having a cylindrical appearance virtually displayed on the input / output display 1 will be described with reference to FIG. In the example, the user performs an operation such as making a hole in the stereoscopic image 2h with the drill 2i as an entity or cutting the stereoscopic image 2h with the cutter 2j. Note that the operation on the stereoscopic image in FIG. 17 is the same as the processing described in the stereoscopic CG graphic operation, and thus the description thereof is omitted.

  In this case, not only information about the hand but also some tools with shapes that are easy to recognize images are registered. By recognizing these tools from the imaging information, the types of operations can be increased. In this example, using a cutter-shaped tool (the cutter itself may be used, but the image recognition is more reliable if a color is added or the shape is simplified so that it can be easily recognized from the imaging information). , “Linear division” is performed. Also, “chamfering” is performed with a ruler-shaped tool, and deformation operations such as “drilling” are performed with a drill-shaped tool.

  In this way, it is possible to perform an input display operation in a three-dimensional space.

  According to the present embodiment, the input coordinates (figure) need only be reflected on the display as it is, so only the first IP is used. The quality of such an input display technology based on IP of only one stage has been improved by the advancement of computer graphic technology. In addition, there is no problem of turning over the reproduced image from the beginning, and there is an effect that the apparatus can be easily configured because the problem of lens distortion and the like need only be one stage.

  In addition, a three-dimensional input display device can be configured with a simple system from a small scale to a large scale. Not only simple shape input, but also editing work such as object deformation can be realized. In addition, if the accuracy of gesture recognition is improved, more natural object operation can be performed, and there is an effect that editing and operation of a three-dimensional object are facilitated.

  In addition, as a point superior to the conventionally used two-dimensional input tablet, information input is obtained as three-dimensional image information, so that it is possible to realize input of a plurality of points and operation by hand shape and area. is there. Therefore, there is an effect that functions as various input / output interfaces can be provided.

  In addition, it is possible to input and display in a three-dimensional space with a single device. For this reason, there exists an effect which becomes very advantageous in terms of material cost and manufacturing cost. In addition, there is an effect that a stereoscopic display with respect to coordinate input can be performed in real time and fed back to the operating side. In the present embodiment, special glasses for observation are not required even during stereoscopic display, and not only binocular parallax but also vertical and horizontal motion parallax is expressed.

  Further, since it can be very small as a three-dimensional coordinate input device, it can be used as a simple three-dimensional scanner. In addition, the two three-dimensional input display devices can be applied to telephone communication or the like by two-way communication with each other, and a stereoscopic image of each other's speakers can be displayed even in a remote place.

  In addition, since the positional relationship between the lens array and the input / output display 1 is always fixed, lens distortion information and the like can be held as eigenvalues of the three-dimensional input display device itself. In this case, once distortion information or the like is measured, distortion information can always be used in the subsequent processing, and the same value can be used for correction processing at the time of display. Therefore, the image display system using the two-stage IP is used. In comparison, there is an effect that it is not necessary to perform strict image adjustment for imaging and display.

  In addition, since the input and display processes both use optical properties and use the same optical path, a predetermined function can be realized even if the lens performance is not strictly required. For this reason, it is possible to manufacture a microlens for imaging and displaying at low cost, and there is an effect of reducing the cost.

  In the above-described embodiment, three types of applications that can be used are illustrated. However, it is possible to output an image that has been input once to create a new display image and show it to the user. Can be used. For example, the optical pen input is displayed as it is, the finger is recognized, the natural image is input, and the like.

  In the above-described embodiment, the TFT 41 is provided in each pixel in the imaging display element. However, a leak current is generated in response to receiving light from the outside, and the pixel is turned on / off. Any transistor may be provided as long as the operation can be switched.

  The series of application processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a network or a recording medium into a general-purpose personal computer or the like.

  Applications are also possible in the entertainment field such as the art field and games. In addition to 3D CG drawing, by defining changes in drawing information by artist gestures, 3D images can be easily obtained, and the range of creativity is widened. Moreover, you may make it use also for the display of the exhibit in a museum etc.

  It is also possible to arrange a large amount of information in a computer or a network in a three-dimensional space, and take out or manipulate information from the three-dimensional space. By doing this, there is an effect that a larger amount of information can be handled efficiently.

It is the block diagram which showed the example of an external structure of the three-dimensional input display apparatus in one embodiment of this invention. It is explanatory drawing which showed the example of the input-output display in one embodiment of this invention. It is the block diagram which showed the internal structural example of the three-dimensional input display apparatus in one embodiment of this invention. It is explanatory drawing which showed the structural example of the imaging display pixel in one embodiment of this invention. It is explanatory drawing which showed the structural example of the imaging display pixel in one embodiment of this invention. It is explanatory drawing which showed the example of a relationship between the gate voltage and drain current in one embodiment of this invention. It is explanatory drawing which showed the example of a display of the one area | region of the input-output display in one embodiment of this invention. It is explanatory drawing which showed the example of the imaging display process of the input-output display in one embodiment of this invention. It is the flowchart which showed the example of the imaging process in one embodiment of this invention. It is the flowchart which showed the example of the display process in one embodiment of this invention. It is the block diagram which showed the example of the imaging display process of the image process part in one embodiment of this invention. It is explanatory drawing which showed the example of the three-dimensional drawing in one embodiment of this invention. It is explanatory drawing which showed the example of operation of the three-dimensional solid image in one embodiment of this invention. It is explanatory drawing which showed the example of operation (pulling) of the three-dimensional solid image in one embodiment of this invention. It is explanatory drawing which showed the example (operation of a real tool) of the three-dimensional solid image in one embodiment of this invention. It is explanatory drawing which showed the example of a production | generation of the multiview two-dimensional image in one embodiment of this invention. It is the flowchart which showed the example of the natural image recognition in one embodiment of this invention, and a feature point coordinate detection process. It is the block diagram which showed the example of the image display system using the conventional one step type IP. It is the block diagram which showed the example of the image display system using the conventional 2 step type IP.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Input / output display, 1a ... Micro lens, 1b ... Partition, 1c ... Imaging display pixel, 1d ... Display surface, 1e ... Display operation line, 1f ... Imaging operation line, 2a ... Input pen, 2b ... Color palette, 2c ... Eraser, 2d, 2g, 2h ... 3D solid image, 2e ... left hand, 2f ... right hand, 2i ... drill, 2j ... knife, 3 ... pen locus, 4 ... input beam, 5 ... reproduction beam, 10 ... information processing Device: 11 ... ROM, 12 ... RAM, 13 ... CPU, 14 ... input unit, 15 ... output unit, 16 ... storage unit, 17 ... communication unit, 18 ... drive, 19 ... storage medium, 21 ... light receiving drive unit, 22 ... Imaging area setting section, 23 ... Single eye image generation section, 25 ... Bus, 31 ... Display drive section, 32 ... Display area setting section, 40 ... Image processing section, 41 ... TFT, 42 ... Source line, 43 ... Gate line 44 EL elements 45 Counter electrode, 46a, 46b ... current, 47a ... input beam, 47b ... reproduction beam, 50, 51 ... subject, 60, 60a, 60b ... imaging plate, 61, 61a, 61b ... fly-eye lens, 62 ... real object, 63 ... Each single-eye image, 64, 64a, 64b ... reproduction illumination light, 65 ... observer, 66,66a ... input ray, 67 ... reproduction ray, 67b ... first IP reproduction ray, 68 ... second IP reproduction ray, 100 ... 3 Dimensional input display device

Claims (4)

A display having a lens array layer in which a plurality of microlenses are arranged in a plane, and a pixel layer in which a plurality of pixels are arranged in a plane for each microlens to capture and display light rays;
A light receiving processing unit that obtains a light receiving image by a signal captured by the pixel layer of the display;
A single-eye image generating unit that generates an image obtained by the light-receiving processing unit as a single-eye image for each microlens;
Generating a multi-viewpoint two-dimensional image from a single-eye image for each microlens, extracting a characteristic shape from the multi-viewpoint two-dimensional image, collating the characteristic shape with an input image recognition template, A feature point is selected, a plurality of single-eye images are selected from the single-eye images, a three-dimensional coordinate of the feature points is calculated from the selected single-eye images, and a three-dimensional image near the surface of the display is calculated. An input image processing unit that performs a process of calculating the amount of movement ,
A display driver that performs display driving of pixels in the pixel layer of the display ;
A storage unit for storing a template for the input image recognition, input display device Ru comprising a. The input display device according to claim 1 ,
The storage unit stores a lens distortion table for correcting distortion for each microlens,
Wherein the predetermined processing of the input image processing section, value correcting using the lens distortion table when performing the calculation of the three-dimensional coordinates for the input image input display device. The input display device according to claim 1 or 2 ,
Wherein the predetermined processing of the input image processing section accumulates the image information input by the input unit in time series, the time-series at the input display Ru accumulated image information is displayed on the display. Input display method for performing input and display using a display having a lens array layer in which a plurality of microlenses are arranged in a plane, and a pixel layer in which a plurality of pixels are arranged in a plane for each microlens to capture and display light rays In
Obtain a received light image by the signal captured in the pixel layer of the display,
The light reception image is generated as a single eye image for each microlens,
Generating a multi-viewpoint two-dimensional image from a single-eye image for each microlens, extracting a characteristic shape from the multi-viewpoint two-dimensional image, collating the characteristic shape with an input image recognition template, A feature point is selected, a plurality of single-eye images are selected from the single-eye images, a three-dimensional coordinate of the feature points is calculated from the selected single-eye images, and a three-dimensional image near the surface of the display is calculated. and facilities processing for calculating a movement amount,
There row display driving by the pixel of the pixel layer of the display,
An input display method for storing the input image recognition template in a storage unit .