JP2009122919A - Display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make a contact decision by simple image processing in a display device with an optical input function. <P>SOLUTION: Edge enhancement is calculated from the edge image of a photographed image, and makes a contact determination on the basis of the edge enhancement. Thus, it is possible to make the contact determination by simple image processing. Also, a threshold as the reference of the contact decision is set to 2/3 of edge enhancement in a frame in which the increase of the edge strength is not detected. Thus, it is possible to stably make the contact determination regardless of the way of falling of the shadow of an object or variation of ambient light. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a display device having a light input function for inputting information from a display screen by light.

  Liquid crystal display devices are widely used as display devices in portable devices such as mobile phones and notebook computers.

In recent years, an optical sensor has been built in a pixel so that an object close to the display screen can be photographed. As an application of a display device equipped with such a light input function, an optical system that detects the position and contact of a finger by reading the shadow of the finger caused by ambient light and the reflected light on the belly of the finger caused by backlight light A touch panel has been proposed (see Patent Document 1).
JP 2006-244446 A

  However, in order to detect the position and contact state of a finger close to the display screen, it is necessary to perform various image processing on the captured image, and there is a problem that the scale of the circuit for image processing increases. It was.

  The present invention has been made in view of the above, and an object of the present invention is to perform contact determination using simple image processing.

  A display device according to the present invention includes a display unit having a display function for displaying an image on a display screen and a light input function for capturing an image of an object close to the display screen, and an edge by detecting an edge of the captured image. An edge detection unit for an image, an edge strength detection unit for detecting the sum of the frequencies of a predetermined gradation of the gradation histogram in the edge image as an edge strength, and an object displayed when the edge strength is a predetermined threshold value or more A contact determination unit that determines that the screen is touched.

  In the present invention, when the object is close to the display screen, the contour of the object in the captured image can be detected as an edge. Therefore, an edge image of the captured image is obtained and the edge strength is calculated. When the edge strength is greater than or equal to a predetermined threshold, it is possible to perform contact determination using a simple image processing algorithm by determining that the object has touched the display screen. As a result, the circuit scale can be reduced, and the price and power consumption can be reduced.

  In the display device, the contact determination unit sets a predetermined ratio of the edge strength in a frame in which the edge strength stops increasing as a predetermined threshold value.

  In the present invention, when an object touches the screen, the edge strength does not increase. Therefore, by making a contact determination based on the edge strength in the frame where the edge strength does not increase, how to cast the shadow of the object. And contact determination in consideration of the influence of ambient light.

  In the display device, the predetermined ratio is preferably 2/3 as a reference for contact determination.

  In the display device, the predetermined ratio is changed according to illuminance.

  In the present invention, it is possible to set an appropriate threshold according to the illuminance.

  In the display device, the predetermined ratio is changed based on a gradation median value or a gradation average value of a captured image.

  In the present invention, the gradation median value and the gradation average value of an image taken in accordance with the illuminance change, so that the threshold value is appropriately set using the gradation median value and the gradation average value. Is possible.

  According to the present invention, contact determination can be performed using simple image processing.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a plan view showing a configuration of a display device in the present embodiment. As shown in the figure, the display device includes a glass substrate 1 and a semiconductor substrate 2. The glass substrate 1 and the semiconductor substrate 2 are connected by a flexible printed board 8. Here, the display device includes, for example, a counter substrate (not shown) disposed to face the glass substrate 1 and a liquid crystal layer (not shown) sandwiched between the counter substrate and the glass substrate 1. A liquid crystal display device is assumed.

  On the glass substrate 1, a pixel portion 3, a signal line driving circuit 4, a scanning line driving circuit 5, and an output circuit 6 are arranged. Each circuit on the glass substrate 1 is formed by, for example, a polysilicon TFT (Thin Film Transistor).

  The pixel unit 3 has a display function for displaying an image and a light input function for capturing an image of an object close to the pixel unit 3. In the pixel portion 3, a plurality of signal lines 11 and a plurality of scanning lines 12 are wired so as to intersect each other, and a pixel 31 is formed at each intersection. An analog signal is applied to the signal line 11 and written in the pixel 31 to be used for display. In the pixel 31, the optical sensor unit 10 is formed and receives light incident on the pixel unit 3. The configuration of the pixel 31 will be described later.

  The signal line drive circuit 4 includes a D / A conversion circuit that converts digital image data into an analog signal suitable for driving the pixel 31, and applies the analog signal to the signal line 11.

  The scanning line driving circuit 5 drives the scanning line 12 and controls the timing of writing the analog signal applied to the signal line 11 to the pixel 31.

  The output circuit 6 includes an A / D conversion circuit that converts an electrical signal corresponding to the amount of received light detected by the optical sensor unit 10 into digital data, and a P / S conversion circuit that converts parallel data into serial data. Data is output to a circuit formed on the semiconductor substrate 2. The glass substrate 1 and the semiconductor substrate 2 transmit and receive various signals via the flexible printed circuit board 8.

  A logic IC that performs display control and image capture control is mounted on the semiconductor substrate 2. The logic IC includes a circuit that performs contact determination using captured images, such as an edge detection circuit 71, an edge strength detection circuit 72, and a contact determination circuit 73. The operation of each circuit will be described later.

  FIG. 2 is a schematic diagram illustrating a circuit configuration of the pixel unit 3. A plurality of signal lines 11 and scanning lines 12 are wired in a matrix in the pixel portion 3. A pixel TFT 15 is disposed at the intersection of the signal line 11 and the scanning line 12. A liquid crystal capacitor C1 and an auxiliary capacitor C2 are connected to the pixel TFT 15 to form one pixel 31. The other terminals of the liquid crystal capacitor C1 and the auxiliary capacitor C2 are connected to the auxiliary capacitor line 13.

  The optical sensor unit 10 is formed for each pixel 31. The optical sensor unit 10 is formed integrally with the display element such as the pixel TFT 15 on the same substrate. For example, when the pixel 31 is composed of red, green, and blue, the optical sensor unit 10 may be disposed at a ratio of one to the three pixels. The optical sensor unit 10 outputs an electrical signal having a magnitude corresponding to the detected amount of received light via the data line 14. Although not shown, the optical sensor unit 10 includes a precharge switch, an output switching unit, an imaging unit, a charge storage unit, and an imaging result storage unit.

  Here, the operation of the optical sensor unit 10 will be described. First, charges are accumulated in a charge accumulation unit such as a sensor capacitor by controlling a precharge switch. During a predetermined exposure time, a leak current flows in accordance with incident light received by an imaging unit such as a PIN photodiode, and the potential of the charge storage unit changes. A signal corresponding to this potential is temporarily stored in the imaging result storage unit. The output switching unit is controlled to output the signal stored in the imaging result storage unit via the data line 14. Since the optical sensor unit 10 outputs an electrical signal having a magnitude corresponding to the amount of received light in this way, the output circuit 6 inputs this electrical signal, converts it into a digital signal, and outputs it to the semiconductor substrate 2 as photographed image data. Output to the formed circuit.

  FIG. 3 shows an image obtained by photographing a finger close to the pixel unit 3. In the image shown in the figure, it can be seen that the finger is darkened at a location close to the finger. This is a shadow that was created when the ambient light was blocked by the finger.

  Next, the contact determination process based on the photographed image will be described.

  First, the edge detection circuit 71 performs edge processing on the captured image. Edge processing is processing that emphasizes lines and edges in an image and is called a difference filter.

The image shown in FIG. 4 is obtained by applying an 8-neighbor Laplacian filter of the difference filters to FIG. The 8-neighbor Laplacian filter calculates the following equation for the pixel of interest g (i, j).

  For edge processing, other difference filters such as a 4-neighbor Laplacian filter and a Forsen filter may be used.

  Subsequently, the edge strength detection circuit 72 calculates a gradation histogram of the edge image subjected to the edge processing. FIG. 5 is a gradation histogram when an object is not close to the pixel unit 3. When the object is not close to the pixel unit 3, there is no outline in the photographed image, and as shown in FIG. 5, a gradation histogram in which gradations are concentrated in the central part is obtained. FIG. 6 is a gradation histogram when an object approaches the pixel unit 3. According to FIG. 6, it can be confirmed that the contour of the object appears as a light gradation and a dark gradation.

  The edge intensity detection circuit 72 calculates the sum of the frequencies in a predetermined gradation in the gradation histogram as the edge intensity. In the example shown in FIG. 6, the sum of the frequencies in the gradations of m or less and n or more is calculated as the edge strength. The threshold values m and n are values determined in advance.

  Subsequently, the contact determination circuit 73 determines whether or not the object has contacted the pixel unit 3 based on the calculated edge strength. FIG. 7 is a graph showing the relationship between the distance between the object and the pixel unit 3 and the edge strength. As shown in the figure, the edge strength tends to increase as the object approaches. When the edge strength equal to or greater than a predetermined threshold is obtained, it is determined that the object has touched the pixel unit 3.

  However, the edge strength varies about 10 times to 100 times depending on how the shadow of the object is applied to the pixel unit 3, and thus it may be inconvenient if the threshold value is determined in advance. FIG. 8 is another graph showing the relationship between the distance between the object and the pixel unit 3 and the edge strength. In an environment where the edge strength is calculated to be small, the edge strength does not reach the threshold when the object touches the pixel unit 3 as shown in FIG.

  Therefore, in this display device, the edge strengths of the three most recent frames are monitored, and two-thirds of the edge strengths in the frames in which the increase in edge strength has ceased are adopted as threshold values. FIG. 9 shows an example of edge strength observed for each frame. In each frame indicated by reference numeral 91, the edge strength is increased. On the other hand, in the frame indicated by reference numeral 92, since the edge strength is smaller than that of the previous frame, two-thirds of the edge strength is adopted as the threshold value. A broken line 93 in FIG.

  This is because the edge strength increases as the object approaches the pixel unit 3, but the edge strength does not increase when the object contacts the pixel unit 3. Thus, by determining the threshold value that is a reference for contact determination based on the change in edge strength, contact determination can be performed regardless of the shadowing of the object and variations in ambient light.

  It should be noted that the ratio of the edge strength in the frame in which the edge strength stops increasing is not limited to two-thirds. For example, the object may be larger than two-thirds of the edge strength. What is necessary is just to calculate by the ratio which can determine with having contacted. The set threshold value is stored in a threshold value register included in the contact determination circuit 73. The threshold value once set is reset when the object moves away from the pixel unit 3, that is, when the edge strength returns to the value before the proximity of the object.

  In this way, when the edge strength is greater than or equal to the threshold value, it is possible to determine not only the moment when the object is in contact but also the state where the object is in contact. It becomes possible to do.

  In addition, as described above, instead of setting the threshold at a predetermined ratio such as two-thirds of the edge strength, for example, the ratio is high when the illuminance of the ambient light is bright, and the ratio is low when it is dark. You may set a threshold value by changing a ratio according to illumination intensity.

  Furthermore, since the gradation median value of the photographed image varies depending on the illuminance, the ratio may be changed according to the gradation median value. Of course, other gradation tendency values such as gradation average values may be used.

  Therefore, according to the present embodiment, the contact determination can be performed with simple image processing by calculating the edge strength from the edge image of the captured image and performing the contact determination based on the edge strength. The circuit scale can be reduced, and the price and power consumption can be reduced.

  According to the present embodiment, the threshold value used as the reference for contact determination is set to two-thirds of the edge intensity in the frame in which the increase in edge intensity ceases, thereby reducing the shadowing of the object and the variation in ambient light. Regardless, the contact determination can be performed stably.

It is a top view which shows the structure of the display apparatus in one embodiment. It is a schematic diagram which shows the circuit structure of the pixel part of the said display apparatus. It is the image which image | photographed the finger | toe close to the pixel part of the said display apparatus. 4 is an edge image obtained by performing edge processing on the image of FIG. 3. It is a gradation histogram of the edge image obtained when an object is not close to the pixel portion of the display device. It is the gradation histogram of the edge image obtained when an object approaches the pixel part of the display device. It is a graph which shows the relationship between the distance of a display apparatus and an object, and edge strength. It is another graph which shows the relationship between the distance of a display apparatus and an object, and edge strength. It is a graph which shows the threshold value calculated | required based on the edge intensity | strength observed for every flame | frame, and the edge intensity | strength.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Glass substrate 2 ... Semiconductor substrate 3 ... Pixel part 31 ... Pixel 4 ... Signal line drive circuit 5 ... Scanning line drive circuit 6 ... Output circuit 8 ... Flexible printed circuit board 10 ... Optical sensor part 11 ... Signal line 12 ... Scanning line 13 ... Auxiliary capacitance line 14 ... Data line 15 ... Pixel TFT
71 ... Edge detection circuit 72 ... Edge strength detection circuit 73 ... Contact determination circuit

Claims (5)

A display unit having a display function for displaying an image on a display screen and a light input function for capturing an image of an object close to the display screen;
An edge detection unit that detects an edge of a captured image and sets it as an edge image;
An edge strength detection unit that detects the sum of the frequencies of predetermined gradations of the gradation histogram in the edge image as edge strength;
A contact determination unit that determines that the object has touched the display screen when the edge strength is equal to or greater than a predetermined threshold;
A display device comprising:   The display device according to claim 1, wherein the contact determination unit sets a predetermined ratio of edge strength in a frame in which the edge strength stops increasing as the predetermined threshold value.   The display device according to claim 2, wherein the predetermined ratio is two-thirds.   The display device according to claim 2, wherein the predetermined ratio is changed according to illuminance.   The display device according to claim 2, wherein the predetermined ratio is changed based on a gradation median value or a gradation average value of the photographed image.