KR100875138B1 - Input device using imaging sensor and method - Google Patents

Abstract

An input device using the imaging sensor of the present invention comprises a sensor unit including an imaging sensor for sensing a subject which is an input medium and outputting a captured image signal; And an operation unit which receives an image signal output from the sensor unit, converts a signal for each pixel of the image signal into a 1-bit signal, and then compares the before and after image data to calculate a motion vector.

An input method using an imaging sensor of the present invention includes: a first step of capturing a subject through an imaging sensor and outputting an image signal; A second step of receiving the image signal output in the first step, converting a signal for each pixel of the image signal into a 1-bit signal, and then comparing the before and after image data to calculate a motion vector; It is carried out including.

According to the present invention, it is possible to provide a more efficient, compact and low power portable device input device and method thereof.

Description

Input device using imaging sensor and its method {Apparatus and method for input using image sensor}

1 is a block diagram of a small pointing device using a conventional finger surface.

2 is a block diagram of an input device using an imaging sensor according to an embodiment of the present invention.

3A to 3I are diagrams for explaining an example of performing optimized motion estimation using a 1-bit bit plane method for motion detected through the sensor unit of the present invention.

4 is a detailed block diagram of a preprocessing unit according to an embodiment of the present invention.

5A to 5C are diagrams illustrating a histogram pattern obtained by scaling a sampled image through a sensor unit according to an exemplary embodiment of the present invention.

6 is a graph showing a histogram of image information of a finger fingerprint input through a sensor unit according to an exemplary embodiment of the present invention.

FIG. 7 is a diagram illustrating an example of converting gray scale information of a sensor unit into a 1-bit signal using a 1-bit algorithm according to an embodiment of the present invention.

8 is a flowchart illustrating an operation of a comparison unit according to an embodiment of the present invention.

9 is a block diagram illustrating an input device using an imaging sensor according to another exemplary embodiment of the present invention.

10 is a flowchart illustrating an operation according to a frame of an operation state controller according to an embodiment of the present invention.

11 is a flowchart illustrating an operation of a power control unit according to an embodiment of the present invention.

12 is a detailed block diagram of a core controller according to an embodiment of the present invention.

13 is a flowchart illustrating an input method using an imaging sensor according to an exemplary embodiment of the present invention.

Explanation of symbols on the main parts of the drawings

100 sensor 110: imaging sensor

120: light source 130: lens

140: transparent plate 200: calculator

210: preprocessing unit 211: DCT unit

212 quantization (Q) unit 213 IDCT unit

220: quantization processing unit 230: temporary storage unit

240: comparison unit 241: XOR operation unit

242: pattern comparison unit 250: direction selection unit

260: operation state selection unit 270: core control unit

280: power control unit 271: light source intensity register

272: quantization threshold register 300: interface unit

400: mobile device 410: mobile phone

420: PDA 430: Notebook

440: digital camera 450: MP3 player

The present invention relates to an input device using an imaging sensor and a method thereof, and more particularly, to an input device using an imaging sensor and a method for sensing an input medium such as a finger using the imaging sensor to be input in a mouse format in a portable device. It is about.

Through digital convergence, various functions are converged in various mobile devices and provide services to users as a single computer. Therefore, a keypad touch pad type user interface provides inconvenience in providing such a service.

In order to provide a user-friendly interface, a two-dimensional input device is required on a graphic interface. In general, a mobile device is composed of a joystick, a direction key, or a touch pad, and these are limited to moving up / down and left / right, which causes a lot of inconvenience. Therefore, research on the combination of a mouse-type pointing device and a mobile device has been actively conducted. In addition, due to the miniaturization of mobile devices, there is a great burden on large-volume input devices such as keypads, and there is a demand for miniaturization of input devices.

Recently, as a miniaturized pointing device, Patent Application No. 10-2003-0022936, Apr. 11, 2003, "A small pointing device using a finger surface" (hereinafter referred to as prior art 1) is disclosed.

1 is a block diagram of a small pointing device using a finger surface of the prior art 1.

As shown therein, the light emitting means 3 which emits light to the contact object controlling the pointer to be moved; Condensing means (5) for condensing an image generated by light emitted from said light emitting means (3) to a contact object; Light receiving means (7) for detecting an analog image collected by said light collecting means (5) and converting it into a digital image; Motion detecting means (9) for grasping the degree of motion from the digital image received from said light receiving means (7); Position calculating means (11) for receiving the degree of movement, i.e., displacement data, from the movement detecting means (9) and determining the direction and distance / distance to which the pointer will move using the displacement data; A casing 13; And a hole 15 formed in the casing 13 to allow the light emitted from the light emitting means 3 to be reflected on the surface of the finger.

The prior art 1 shows a small pointing device that detects an image of a finger surface and calculates a displacement vector of a pointer, but does not provide an efficient method of condensing and receiving light, motion detecting means, and position calculating means. There is a limit to providing a low power input device.

In particular, in order to miniaturize the input device, the sensor and the processor must be miniaturized, but the prior art does not provide an efficient method.

In addition, in order to provide a user-friendly input device, various input methods must be implemented, and the related art does not provide a method for implementing a click and drag operation like a general mouse.

The present invention is to solve the inconvenience of the conventional input device, and to provide a more compact and low power efficient input device.

Accordingly, the present invention has been proposed to solve the conventional problems as described above, and an object of the present invention is to convert a captured image signal into an optimized minimum bit (1 bit) signal more efficiently with a minimum amount of data. The present invention provides a method for detecting an motion vector and a control method for minimizing idle power to provide an input device using an image sensor that is compact and consumes low power.

In addition, an object of the present invention is to provide an input device for a portable device and a method thereof that are convenient for a user by providing an efficient algorithm capable of various input methods such as a click and a drag.

In order to achieve the above object, the input device using the image pickup sensor of the present invention comprises a sensor unit including an image pickup sensor for detecting a subject which is an input medium and outputs a captured image signal; And an operation unit configured to receive an image signal output from the sensor unit, convert a signal for each pixel of the image signal into a 1-bit signal, and then compare the before and after image data to calculate a motion vector.

In addition, the input device using another imaging sensor of the present invention comprises a sensor unit including an imaging sensor for detecting a subject which is an input medium, and outputs a signal for each pixel of the captured image signal as a 1-bit signal; And an operation unit configured to receive the image signal output through the sensor unit and compare the front and rear image data to calculate a motion vector.
In addition, an input device using another imaging sensor of the present invention includes a sensor unit including an imaging sensor for sensing a subject, which is an input medium, and outputting a captured image signal; An operation unit including an operation state selection unit which receives an image signal from the sensor unit and determines an operation state of a subject based on the number or time of valid frames that are in focus with respect to the current image frame and invalid frames that are not in focus; It is made to include.

In addition, the input method using the image sensor of the present invention for achieving the above object comprises a first step of capturing a subject through the image sensor to output an image signal; And a second step of receiving the image signal output in the first step, converting a signal for each pixel of the image signal into a 1-bit signal, and comparing the before and after image data at a predetermined time interval to calculate a motion vector. To perform.

In addition, another input method using an image pickup sensor of the present invention for achieving the above object comprises a first step of capturing a subject through the image pickup sensor to output an image signal; And a second step of determining an operation state of the subject based on the number or time of valid frames that are in focus with respect to the current image frame and invalid frames that are not in focus, by receiving the image signal output in the first step. Do it.

In addition, in order to achieve the above object, another input method using the imaging sensor of the present invention, the first step of imaging the subject through the imaging sensor to output an image signal; And a second step of determining an operation state of the subject based on the number or time of valid frames that are in focus with respect to the current image frame and invalid frames that are not in focus, by receiving the image signal output in the first step. Do it.

Hereinafter, the technical configuration of the present invention as described above will be described in detail with reference to the drawings.

2 is a block diagram of an input device using an imaging sensor according to an embodiment of the present invention.

As shown in the drawing, the input device using the image pickup sensor of the present invention includes an image pickup sensor 110 that detects a subject 150 that is an input medium, and the sensor unit 100 that outputs the detected image signal. Wow; And an operation unit 200 which receives an image signal output from the sensor unit 100, converts a signal for each pixel of the image signal into a 1-bit signal, and then compares the before and after image data to calculate a motion vector. It is done.

In general, the sensor unit 100 is approached by an imaging sensor 110 for sensing the subject 150, a light source 120 serving as a light emitting means for emitting light to the subject, and light supplied from the light source 120. It comprises a lens 130 which is a light collecting means for transmitting the image to the imaging sensor 110.

The sensor unit 100 may include an imaging sensor 110 having a minimum number of pixels optimized within a limit capable of imaging the subject 150 and calculating a motion vector in the operation unit 200. Such an image sensor preferably has a pixel number of 10,000 pixels or less, and an image sensor having a pixel number of about 1000 pixels can sufficiently calculate a motion vector.

The object 150 captured by the sensor unit 100 preferably has a surface to be formed with a plurality of minute irregularities, and a fingerprint forming part of a finger may be mainly used.

The operation unit 200 of the input device using the imaging sensor of the present invention converts an image signal input from the sensor unit into a 1-bit signal after converting the signal for each pixel of the image signal into a 1-bit signal using a 1-bit bitplane calculation scheme. It is characterized by calculating a vector.

The operation unit 200 having the above characteristics includes: a quantization processor 220 for converting a signal for each pixel of the image signal input from the sensor unit 100 into a 1-bit signal; A temporary storage unit 230 for temporarily storing image data processed by the quantization processing unit 220; A comparator 240 which obtains and compares the image data processed by the quantization processor 220 and the image data stored in the temporary storage 230 to obtain a difference between the front and rear image patterns at a predetermined time interval; A direction selector 250 for calculating a motion vector from the value output from the comparator 240; And a core controller 270 for controlling the operation of the calculator 200.

The calculating part 200 of the present invention may further include a preprocessing part 210 which removes a noise component from an image signal input from the sensor part 100.

In this case, as illustrated in FIG. 4, the preprocessor 210 performs a discrete cosine transform (DCT) on the image signal output from the sensor unit 100 to convert the information of the spatial domain into the frequency domain. 211; A filter unit 212 for performing a filter process by quantizing the image signal subjected to DCT in the DCT unit 211; And an IDCT unit 213 for performing inverse discrete cosine transform (IDCT) on the image signal processed by the filter unit 212 and converting the image signal into information of a spatial domain.

In the present invention, it is preferable to perform DCT and IDCT by using a Paig's algorithm, which is one of DCT algorithms capable of quickly calculating DCT that requires a large amount of computation to be suitable for an ultra high density integrated circuit (VLSI).

In addition, the preprocessing unit 210 further includes a stretching unit 214 that spreads a histogram of a sampled image with respect to the image signal output from the sensor unit 100 to improve the black and white contrast of the image pattern. It is desirable to. 5A, 5B, and 5C illustrate stretching by scaling the sampled image histogram, respectively.

In the operation unit 200, the comparison unit 240 performs an XOR operation on the current image data output from the quantization processing unit 220 with previous image data stored in the temporary storage unit 230, as shown in FIG. 8. An XOR operator 241; And a pattern comparison unit 242 that finds the minimum value of the difference between the front and rear image patterns by receiving the output of the XOR operator.

The comparator 240 uses a method of obtaining a difference of an image pattern while moving the current image by one pixel with respect to the previous image.

In the operation unit 200, the core controller 270 is configured to include a threshold value register 272 for setting the imaging sensor 110 to have a threshold value suitable for the surrounding environment, as shown in FIG. 12. In addition, the control unit 200 controls each functional block. The core controller 270 may further include a light source intensity register 271 for adjusting the light source intensity in the sensor unit 100.

In the input device using the image sensor of the present invention, an input method such as a click and a drag can be implemented like a computer mouse. To this end, the operation unit 200 may include an operation state selection unit 260.

The operation state selector 260 determines an operation state of the subject based on the number or time of valid frames that are in focus with respect to the current image frame and invalid frames that are not in focus.

As illustrated in FIG. 10, the operation state selector 260 determines that the current image frame is clicked when the number or time of valid frames following the invalid frame is smaller than the predetermined threshold value N _o . Also, for the current image frame, the number or time of the first valid frame following the first invalid frame is less than the predetermined threshold value N _o , and the number or time of the second invalid frame following the first valid frame is constant threshold value. If smaller than (N ₁ ), if the number or time of second valid frames following the second invalid frame is smaller than the predetermined threshold value (N _o ), it is determined by a double click operation. In addition, if the number or time of valid frames following the invalid frame is greater than the predetermined value N _o for the current image frame, it is determined as a drag operation. The threshold value N _o and the threshold value N ₁ may be arbitrarily set by a user as a register value.

The calculation unit 200 of the present invention preferably further includes a power supply control unit 280 to reduce power wasted in the idle state without input.

When there is no input, the power supply controller 280 turns off the power of the imaging sensor and the calculation unit when the subject is not in focus even though there is an input and when the subject is not operated for a predetermined time or more even when the subject is in focus. Control to use the minimum power.

The input device using the imaging sensor of the present invention may be designed to output a signal for each pixel of the image signal picked up by the sensor unit as one bit, as follows.

The input device using the image sensor includes an image sensor 110 that detects a subject 150, which is an input medium, and includes a sensor unit 100 that outputs a signal for each pixel of the captured image signal as a 1-bit signal. )Wow; And an operation unit 200 that receives the image signal output through the sensor unit 100 and compares the before and after image data at a predetermined time interval to calculate a motion vector.

As such, when the sensor unit itself outputs a signal of each pixel as one bit, the preprocessor 210 and the quantization processor 220 described above may be omitted by the calculator 200. In other words, the calculation unit includes a temporary storage unit 230, a comparison unit 240, a direction selector 250, a core control unit 270, and a power control unit 280.

The input device using the imaging sensor of the present invention is mainly used in connection with a portable device. In this case, the input device using the imaging sensor, the interface unit 300 for connecting the operation unit and the portable device; The mobile device 400 is operated by receiving the information calculated by the operation unit through the interface unit 300; In this case, the portable device may be a mobile phone, a PDA, a notebook computer, a digital camera, an MP3 player, or the like.

It is preferable that both the sensor unit and the calculation unit of the present invention are made of one integrated circuit chip, and the calculation unit of the present invention may be made of one integrated circuit chip. In this case, a smaller input device can be implemented.

13 is a flowchart illustrating an input method using an imaging sensor according to an exemplary embodiment of the present invention.

As shown here, the input method using the imaging sensor of the present invention comprises: a first step (ST1) of capturing a subject through an imaging sensor and outputting an image signal; A second step (ST2 to ST5) of receiving the image signal output in the first step (ST1), converting a signal for each pixel of the image signal into a 1-bit signal, and comparing the before and after image data to calculate a motion vector; It is carried out including.

In the second steps ST2 to ST5, an operation of calculating a motion vector after converting the image signal output in the first step ST1 into a 1-bit value by converting the signal per pixel of the image signal into a 1-bit value using a bit plane calculation method is performed. To perform.

These second steps ST2 to ST5 include a quantization process step ST3 for converting a signal per pixel of the image signal input from the first step ST1 into a 1-bit signal; A comparison step (ST4) for obtaining a difference between the front and rear image patterns at predetermined time intervals with respect to the image data processed in the quantization processing step (ST3); And a direction selection step of calculating a motion vector from the values output in the comparison step ST5.

The second steps ST2 to ST5 may further include a preprocessing step ST2 for removing noise components from the image signal input from the first step ST1 before the quantization processing step ST3.

The preprocessing step ST2 may include performing a DCT on the image signal output in the first step ST1 to convert information of the spatial domain into a frequency domain; Performing filter processing by quantizing the image signal subjected to DCT; And performing IDCT on the quantized image signal and converting the information into a spatial domain information.

In addition, the preprocessing step ST2 may include: a stretching step of unfolding the distribution of the histogram with respect to the sampled image with respect to the image signal output in the first step ST1 to improve the black and white contrast when the histogram is narrow; It is preferable to further include.

In the preprocessing step ST2, it is preferable to perform DCT and IDCT using a Paig's algorithm, which is one of fast DCT algorithms.

In the second step, when the optimized output bit for each pixel of the image sensor is output as one bit in the first step ST1, the preprocessing step ST2 and the quantization step ST3 may be omitted.

The comparing step ST4 includes an XOR operation step of performing XOR operation on the current image data output in the quantization processing step ST2 with the previous image data; And a pattern comparison step of determining a minimum value among differences between the front and rear image patterns by using the result of the XOR operation in the XOR operation step.

In the comparison step ST4, it is preferable to use a method of obtaining the difference of the image pattern while moving the current image by one pixel with respect to the previous image.

The input method using the imaging sensor of the present invention uses a power supply control method in order to prevent power consumption in the idle state without input.

As shown in FIG. 11, the power supply control method determines whether there is an input, and when there is no input, switches to a standby state, and when there is an input, switches to an operation state. In addition, after switching to the operation state, the focus of the subject is checked, and when the focus is not achieved, the operation is switched to the standby state, and when the focus is achieved, the operation or the motion state is estimated. In addition, after switching to the step of estimating the motion or the operation state, it is determined whether the subject continues to be operated, and when there is no continuous operation, the operation is switched to the standby state.

In the power supply control step, it is preferable to determine whether there is an input, determine the focus of the subject, and whether the continuous operation is performed based on a threshold value of a valid frame and / or an invalid frame for the current image frame.

Referring to the accompanying drawings, a preferred embodiment of the input device and the method using the image sensor according to the present invention configured as described above will be described in detail as follows.

In the following description of the present invention, detailed descriptions of well-known functions or configurations will be omitted if it is determined that the detailed description of the present invention may unnecessarily obscure the subject matter of the present invention. In addition, terms to be described below are terms defined in consideration of functions in the present invention, which may vary according to intention or precedent of a user or an operator, and thus, the meaning of each term should be interpreted based on the contents throughout the present specification. will be.

First, the present invention is to input a mouse type in a mobile device by optimizing a motion vector detection method in an input device using an imaging sensor. In other words, in response to changes in the operating system (OS) and visual interface for mobile devices that require multi-directional movements in place of the conventional navigation keys, the mouse type that a user is familiar with in a personal computer (PC) environment is used. It is to provide an input device. Desktop PCs use optical mice and trackballs with optical sensors, and laptops use touchpads or jogs. However, these pointing devices are not suitable for portable devices.

Accordingly, the present invention proposes an efficient algorithm for calculating a motion vector by estimating the movement of a subject (finger, etc.) using an imaging sensor, and provides a technique for developing a device that can be used as an input device of a portable device.

2 is a block diagram of an input device using an imaging sensor according to an embodiment of the present invention.

As shown in FIG. 2, the input device using the imaging sensor of the present invention includes a sensor unit 100 and a calculation unit 200. In addition, the operation unit 200 is connected to the mobile device 400 through the interface unit 300 to be used.

In the present invention, the sensor unit 100 including an image pickup sensor optimized to the minimum number of pixels, an efficient method of calculating a motion vector by converting a signal per pixel to a minimum bit (1 bit), click and drag, etc. It is proposed to provide a more efficient, compact and low-power input device by proposing a calculation unit 200 that operates in a power control method that can be applied to various input forms and minimizing the idle power.

1. Key features of the present invention

1-1) Low-power and low-area motion estimation device by determining the minimum number of pixels of the imaging sensor that can recognize the movement of the subject

In the case of using a conventional commercially available CMOS image sensor (CIS), the QCIF (176x144), for example, a sensor of more than 10,000 pixels is used to consume a large area and a lot of power to use as an input device.

Therefore, in the present invention, the optimized image sensor is used by recognizing the movement of the subject (finger, etc.) and determining the minimum number of pixels that can find the motion vector.

Available imaging sensors are already available commercially available CMOS Image Source (CIS), Charge Coupled Device (CCD), Photoconductor, Photodiode, MOSFET (Metal Oxide Semiconductor Field Effect Transistor, Metal Oxide Semiconductor Field Effect Transistor) Based sensors.

The imaging sensor of the present invention can be reduced to 1/10 or less than the conventional method using 1000 pixels or less through the optimized two-dimensional array type imaging sensor. In addition, input information that needs to be processed in real time is reduced, and the number of input frames can be increased by more than 10 times to process input information of 1000 frames per second, and the input information can be processed by a small number of computing devices.

1-2) Motion Estimation Algorithm Optimized as Core-Logic of Imaging Sensor

The motion estimation algorithm proposed in the present invention can be implemented in one chip, and the overall block diagram thereof is shown in FIG. 2.

In the conventional inventions, the motion estimation vector value is calculated through software using the output image data of the sensor using a processor such as a digital signal processor (DSP).

In contrast, the input device proposed by the present invention can calculate and output a motion estimation vector value by itself (hardware also), thereby minimizing chip size.

The motion estimation operation is based on the experimental results as shown in FIG. 3 and employs a 1-bit bit plane method, thereby simply implementing logic itself and optimizing for a portable device.

3A to 3I show experimental results of eight bit plane images, and FIG. 3A is an original image, and FIGS. 3B to 3I show a first bit image (FIG. 3I) in an eighth bit image (FIG. 3B). Respectively. In FIG. 3A to FIG. 3I, the difference between the original image and the image pattern is shown by bit plane images of different bit levels, and it can be seen that it is sufficient to perform motion estimation within the upper 4 bits. This experimental result shows that the motion estimation can be performed using only one bit of information in the present invention.

Therefore, the calculation unit converts the image signal input from the sensor unit into a 1-bit value using a 1-bit bitplane calculation scheme and then calculates a motion vector. That is, a method of efficiently calculating a motion vector with a minimum amount of data is provided.

1-3) Implementation of click and drag gestures

The click and drag operation may be implemented by determining an operation state of the subject based on the number or time of the effective frame that is in focus with respect to the current image frame and the invalid frame that is not in focus.

In addition to the above-described core features of the present invention, a power supply control method and a method of controlling the optimum criteria of the sensor unit according to the variable environment of the sensor unit are proposed to be applied to a portable device with low power consumption.

2. Detailed configuration and operation of the present invention

As shown in FIG. 2, the present invention is largely divided into a sensor unit 100 including an imaging sensor 110 and an operation unit 200 for obtaining a motion vector.

The sensor unit 100 includes an imaging sensor 110, a light source 120, a lens 130 for proximity images, and a transparent plate 140. The operation unit 200 includes a preprocessor 210 and a quantization processor ( 220, a temporary storage unit 230, a comparator 240, a direction selector 250, a power control unit 260, and a core control unit 270. In addition, the interface unit 300 and the portable device 400 may be further included. The mobile device may be configured as a mobile phone 410, PDA 420, notebook 430, digital camera 440, MP3 player 450 and the like.

2-1) Sensor

In general, the sensor unit 100 is approached by an imaging sensor 110 for sensing the subject 150, a light source 120 serving as a light emitting means for emitting light to the subject, and light supplied from the light source 120. It comprises a lens 130 which is a light collecting means for transmitting the image to the imaging sensor 110. In addition, the transparent plate 140 may be provided to protect the lens 130 and to contact the subject.

In the sensor unit 100, when using an imaging sensor having excellent sensitivity to light, or when the subject emits light (including electromagnetic waves) directly, the light source 120 as a light emitting means may be omitted.

The imaging sensor 110 constituting the sensor unit 100 includes a CMOS image source (CIS), a charge coupled device (CCD), a photoconductor, a photodiode, a MOSFET (metal oxide semiconductor field effect transistor), Metal oxide semiconductor field effect transistors) and the like can be used.

As described above, the sensor unit 100 detects the movement of the subject 150 and operates the imaging sensor 110 having the minimum number of pixels optimized within the limit in which the calculation unit 200 can calculate the motion vector. It is characterized by including. Such an image sensor preferably has a pixel number of 10,000 pixels or less, and an image sensor having a pixel number of about 1000 pixels can sufficiently calculate a motion vector.

When using an image sensor having 1000 pixels or less, the number of pixels can be reduced to 1/10 or less than conventional methods. In addition, input information that needs to be processed in real time is reduced, and the number of input frames can be increased by more than 10 times to process input information of 1000 frames per second, and the input information can be processed by a small number of computing devices.

The object 150 captured by the sensor unit 100 preferably has a surface to be formed with a plurality of minute irregularities, and a fingerprint forming part of a finger may be mainly used. In the case of minute irregularities such as finger prints, black and white contrast is well represented in the image generated by the light reflected from the valleys and the floor, and the motion vector is converted into a 1-bit value by converting the signal for each pixel of the image signal as described below. It is easy to calculate.

Hereinafter, the configuration of the calculation unit 200 will be described based on the operation.

2-2) Preprocessor

Since the image signal input through the sensor unit 100 includes noise components, performance of performing motion estimation is degraded.

Therefore, the preprocessor 210 removes the noise component and increases the accuracy when the quantization processor 220 converts the signal per pixel into 1 bit.

In general, the noise component added to the input image signal has a high frequency characteristic. Therefore, the largest function of the preprocessor 210 is to convert the information of the spatial domain to the frequency domain using a discrete cosine transform (DCT), perform a filter process for improving image quality, and transmit the information to the quantization processor 220. In addition, the next quantization processor 220 needs to perform inverse DCT (IDCT) because information of a spatial domain rather than a frequency domain is required.

In addition, when the histogram distribution of the input image is excessively dense, the histogram should be unfolded for the operation of the quantization processor. Therefore, the stretching unit 214 is required to perform a stretching process to spread the histogram distribution.

Therefore, the preprocessing unit 210 is largely composed of a DCT unit 211-> quantization (Q) unit 212-> IDCT unit 213-> stretching unit 214, the detailed configuration is as shown in FIG.

4 is a detailed block diagram of the preprocessor of FIG. 2.

2-2-1) DCT / IDCT

In general, DCT requires a lot of computation and has a lot of difficulty in designing with hardware. Therefore, in the present invention, it is more suitable for VLSI (Very Large Scale Integrated circuit) by using a Tag algorithm (Feig's algorithm) which is one of the fast DCT algorithm.

2-2-2) Filter

In the present invention, the experiment for designing a filter suitable for the designed motion estimation apparatus is preceded. Algorithm development and optimization during the development of the motion estimation device design the imaging sensor at the same time, the filter coefficient is determined in consideration of the characteristics of the designed sensor unit 100 serves to remove the noise component in the high frequency region.

Therefore, the input information through the DCT unit 211 that changes the information of the spatial domain to the frequency domain removes the information of the high frequency region that is a noise component through the quantization (Q) unit 212. The input information from which the noise component is removed is inversely transformed by the IDCT unit 213 to be converted into information of the spatial domain.

2-2-2) Stretching part

The image sampled through the sensor unit 100 may be a pattern of a histogram having a narrow density according to the surrounding environment. This provides a limitation in performing quantization in a quantization processor to be described later. Therefore, a stretching process is needed to expand the histogram distribution. 5A and 5B show a histogram of a sampled image and a histogram of a scaled image, and FIG. 5C shows an image of a result of stretching the original sample image.

2-3) Quantization Processing Unit

When the input information of the spatial region from which the noise component is removed by the preprocessor 210 is input, the quantization processor 220 quantizes the signal for each pixel of the image signal into 1 bit. In general, in image processing, quantization is used to remove duplicate image information, but in the present invention, an optimal threshold value obtained through experiments is applied to recognize a Boolean value, that is, "black" and "white". It converts into a bit signal and represents a high or low intensity value using brightness.

That is, assuming that the nth value is g (n) and the threshold value is Dth, the output result of the quantization process is as follows.

g (n) = 0 (g (n) <Dth)

g (n) = 1 (g (n) ≥ Dth)

6 is a graph showing a histogram of pattern information of a finger fingerprint input through a sensor unit according to an exemplary embodiment of the present invention.

As shown in FIG. 6, the fingerprint of the finger is formed in a valley and a floor, and the black and white contrast appears well. Therefore, the present invention uses a bit plane calculation method that treats an image signal input through the imaging sensor 110 as binary data per pixel and converts information of each pixel into a 1-bit value.

The imaging sensor used in the present invention does not require the use of a color filter having information of a red green blue (RGB) region, and only outputs black and white information through gray scale input information. The valleys and floors of the fingerprint may be displayed in black and white information, and are quantized by a predetermined threshold T as shown in FIG. 6. Therefore, the image signal output from the imaging sensor 110 through the quantization processor 220 is converted into 1-bit information for each pixel.

As such, when the signal for each pixel is processed into 1 bit, a very fast logic operation is performed. That is, the difference from the previous image pattern can be easily obtained by using XOR (exclusive OR), and the size of hardware is reduced.

2-4) Comparison

The comparator 240 calculates a difference between the previous value and the current value output from the quantization processor 220. To this end, the temporary storage unit 230 temporarily stores image data processed by the quantization processor 230.

In addition, the XOR calculator 241 of the comparator 240 XORs the image data output from the quantization processor 220 with the previous image data, and compares the difference between the image patterns in the pattern comparator 242 to thereby compare the previous image pattern. And the difference between the current image pattern can be easily obtained. That is, it has a value of 1 for the part where the difference occurs. It moves the current image by one bit with respect to the previous image and finds the smallest value.

The comparison unit uses the One Pixel Full search method. One pixel full search method is to find the difference of the image pattern by moving the current image by 1 bit with respect to the previous image.

The comparator 240 is a part having the largest amount of computation since the preprocessor 210.

FIG. 7 is a diagram illustrating an example of converting gray scale information of a sensor unit into 1 bit information of each pixel by a 1 bit algorithm according to an embodiment of the present invention.

FIG. 7 illustrates a result of converting grail scale information of an imaging sensor into a 1-bit signal for each pixel using a 1-bit algorithm. In the direction selector, a motion vector may be obtained based on a difference between the current frame and the previous frame.

In the present invention, it is preferable that an imaging sensor having an optimized array size outputs 4-bit grail scale information, and the 4-bit data output from the sensor unit is converted into 1-bit information through preprocessing and quantization. do.

8 is a flowchart illustrating an operation of a comparison unit according to an embodiment of the present invention.

As shown in FIG. 8, the operation algorithm of the comparator repeatedly performs until the difference between the image pattern for the current image and the previous image is minimum. In this case, the current image shifted by 1 bit may be used by the direction selector 250 to calculate a vector.

2-5) Direction Selector

The direction selector 250 finds a motion vector by finding movement information of the smallest value, that is, the pattern that most matches the value output from the comparator 240.

The comparator 240 may move the current image and receive the moved current image information having the smallest value of the difference from the previous image to obtain a motion vector in consideration of the input period of the imaging sensor. Since the image information comes in at a certain time, if only the moving distance is calculated, the scalar operation value can be changed into a vector value through the time information.

2-6) Operation state selector

As shown in FIG. 9, the operation unit 200 of the present invention may further include an operation state selection unit 260 capable of implementing an input method such as a click and a drag like a computer mouse.

The operation state selector 260 detects an operation state by distinguishing between a valid frame in which the current frame is focused and an invalid frame in out of focus.

According to the click information indicating the state of the button, the operation state includes the one-click operation, which is the effect of pressing the button once, the double-click operation such as pressing the button twice consecutively, and the button. It is divided into Drag, the effect of moving the pointer while holding it.

The extraction of the operating state is determined by the number or time of valid and invalid frames. 10 is a flowchart illustrating an operation according to a frame of an operation state controller. As shown in this figure, the click of the button is estimated when the number of the first valid frames following the invalid frame is smaller than the predetermined threshold value N0. If the operation of the one click is performed continuously, if the number of intermediate invalid frames is smaller than the predetermined threshold value N1, the double click operation is selected. In addition, if the number of valid frames following the invalid frame is larger than the predetermined threshold value N0, a drag operation is determined.

The above-mentioned constant threshold values N0 and N1 are register values, and can be reset by the user according to a usage habit.

2-7) Power Control Unit

The power supply control unit is a key device for low power design suitable for an input device for a portable device.

The input device cannot predict when input information will come in, so it must always be powered on. In particular, when using an imaging sensor other than the button method, a lot of power is wasted in the idle state.

Therefore, the present invention proposes a hardware block for controlling the power supply of the light source included in the sensor unit and the power supply of the sensor unit. If there is no input for a certain period of time, the sensor is idled to minimize power consumption.

11 is a flowchart illustrating an operation of a power control unit according to an embodiment of the present invention.

As shown in the figure, it is first determined whether there is an input, and when there is no input, it is switched to a standby state, and when there is an input, it is switched to an operating state. After switching to the operation state, the focus of the subject is checked, and when the focus is not achieved, the operation is switched to the standby state, and when the focus is achieved, the motion or operation state is estimated. After switching to the step of estimating the motion or the operation state, it is determined whether the subject continues to operate, and when there is no continuous operation, the operation is switched to the standby state.

An object of the present invention is to provide an input device characterized by low power. In the standby state without input, the intensity of the light source is minimized and the power of the imaging sensor and the calculation unit is controlled to use the minimum power.

2-7) Core Control Unit

The core control unit controls each function block described above, and sets control registers using simple structured buses such as Integrated Injection Logic (I2C), Serial Peripheral Interface (SPI), and Universal Asynchronous Receiver and Transmitter (UART). It has the general feature of controlling the function block.

12 is a detailed block diagram of a core controller according to an embodiment of the present invention.

12 briefly illustrates an algorithm for determining the light source intensity and the quantization threshold of the sensor unit 100 through the core controller 270.

If the 1-bit output data of the quantization process is excessively biased at '0' or '1', the sensitivity of the imaging sensor may not be suitable for motion estimation. Therefore, to compensate for this, the core controller 270 includes a function of detecting such a problem and writing two special purpose registers.

First, the light source intensity register 271 for the sensor stage light source intensity is used to turn on or turn off the light source. The quantization threshold register 272 serves to set a threshold value suitable for each environment in consideration of the characteristic of the imaging sensor being changed by the surrounding environment.

2-8) Interface

The interface unit is for connecting the designed core and the mobile device. It is generally preferable to use the PS / 2 interface of the keyboard or mouse used in a computer and the serial device such as USB and I2C, SPI, and UART in the mobile device. Do.

The interface unit 300 converts the motion vector quantity and the operation state information output from the operation unit 200 into a form suitable for a standard bus required by a host controller, an embedded processor, or a portable device, and transmits the converted information. As the standard bus, as shown in FIG. 2, a PS / 2 interface unit 301, a universal serial bus (USB) interface unit 302, a serial peripheral interface (SPI) 303, and I2C (Integrated Injection Logic) 304, UART (Universal Asynchronous Receiver and Transmitter) 305, and the like.

2-9) Compensation of sensor function through variable environment setting of sensor part

The output result of the imaging sensor differs depending on the surrounding environment. That is, the value of the output data of the sensor unit changes according to the surrounding light source. Such a change may lead to a malfunction of the operation unit by the 1-bit bit plane method.

Therefore, in the present invention, the core control unit sets the intensity and the threshold of the light source of the sensor unit according to the surrounding environment. That is, as shown in FIG. 12, the function of the sensor unit 100 is complemented by setting the registers 271 and 272 through the core controller 270.

13 is a flowchart illustrating an input method for a portable device using an imaging sensor according to an exemplary embodiment of the present invention.

When the motion is detected through the imaging sensor (ST1), the preprocessing unit performs preprocessing by performing DCT-> Q-> IDCT (ST2), and then performs quantization processing by the quantization processor (ST3) Find that the difference between the image patterns is the smallest (ST4). Then, the motion vector of the smallest value, that is, the most matching image, is found among the values output by the direction selector (ST5).

The motion vector calculated by the direction selector controls the pointer of the mobile device input device by performing an interface with the mobile device through the interface unit. The portable device may be a mobile phone, a PDA, a notebook, a digital camera, an MP3 player, or the like.

In the input device of the present invention, a 1-bit signal may be output for each pixel from the sensor unit itself. That is, the sensor unit performs a quantization process on the picked-up image signal and outputs a signal of each pixel as one bit. In this case, the above-described preprocessor and quantization processor may be omitted in the calculator.

Although the above has been described as being limited to the preferred embodiment of the present invention, the present invention is not limited thereto and various changes, modifications, and equivalents may be used. Therefore, the present invention can be applied by appropriately modifying the above embodiments, it will be obvious that such application also belongs to the scope of the present invention based on the technical idea described in the claims below.

As described above, the present invention optimizes the image pickup sensor to the minimum number of pixels, converts a signal per pixel to 1 bit, calculates a motion vector with minimum data, and minimizes power by minimizing idle power. It is possible to provide a low power input device and a method thereof.

In addition, the present invention can provide an input device and a method that can be input in the form of a mouse in a mobile device by providing a method for implementing a click and drag function, the need for a simple input device according to the increase in the use of the mobile device In this regard, it is possible to meet the limitations of the conventional input device.

Claims (43)

A sensor unit including an imaging sensor for sensing a subject, which is an input medium, and outputting a captured image signal; An operation unit configured to receive an image signal output from the sensor unit, convert a signal for each pixel of the image signal into a 1-bit signal, and then calculate a motion vector by comparing front and rear image data; Input device using an imaging sensor comprising a. The method according to claim 1, The calculation unit, And a motion vector is calculated after converting the image signal input from the sensor unit into a 1-bit signal for each pixel of the image signal using a 1-bit bitplane calculation scheme. The method according to claim 1, The calculation unit, A quantization processor converting a signal per pixel of the image signal input from the sensor unit into a 1-bit signal; A temporary storage unit that temporarily stores the image data processed by the quantization processing unit; A comparison unit comparing the image data processed by the quantization processor with the image data stored in the temporary storage unit to obtain a difference between the front and rear image patterns; A direction selector for calculating a motion vector from the value output from the comparator; A core controller which controls an operation of the calculator; Input device using an imaging sensor, characterized in that configured to include. delete delete The method according to claim 3, The calculation unit, A preprocessor which removes noise components from the image signal input from the sensor unit; Input device using an image sensor, further comprising a. The method according to claim 6, The preprocessing unit, A DCT unit performing a DCT on the image signal input from the sensor unit to convert a signal in the spatial domain into a frequency domain; A filter unit filtering the image signal in which the DCT is performed in the DCT unit; An IDCT unit which performs IDCT on the image signal processed by the filter unit and converts the image signal into a signal of a spatial domain; Input device using an imaging sensor, characterized in that configured to include. The method according to claim 7, The preprocessing unit, A stretching unit that spreads a histogram distribution of the sampled image with respect to the image signal output from the sensor unit; Input device using an image sensor, further comprising a. delete The method according to claim 3, The comparison unit, An XOR calculator configured to XOR the current image data output from the quantization processor with previous image data stored in the temporary storage unit; A pattern comparison unit which receives the output of the XOR operation unit and finds a minimum value among differences between front and rear image patterns; Input device using an imaging sensor, characterized in that configured to include. delete The method according to claim 3, The core control unit, A threshold register which adjusts a threshold which is a reference in the quantization processor; Input device using an imaging sensor, characterized in that configured to include. The method according to claim 12, The core control unit, A light source intensity register for adjusting the light source intensity in the sensor unit; Input device using an imaging sensor, characterized in that it further comprises. The method according to claim 3, The calculation unit, An operation state selection unit that determines an operation state of the subject based on the number or time of the effective frame and the invalid frame that are not in focus with respect to the current image frame; Input device using an image sensor, further comprising a. A sensor unit including an image sensor which detects a subject, which is an input medium, and outputs a captured image signal; An operation state selection unit which receives an image signal from the sensor unit and determines an operation state of a subject based on the number or time of valid frames that are in focus with respect to the current image frame and invalid frames that are not in focus; Input device using an imaging sensor, characterized in that comprises a. The method according to any one of claims 3 to 15, The calculation unit, Further comprising a power control unit; The power control unit, If there is no input, if the subject is not in focus even if there is an input, and if the subject does not continue to operate for a certain time even if the subject is in focus, The input device using the image sensor, characterized in that the control to use the minimum power by controlling the power of the image sensor and the calculation unit. delete delete The method according to claim 1, The sensor unit, And an imaging sensor having a minimum number of pixels optimized within a limit capable of detecting a movement of the subject and calculating a motion vector in the calculator. The method according to claim 19, The imaging sensor of the sensor unit, An input device using an imaging sensor, characterized by having a pixel count of 10,000 pixels or less. The method according to claim 1, The subject photographed by the sensor unit, An input device using an imaging sensor, characterized in that it has a surface to be formed with a plurality of fine irregularities. The method according to claim 21, The subject is, Input device using an image sensor, characterized in that the fingerprint forming part of the finger. delete An integrated circuit chip comprising the sensor unit and the calculation unit according to any one of claims 1 to 15, which is composed of one chip. An integrated circuit chip comprising the computing unit according to any one of claims 3 and 15, wherein the integrated circuit chip is configured as one chip. A first step of photographing a subject through an imaging sensor and outputting an image signal; A second step of receiving the image signal output in the first step, converting a signal for each pixel of the image signal into a 1-bit signal, and then comparing the before and after image data to calculate a motion vector; Input method using an imaging sensor, characterized in that comprises a. The method of claim 26, The second step, And a motion vector is calculated after converting the image signal input from the step 1 into a 1-bit signal by using a 1-bit bitplane calculation method. The method of claim 26, The second step, A quantization processing step of converting a signal per pixel of the image signal input from the first step into a 1-bit signal; Input method using an imaging sensor, characterized in that comprises a. delete delete The method according to claim 28, The second step, A preprocessing step of removing a noise component from the image signal input from the first step; Input method using an image sensor, further comprising a. The method according to claim 31, The pretreatment step, Performing DCT on the image signal input from the first step to convert information of the spatial domain into a frequency domain; Performing a filter process by quantizing the image signal subjected to the DCT; Performing IDCT on the quantized image signal and converting the information into spatial information; Input method using an imaging sensor, characterized in that performed including. The method according to claim 32, The pretreatment step, A stretching step of spreading a distribution of the histogram with respect to the sampled image with respect to the image signal input from the first step; Input method using an image sensor, further comprising a. delete delete delete A first step of photographing a subject through an imaging sensor and outputting an image signal; A second step of receiving an image signal output in the first step and determining an operation state of the subject based on the number or time of valid frames that are in focus with respect to the current image frame and invalid frames that are not in focus; Input method using an imaging sensor to perform including. The method of claim 37, The second step, An input method using an image pickup sensor, characterized in that it is determined by clicking when the number or time of valid frames following the invalid frame is smaller than a predetermined threshold value (N _o ) for the current image frame. The method of claim 37, The second step, The number or time of first valid frames following the first invalid frame for the current image frame is less than the predetermined threshold value N _o , and the number or time of second invalid frames following the first valid frame is the constant threshold value N. If less than ₁ ), if the number or time of the second valid frame subsequent to the second invalid frame is less than a predetermined threshold value (N _o ) is determined by a double click operation, input method using an image sensor. The method of claim 37, The second step, An input method using an image sensor, characterized in that the drag operation is determined if the number or time of valid frames following the invalid frame is greater than a predetermined threshold value (N _o ) for the current image frame. The method according to any one of claims 38 to 40, The threshold value N _o and the threshold value N ₁ are An input method using an imaging sensor, which can be arbitrarily set by a user as a register value. The method according to any one of claims 26 and 37, Input method using image sensor, A power supply control method; The power control method, Determining whether there is an input, and when there is no input, switching to a standby state and when there is an input, switching to an operating state; Checking the focus of the subject after switching to the operation state, switching to the standby state when focus is not achieved, and estimating movement or operation state when focus is achieved; Determining whether to continue the operation of the subject after switching to the step of estimating the movement or operation state, and switching to the standby state when there is no continuous operation; Input method using an imaging sensor, characterized in that performed including. The method of claim 42, wherein The input method using the imaging sensor, characterized in that the determination of the presence or absence of the input, the focus confirmation of the subject, and the determination of the continuous operation is determined based on the threshold value of the valid frame and / or invalid frame for the current image frame.

Images