CN110989873B - Optical imaging system for simulating touch screen - Google Patents

Abstract

An optical imaging system for simulating a touch screen, including an optical imaging component, an image preprocessing module, a checksum calculation module on the warning line, a module for obtaining fingertip positions, a mechanism for judging the type of operation, and a module for performing operations and the next sampling Sampling start line number fallback mechanism module; the optical imaging component is composed of a fixing clip and an optical structure component; in the image preprocessing module, the CPU calls the camera image at regular intervals; the original 24bit storage space is used to represent The information of the corresponding color corresponding to the position of the pixel is represented by 1 bit; in the judgment operation type mechanism and execution operation module, when the number of array rows is equal to 3, it indicates that the array of 3 rows and 2 columns is full, and then starts Execute the mechanism for judging the operation type to judge the elements in the array of 3 rows and 2 columns. The present invention can realize the touch effect for the display screen without using the touch screen.

Description

An optical imaging system for simulating a touch screen

technical field

The invention relates to the field of optical imaging system and image processing, in particular to an optical imaging system for simulating a touch screen.

Background technique

With the rapid development of the times, more and more electronic devices have entered thousands of households and entered the streets and alleys, and the display screen, as a medium for human-machine interaction, plays a pivotal role in people's daily life. No matter in the car, in the shopping mall, in the classroom, or even in the streets and alleys, the display screen can be seen everywhere. Outdoor display screens as large as hundreds of square meters, as small as smart watches for mobile phones, it can be said that it is precisely because of the existence of display screens that people can truly enjoy the comfort and convenience brought by the development of technology to people's lives.

But there are two concepts that are easy to confuse, that is, the difference between the display screen and the touch screen. The reason why some people are confused about these two concepts is largely because the touch screen and display screen are usually superimposed and used together. Take a mobile phone as an example. The screen of a mobile phone is divided into an inner screen and an outer screen. The inner screen is a display screen that truly presents different images. People will ignore its existence and mistakenly think that the touch screen is the display screen.

A touch screen has a very important measure, that is light transmittance. The light transmittance of a good touch screen can reach more than 99%, but at the same time the cost is very high, and the light transmittance of a common touch screen can only reach 70% to 80%. In the traditional sense, a large part of the display screen is used to interact with people. Therefore, the touch screen is usually used superimposed with the display screen, and at the same time, a contradiction arises. If you choose a good-quality touch screen, although you can truly display the screen displayed on the screen, the cost is too expensive; if you choose a moderately priced touch screen, it will greatly affect the display effect of the screen. Therefore, many screens used for interaction on the market, such as the screens of ATM cash machines, the screens of self-service ticket machines at railway stations, and the screens of self-service vending machines, all have a gray feeling, which is not because of the quality of the display screens. Not good, but because the light transmission of the touch screen is too poor, which affects the display effect of the display.

The role of laptops in people's daily life is self-evident. Whether it is study, work, life or entertainment, laptops are inseparable. There is no doubt that laptops have brought great changes to people's lifestyles, but The fly in the ointment is that most notebook computers do not support touch screen functions. Only a very small number of top-end computers of the brands of manufacturers have touch screen functions. However, the general price of these computers is more than 8,000 yuan, so only relying on the manufacturer to superimpose the touch screen on the display screen of the notebook computer to realize the touch function for the notebook computer cannot satisfy the ordinary user's touch function of the notebook computer. demand.

Contents of the invention

In order to overcome the fact that the existing touch screen notebook computer is expensive and cannot meet the needs of ordinary users for the touch screen function of the notebook computer. The invention provides an optical imaging system for simulating a touch screen, which can realize a touch effect for a display screen without using the touch screen.

The technical solution adopted by the present invention to solve its technical problems is:

An optical imaging system for simulating a touch screen, including an optical imaging component, an image preprocessing module, a checksum calculation module on the warning line, a module for obtaining fingertip positions, a mechanism for judging the type of operation, and a module for performing operations and the next sampling Sampling start line number fallback mechanism module;

The optical imaging component is composed of a fixing clip that can be clamped at the position of the camera on the screen of the notebook computer, and an optical structural component that can be combined with the fixing clip in a threaded way or a magnetic way;

In the image preprocessing module, the CPU calls the camera picture at regular intervals, that is, performs a sampling; the information of the corresponding color corresponding to the position of the pixel that needs to be represented by the storage space of 24bit is used by 1 bit To represent;

In the checksum calculation module on the warning line, in the initial state, there is no approximate color of skin color or the color selected as the main body color of the stylus in the camera observation range, so after the preprocessing process, all pixels in the camera field of view The output values are all zero. When a finger or stylus enters the observation range of the camera, the checksum of the 1920 pixel elements in the first row at the top of the field of view is not zero at first, and the 1920 pixel points in the first row are the warning line , the result of the checksum of the warning line is the criterion for whether there is a finger to manipulate the screen. If the checksum is zero, it means that no finger is stretched out; when the checksum on the warning line is non-zero, the fingertip Identification of the position and identification of the operation type; otherwise, after a delay of 0.02 seconds, proceed to the next sampling, and continue to judge whether the checksum on the warning line is zero;

In the module of obtaining the fingertip position, first pass the value of k to j, and start to calculate the checksum of the element in the jth row, k is initialized to 1, and j is the number of pixel rows of the display screen, and the maximum value is 1080. When the checksum of the i-th element in the j-th row is not zero, record the value of i at this time, i is the number of pixel columns on the display screen, and it will be initialized to 1 when the checksum calculation of each row is started, the maximum The value is 1920, and then calculate the checksum of the j+1th row; when the checksum of all 1920 elements in the jth row is still 0, it means that the jth row has no color within the tolerance range for the time being Objects (for subsequent descriptions of objects with colors within the tolerance range are directly described as "objects" in subsequent descriptions), record the value of j-1 at this time; at this time, the value in i and the value in j-1 represent The x-coordinate and y-coordinate of the current position of the "object". Store these two values in the first row of the array of 3 rows and 2 columns, and then add one to the row number of the array to indicate that the next row of the array is used to store the x and y coordinates of the object's position during the next sampling;

In the next sampling, in the sampling start row number fallback mechanism module, the x coordinate and y coordinate of the current position of the "object" are stored in the array of 3 rows and 2 columns in each sampling, and then the j-20 The value is assigned to k as the number of lines starting from the next sampling. Since the object is already in the jth line at this time, it is not necessary to calculate the checksum from the first line when sampling next time, and the object can be directly calculated from the last line. The jth line where it is located starts to calculate the checksum, but at the same time, in order to avoid the phenomenon of "object" turning back in the next sampling, 20 is set as the fallback value, so j-20 is selected as the number of lines to start the next sampling , this value can be adjusted through subsequent optimization, and at the same time, when the value of j-20 is less than 1, k is still equal to 1. At this point, the description in the first line of this paragraph "first pass the value of k to j to start calculating the checksum of the elements in row j" forms a closed loop of the cyclic sampling process;

In the mechanism for judging the type of operation and the module for executing the operation, when the number of rows in the array is equal to 3, it indicates that the array of 3 rows and 2 columns is full, and then the mechanism for judging the type of operation is started to judge the elements in the array of 3 rows and 2 columns .

Further, in the mechanism for judging the type of operation and the module for executing the operation, the data in the first column of each row of the array is represented by a1, a2, and a3 respectively, and the data in the second column of each row of the array is represented by b1, b2, and b3 respectively, The mechanism for determining the operation type is as follows:

(1) When b1<b2<b3, it means that the object is still moving forward;

(2) When b1<b2 and b3<b2, it means that the object has reached the touch position and starts to turn back. At this time, b2 is the turnback point. The information a2 and b2 of the second row element in the array represent the touch position Coordinate information, at this time, it is necessary to judge whether the operation is a double-click;

(2.1) If the situation (4) occurs within the limited time, it is determined that the operation type of this operation is double-click;

(2.2) If there is no situation within the limited time (4), it is determined that the operation type of this operation is single click;

(3) When b1>b2>b3, it means that the object is moving away;

(4) When b1>b2 and b3>b2, it means that the object starts to turn back and is ready to touch the screen again;

(5) When b1=b2=b3 (b1, b2, b3 are regarded as equal when the difference is very small), further judgment is carried out;

(5.1) When a1=a2=a3 (a1, a2, a3 are regarded as equal when the difference is very small), it means that the object is performing the right mouse button operation;

(5.2) When a1<a2<a3 or a1>a2>a3, it means that the object is turning the page up with the scroll wheel;

(5.3) When a1>a2>a3 or a1<a2<a3, it means that the object is turning the scroll wheel down;

After the situation (2) occurs, the two data a2 and b2 representing the horizontal and vertical coordinate information of the touch position in the second row of the array will be stored in X and Y respectively, and then it will be judged whether the operation is a double-click; The method of judging whether the operation is a double-click is similar to the principle of judging whether the "object" appears (2), that is, the sample is cyclically sampled within a limited time, and the sampled data is stored in an array of 3 rows and 2 columns, and the data in the array is compared The mechanism judges the information in the second column of the array. If the situation (4) occurs within a limited time, it means that the object will turn back within a limited time after the turning point of the situation (2), which means that the "object" is about to Touch the screen again, it is determined that the type of operation is a double-click, and a double-click command will be sent to the CPU, and the position information is in X, Y; if there is no situation (4) within the limited time, it will be determined after the limited time expires If the operation type of the second operation is click, a click command will be sent to the CPU, and the position information is in X, Y; after sending the single-click or double-click command, the program will return to the main function and continue to calculate the warning line at a sampling frequency of 50hz The checksum on the above completes the closed-loop operation of the whole process;

Make further judgments after the situation (5) occurs;

When a1=a2=a3 (a1, a2, a3 are regarded as equal when the difference is very small), it means that the object is operating the right button of the mouse. It will send the right button command to the CPU, and the position coordinates are in a1, b1 (or a2, b2 or a3, b3);

When a1<a2<a3 or a1>a2>a3, it means that the object is turning the page up with the scroll wheel. The page-up operation will be sent to the CPU;

When a1>a2>a3 or a1<a2<a3, it means that the object is turning the scroll wheel down. A page-down operation will be sent to the CPU;

After the instruction is sent, the program will return to the main function to continue to calculate the checksum on the warning line with a sampling frequency of 50hz, and complete the closed loop operation of the whole process;

If there is a situation other than case (2) and case (5), the data in the second row is assigned to the first row, and the data in the third row is assigned to the second row. At the same time, the number of rows in the array is reduced by one to indicate the next sampling The "object" position information obtained at the time will be stored in the third row of the array; after the next sampling, the number of rows in the array will be increased by one and then equal to three, and it will enter the closed loop of cyclic sampling and operation type judgment again until a situation occurs (2) or case (5);

Through the above process steps, you can use the laptop's own camera to add the effect of clicking, double-clicking, right-clicking, and page-turning functions to the laptop. In addition, you can also implement more complex functions through code design on this basis. Gesture operations such as zooming in and out, dragging, sliding down with three fingers to take screenshots, double-tapping with knuckles to take screenshots, etc.

Still further, the optical structure assembly is composed of three flat mirrors inclined at 45 degrees and a wide-angle lens with an observation angle of 160 degrees. The direction of view of the camera is turned to be parallel to the direction of the screen upwards; the plane mirror 2 is located at the direction of the plane mirror 1 upwards along the direction of the screen, and is parallel to the plane mirror 1, and is used to turn the direction of view of the computer camera back to be perpendicular to the direction of the computer screen; the plane mirror 3 is located at the direction of the plane mirror 2 and is perpendicular to The direction of the screen is forward and perpendicular to the plane mirror 2, which is used to turn the direction of view of the computer camera to be parallel to the direction of the screen downward; the wide-angle lens is located on the plane mirror 3 and is parallel to the direction of the screen downward to increase the viewing angle of view.

Plane mirror 1 and plane mirror 2 are used to elevate the viewing angle of the camera. Plane mirror 3 is used to turn the observation range of the camera by 90 degrees from perpendicular to the direction of the computer screen to parallel to the direction of the computer screen. The wide-angle lens is used to increase the viewing angle of view. The distance between the plane mirror 1 and the plane mirror 2 depends on the selection of the observation angle of the wide-angle lens. The larger the observation angle of the wide-angle lens, the shorter the distance between the plane mirror 1 and the plane mirror 2. The distance between the plane mirror 2 and the plane mirror 3 is to make the viewing direction of the camera form a certain angle with the screen plane, so as to facilitate subsequent recognition of the touch position. The design of the optical component can realize that the entire display panel can be observed through the camera when the camera picture is called, and then the position and type of the touch screen operation obtained by the camera can be identified through a certain image processing method and sent to the CPU. Respond accordingly.

Furthermore, a circular via hole is formed on one of the clamping surfaces of the fixing clip. The circular via hole is used to assist the user to judge whether the clamping position of the fixing clip is correct. The circular via hole is the optical path hole connecting the camera of the notebook computer with the optical component, and the design of separating the fixing clip from the optical component can avoid fixing Improper clamping position blocks the camera or the angle between the camera and the optical assembly is deviated.

In the image preprocessing module, select the color of the skin or the stylus as the reference color, and set the tolerance range, if the camera obtains the color information of the pixels of the picture within the tolerance range, then output the result 1, if not Within this tolerance value range, the output result is 0; the judgment method is to read the memory address corresponding to the pixel point, take out the 24 bits representing the color information of the pixel point in the corresponding memory address, and obtain the red color in the 24 bits by bit operation. , green, blue three primary colors corresponding values, and then get the corresponding value range of RGB three primary colors according to the set reference color tolerance range, and judge whether the corresponding values of red, green, blue three primary colors in the 24 bits of the pixel are within the tolerance within the range to complete the judgment.

In the image preprocessing module, the sampling frequency can be set manually, and the sampling frequency is 50 Hz as an example for illustration. When the laptop camera is working, it will apply for a memory area from the CPU to store the acquired image information, and the first address of the applied memory area is indicated by the pointer *p. Taking a computer with a resolution of 1920×1080 as an example, the data obtained by the camera is 1920*1080 pixels. Each pixel is encoded according to the RGB888 encoding method, that is, each pixel requires 24bit storage space to store the acquired color information.

The present invention provides a method of virtual touch screen, so as to realize the touch effect for the display screen without using the touch screen through the optical imaging system and the image processing method in the corresponding use scene. In the case of saving the cost of the touch screen itself, the negative impact on the display effect of the display screen due to the use of the touch screen is completely avoided. The present invention can be widely used in all-in-one machines in classrooms, display screens for shopping guides in shopping malls, self-service vending machines, and other screens that require an additional touch screen to achieve human-computer interaction. One of the usage scenarios is highlighted here, that is, using an optical imaging system and an image processing method for simulating a touch screen proposed by the present invention to add a virtual touch screen function to the display screen of a notebook computer.

The beneficial effects of the present invention are mainly manifested in: without using the touch screen, the touch effect can be realized for the display screen, and the negative effects caused by the use of the touch screen on the display effect of the display screen can be completely avoided while saving the cost of the touch screen itself. Influence.

Description of drawings

FIG. 1 is a flowchart of an image processing method.

2 is a design diagram of the optical imaging system, wherein, 1 is a plane mirror 1, 2 is a plane mirror 2, 3 is a plane mirror 3, 4 is a wide-angle lens, 5 is a fixing clip, and 6 is a notebook computer camera.

Fig. 3 is a flow chart of the mechanism for judging the operation type.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

Referring to Figures 1 to 3, an optical imaging system for simulating a touch screen, including an optical imaging component, an image preprocessing module, a checksum calculation module on the warning line, a module for obtaining fingertip positions, a mechanism for judging the type of operation, and execution The operation module and the sampling start line number fallback mechanism module at the next sampling;

The optical imaging component is composed of a fixing clip that can be clamped at the position of the camera on the screen of the notebook computer, and an optical structural component that can be combined with the fixing clip in a threaded way or a magnetic way;

In the image preprocessing module, the CPU calls the camera picture at regular intervals, that is, performs a sampling; the information of the corresponding color corresponding to the position of the pixel that needs to be represented by the storage space of 24bit is used by 1 bit To represent;

In the checksum calculation module on the warning line, in the initial state, there is no approximate color of skin color or the color selected as the main body color of the stylus in the camera observation range, so after the preprocessing process, all pixels in the camera field of view The output values are all zero. When a finger or stylus enters the observation range of the camera, the checksum of the 1920 pixel elements in the first row at the top of the field of view is not zero at first, and the 1920 pixel points in the first row are the warning line , the result of the checksum of the warning line is the criterion for whether there is a finger to manipulate the screen. If the checksum is zero, it means that no finger is stretched out; when the checksum on the warning line is non-zero, the fingertip Identification of the position and identification of the operation type; otherwise, after a delay of 0.02 seconds, proceed to the next sampling, and continue to judge whether the checksum on the warning line is zero;

In the module of obtaining the fingertip position, first pass the value of k to j, and start to calculate the checksum of the element in the jth row, k is initialized to 1, and j is the number of pixel rows of the display screen, and the maximum value is 1080. When the checksum of the i-th element in the j-th row is not zero, record the value of i at this time, i is the number of pixel columns on the display screen, and it will be initialized to 1 when the checksum calculation of each row is started, the maximum The value is 1920, and then calculate the checksum of the j+1th row; when the checksum of all 1920 elements in the jth row is still 0, it means that the jth row has no color within the tolerance range for the time being Objects (for subsequent descriptions of objects with colors within the tolerance range are directly described as "objects" in subsequent descriptions), record the value of j-1 at this time; at this time, the value in i and the value in j-1 represent The x-coordinate and y-coordinate of the current position of the "object". Store these two values in the first row of the array of 3 rows and 2 columns, and then add one to the row number of the array to indicate that the next row of the array is used to store the x and y coordinates of the object's position during the next sampling;

In the next sampling, in the sampling start row number fallback mechanism module, the x coordinate and y coordinate of the current position of the "object" are stored in the array of 3 rows and 2 columns in each sampling, and then the j-20 The value is assigned to k as the number of lines starting from the next sampling. Since the object is already in the jth line at this time, it is not necessary to calculate the checksum from the first line when sampling next time, and the object can be directly calculated from the last line. The jth line where it is located starts to calculate the checksum, but at the same time, in order to avoid the phenomenon of "object" turning back in the next sampling, 20 is set as the fallback value, so j-20 is selected as the number of lines to start the next sampling , this value can be adjusted through subsequent optimization, and at the same time, when the value of j-20 is less than 1, k is still equal to 1. At this point, the description in the first line of this paragraph "first pass the value of k to j to start calculating the checksum of the elements in row j" forms a closed loop of the cyclic sampling process;

In the mechanism for judging the type of operation and the module for executing the operation, when the number of rows in the array is equal to 3, it indicates that the array of 3 rows and 2 columns is full, and then the mechanism for judging the type of operation is started to judge the elements in the array of 3 rows and 2 columns .

Further, in the mechanism for judging the type of operation and the module for executing the operation, the data in the first column of each row of the array is represented by a1, a2, and a3 respectively, and the data in the second column of each row of the array is represented by b1, b2, and b3 respectively, The mechanism for determining the operation type is as follows:

(1) When b1<b2<b3, it means that the object is still moving forward;

(2) When b1<b2 and b3<b2, it means that the object has reached the touch position and starts to turn back. At this time, b2 is the turnback point. The information a2 and b2 of the second row element in the array represent the touch position Coordinate information, at this time, it is necessary to judge whether the operation is a double-click;

(2.1) If the situation (4) occurs within the limited time, it is determined that the operation type of this operation is double-click;

(2.2) If there is no situation within the limited time (4), it is determined that the operation type of this operation is single click;

(3) When b1>b2>b3, it means that the object is moving away;

(4) When b1>b2 and b3>b2, it means that the object starts to turn back and is ready to touch the screen again;

(5) When b1=b2=b3 (b1, b2, b3 are regarded as equal when the difference is very small), further judgment is carried out;

(5.1) When a1=a2=a3 (a1, a2, a3 are regarded as equal when the difference is very small), it means that the object is performing the right mouse button operation;

(5.2) When a1<a2<a3 or a1>a2>a3, it means that the object is turning the page up with the scroll wheel;

(5.3) When a1>a2>a3 or a1<a2<a3, it means that the object is turning the scroll wheel down;

After the situation (2) occurs, the two data a2 and b2 representing the horizontal and vertical coordinate information of the touch position in the second row of the array will be stored in X and Y respectively, and then it will be judged whether the operation is a double-click; The method of judging whether the operation is a double-click is similar to the principle of judging whether the "object" appears (2), that is, the sample is cyclically sampled within a limited time, and the sampled data is stored in an array of 3 rows and 2 columns, and the data in the array is compared The mechanism judges the information in the second column of the array. If the situation (4) occurs within a limited time, it means that the object will turn back within a limited time after the turning point of the situation (2), which means that the "object" is about to Touch the screen again, it is determined that the type of operation is a double-click, and a double-click command will be sent to the CPU, and the position information is in X, Y; if there is no situation (4) within the limited time, it will be determined after the limited time expires If the operation type of the second operation is click, a click command will be sent to the CPU, and the position information is in X, Y; after sending the single-click or double-click command, the program will return to the main function and continue to calculate the warning line at a sampling frequency of 50hz The checksum on the above completes the closed-loop operation of the whole process;

Make further judgments after the situation (5) occurs;

When a1=a2=a3 (a1, a2, a3 are regarded as equal when the difference is very small), it means that the object is operating the right button of the mouse. It will send the right button command to the CPU, and the position coordinates are in a1, b1 (or a2, b2 or a3, b3);

When a1<a2<a3 or a1>a2>a3, it means that the object is turning the page up with the scroll wheel. The page-up operation will be sent to the CPU;

When a1>a2>a3 or a1<a2<a3, it means that the object is turning the scroll wheel down. A page-down operation will be sent to the CPU;

After the instruction is sent, the program will return to the main function to continue to calculate the checksum on the warning line with a sampling frequency of 50hz, and complete the closed loop operation of the whole process;

If there is a situation other than case (2) and case (5), the data in the second row is assigned to the first row, and the data in the third row is assigned to the second row. At the same time, the number of rows in the array is reduced by one to indicate the next sampling The "object" position information obtained at the time will be stored in the third row of the array; after the next sampling, the number of rows in the array will be increased by one and then equal to three, and it will enter the closed loop of cyclic sampling and operation type judgment again until a situation occurs (2) or case (5);

Through the above process steps, you can use the laptop's own camera to add the effect of clicking, double-clicking, right-clicking, and page-turning functions to the laptop. In addition, you can also implement more complex functions through code design on this basis. Gesture operations such as zooming in and out, dragging, sliding down with three fingers to take screenshots, double-tapping with knuckles to take screenshots, etc.

Still further, the optical structure assembly is composed of three flat mirrors inclined at 45 degrees and a wide-angle lens with an observation angle of 160 degrees. The direction of view of the camera is turned to be parallel to the direction of the screen upwards; the plane mirror 2 is located at the direction of the plane mirror 1 upwards along the direction of the screen, and is parallel to the plane mirror 1, and is used to turn the direction of view of the computer camera back to be perpendicular to the direction of the computer screen; the plane mirror 3 is located at the direction of the plane mirror 2 and is perpendicular to The direction of the screen is forward and perpendicular to the plane mirror 2, which is used to turn the direction of view of the computer camera to be parallel to the direction of the screen downward; the wide-angle lens is located on the plane mirror 3 and is parallel to the direction of the screen downward to increase the viewing angle of view.

Plane mirror 1 and plane mirror 2 are used to elevate the viewing angle of the camera. Plane mirror 3 is used to turn the observation range of the camera by 90 degrees from perpendicular to the direction of the computer screen to parallel to the direction of the computer screen. The wide-angle lens is used to increase the viewing angle of view. The distance between the plane mirror 1 and the plane mirror 2 depends on the selection of the observation angle of the wide-angle lens. The larger the observation angle of the wide-angle lens, the shorter the distance between the plane mirror 1 and the plane mirror 2. The distance between the plane mirror 2 and the plane mirror 3 is to make the viewing direction of the camera form a certain angle with the screen plane, so as to facilitate subsequent recognition of the touch position. The design of the optical component can realize that the entire display panel can be observed through the camera when the camera picture is called, and then the position and type of the touch screen operation obtained by the camera can be identified through a certain image processing method and sent to the CPU. Respond accordingly.

Furthermore, a circular via hole is formed on one of the clamping surfaces of the fixing clip. The circular via hole is used to assist the user to judge whether the clamping position of the fixing clip is correct. The circular via hole is the optical path hole connecting the camera of the notebook computer with the optical component, and the design of separating the fixing clip from the optical component can avoid fixing Improper clamping position blocks the camera or the angle between the camera and the optical assembly is deviated.

In the image preprocessing module, select the color of the skin or the stylus as the reference color, and set the tolerance range, if the camera obtains the color information of the pixels of the picture within the tolerance range, then output the result 1, if not Within this tolerance value range, the output result is 0; the judgment method is to read the memory address corresponding to the pixel point, take out the 24 bits representing the color information of the pixel point in the corresponding memory address, and obtain the red color in the 24 bits by bit operation. , green, blue three primary colors corresponding values, and then get the corresponding value range of RGB three primary colors according to the set reference color tolerance range, and judge whether the corresponding values of red, green, blue three primary colors in the 24 bits of the pixel are within the tolerance within the range to complete the judgment.

In the image preprocessing module, the sampling frequency can be set manually, and the sampling frequency is 50 Hz as an example for illustration. When the laptop camera is working, it will apply for a memory area from the CPU to store the acquired image information, and the first address of the applied memory area is indicated by the pointer *p. Taking a computer with a resolution of 1920×1080 as an example, the data obtained by the camera is 1920*1080 pixels. Each pixel is encoded according to the RGB888 encoding method, that is, each pixel requires 24bit storage space to store the acquired color information.

As shown in the flow chart of the image processing method in Figure 1, after entering the main function, the CPU will read the picture captured by the camera every 0.02 seconds. Color information in RGB888 encoding format of pixels. The CPU will start from the first address of the memory area to calculate the results of representing the color information of the pixels on the warning line processed by the preprocessing function and add up to obtain the checksum. The input parameter of the preprocessing function is a pointer pointing to the starting memory address of each pixel. After the internal processing of the function, it is judged whether the color represented by the data stored at the memory address pointed to by the pointer is within the tolerance range of the reference color, and if it is within the tolerance range If it is inside, it outputs 1, otherwise it outputs 0. After the judgment is completed, the pointer moves backwards to the start address of the next pixel memory. When the checksum on the warning line is calculated (that is, the sum of 1920 0 or 1 results after preprocessing of the first 1920 pixels in the top row of the field of view), if the checksum is zero, a After the delay function of 0.02 seconds, return to the main function to continue sampling and calculate the checksum on the warning line again.

Execute the function to get the fingertip position when the checksum is not zero. The method is to calculate the checksum of 1920 elements in each row starting from the first row. When the i-th element of the 1920 elements is calculated, the checksum is not zero at the beginning, indicating that there are "objects" in this row, and "objects" " is at the i-th pixel of this row, then store the value of i at this time in the first column of the first row of the 3-row, 2-column array, and check all 1920 pixels when the j-th row is calculated and are both zero, it means that there is no "object" in this line, that is, the "object" is in line j-1. Then store j-1 in the second column of the first row of the array of 3 rows and 2 columns. Then add one to the number of rows in the array to point to the second row of the array. After a delay of 0.02 seconds, sample again, and also get a set of i and j-1 representing the horizontal and vertical coordinates of the "object". Store these two numbers in the first and second columns of the second row of the array, and then the number of rows in the array Add one to continue receiving the third sampling data. When the number of rows in the array is equal to three, the array of 3 rows and 2 columns is full, and the mechanism for judging the operation type is started to compare the information in the array.

The mechanism for judging the operation type will be described in detail below in conjunction with FIG. 3 .

When the situation (2) occurs, it means that the object has reached the touch position and starts to return. At this time, b2 is the return point, and the information of the second row element in the array represents the coordinate information of the touch position. At this time, the two data a2 and b2 representing the abscissa and ordinate information of the touch position in the second row of the array will be stored in X and Y respectively, and then the process of judging whether this operation is a double-click will be performed.

The judging method for judging whether this operation is a double-click is similar to the principle of judging whether the "object" appears (2), which is to cyclically sample within a limited time, and store the sampled data in an array of 3 rows and 2 columns. The judging operation type mechanism judges the second column information of the array.

If the situation (4) occurs within a limited time, it means that the "object" will turn back within a limited time after the turning point of the situation (2), which means that the "object" is about to touch the screen again, and then judge If the operation type is double-click, a double-click instruction will be sent to the CPU, and the position information is in X, Y.

If the situation (4) does not occur within the limited time, after the limited time expires, it will be determined that the operation type is a click, and a click command will be sent to the CPU, and the position information is in X, Y. After sending the single-click or double-click command, the program will return to the main function and continue to calculate the checksum on the warning line at a sampling frequency of 50hz, thus completing the closed-loop operation of the whole process.

When situation (5) occurs, further judgment is made.

When the situation (5.1) appears, it means that the object is performing the right mouse button operation. The position coordinates of the right button command will be sent to the CPU in a1, b1 (or a2, b2 or a3, b3).

When the situation (5.2) appears, it means that the object is turning the scroll wheel up. It will send the page up operation to the CPU.

When the situation (5.3) appears, it means that the object is turning the scroll wheel down. A page-down operation will be sent to the CPU.

After judging the type of operation and sending the corresponding command, the program will return to the main function and continue to calculate the checksum on the warning line at a sampling frequency of 50hz, thus completing the closed-loop operation of the whole process.

If case (2) or case (5) does not occur, the data in the second row in the array will be moved to the first row, the data in the third row will be moved to the second row, and the third row will be vacated, and the number of rows in the array will be reduced 1, indicating that the third row can be used to receive a new sampling data. After the next sampling, the number of rows in the array is increased by one and equal to three again, and then the mechanism for judging the operation type will continue. Until situation (2) or situation (5) occurs.

The design of the optical imaging system will be described in detail below in conjunction with FIG. 2 .

The design diagram of the optical imaging system is shown in Figure 2. The design of the optical imaging system consists of a fixed clip that can be clamped at the position of the camera on the screen of the laptop computer, and an optical structural component that can be combined with the fixed clip by screw thread or magnetic force. A circular through hole is formed on one of the clamping surfaces of the fixing clip. The circular through hole is used to assist the user to judge whether the clamping position of the fixing clip is correct. The circular via hole is the optical path hole for connecting the camera of the notebook computer with the optical component.

The main body of the optical component is composed of three flat mirrors inclined at 45 degrees and a wide-angle lens with a viewing angle of 160 degrees. The plane mirror 1 is positioned at the front of the camera of the notebook computer, and forms an angle of 45 degrees with the plane of the computer screen. The plane mirror 2 is located upward along the screen direction of the plane mirror 1 and is parallel to the plane mirror 1 . The function is to turn the direction of view of the computer camera back to be perpendicular to the direction of the computer screen. The plane mirror 3 is located forward of the plane mirror 2 along a direction perpendicular to the screen, and is perpendicular to the plane mirror 2 . The function is to turn the direction of view of the computer camera to be parallel to the direction of the screen and downward. The wide-angle lens is located at the plane mirror 3 and is parallel to the screen downwards, and the effect is to increase the viewing angle of view. The function of plane mirror 1 and plane mirror 2 is similar to that of a periscope, which is equivalent to raising the height of the camera. The function of plane mirror 3 is to turn the observation range of the camera by 90 degrees from perpendicular to the direction of the computer screen to downward parallel to the direction of the computer screen. Wide-angle lens is used to increase the viewing angle. The distance between the plane mirror 1 and the plane mirror 2 depends on the selection of the observation angle of the wide-angle lens. The larger the observation angle of the wide-angle lens, the shorter the distance between the plane mirror 1 and the plane mirror 2. The distance between the plane mirror 2 and the plane mirror 3 is to make the viewing direction of the camera form a certain angle with the screen plane, so as to facilitate subsequent recognition of the touch position.

In the description of this specification, the described specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner.

The above is only a preferred specific implementation of the patent of the present invention on the screen of a notebook computer, but the scope of protection of the patent of the present invention is not limited thereto. Within the scope of the invention patent, equivalent replacements or changes based on the technical solutions and concepts of the invention patent shall be covered within the scope of protection of the invention patent.

Claims (6)

1. The optical imaging system for simulating the touch screen is characterized by comprising an optical imaging component, an image preprocessing module, a checksum calculation module on a warning line, a fingertip position acquisition module, an operation type judging mechanism, an operation executing module and a sampling start line number rollback mechanism module in the next sampling;

the optical imaging assembly consists of a fixing clamp which can be clamped at the position of a notebook computer screen camera, and an optical structure assembly which can be combined with the fixing clamp in a threaded mode or a magnetic mode;

in the image preprocessing module, a CPU (Central processing Unit) invokes a camera picture once at regular intervals, namely, samples once; the information of the corresponding color corresponding to the position of the pixel point which is originally required to be represented by a 24-bit storage space is represented by 1 bit;

in the checksum calculation module on the warning line, no skin color approximate color or color which is selected as the main body color of the touch pen is in the observation range of the camera in the initial state, so that the output values of all pixel points in the visual field range of the camera are zero after the preprocessing process, when a finger or the touch pen enters the observation range of the camera, the checksum of 1920 pixel elements in the first row at the top of the visual field is firstly not zero, the 1920 pixel points in the first row are warning lines, the result of the warning line checksum is a criterion for controlling a screen by a finger or not, and if the checksum is zero, no finger extends; when the checksum on the warning line is non-zero, the fingertip position identification and the operation type identification are started; otherwise, continuing to sample the next time after the delay of 0.02 seconds, and continuing to judge whether the checksum on the warning line is zero;

in the fingertip position acquisition module, firstly, transmitting a value of k to j, starting to calculate a checksum of an element in a j line, initializing k to 1, wherein j is the number of pixel lines of a display screen, the maximum value is 1080, recording the value of i when the checksum is not zero when the element in the j line is calculated, i is the number of pixel lines of the display screen, initializing 1 when the checksum in each line is calculated, the maximum value is 1920, and then calculating the checksum of the j+1th line; when the checksum of all 1920 elements in the j row is calculated to be still 0, the j row is represented that the object with the color within the tolerance range is not existed temporarily, and the value of j-1 is recorded; at this time, the value in i and the value in j-1 represent the x coordinate and the y coordinate of the current position of the 'object', the two values are stored in the first row of the 3 row and 2 column array, and then the row of the array is added with a value representing the next row pointing to the array for storing the x coordinate and the y coordinate of the position of the object when the next sampling is performed;

in the sampling start line number rollback mechanism module in the next sampling, storing an x coordinate and a y coordinate of a current position of an object in a 3-line 2-column array in each sampling, and then assigning a value of j-20 to k as a line number started in the next sampling;

in the judging operation type mechanism and the executing operation module, when the number of the array lines is equal to 3, the array lines of 3 lines and 2 columns are indicated to be full, and then the executing operation type mechanism is started to judge the elements in the array lines of 3 lines and 2 columns.

2. The optical imaging system for simulating a touch screen according to claim 1, wherein in the determining operation type mechanism and the executing operation module, a first column data of each row of the array is denoted by a1, a2, a3, and a second column data of each row of the array is denoted by b1, b2, b3, respectively, and the determining operation type mechanism is as follows:

(1) When b1< b2< b3, it is stated that the object is also moving forward;

(2) When b1< b2 and b3< b2, it indicates that the object has reached the touch position and starts turning back, at this time, b2 is a turning back point, and the information a2, b2 of the second row element in the array represents the coordinate information of the touch position, at this time, it needs to determine whether the operation is double-click or not;

(2.1) judging that the operation type is double-click when the situation (4) occurs within a limited time;

(2.2) judging the operation type of the operation is a click if no condition (4) occurs within a limited time;

(3) When b1> b2> b3, it is indicated that the object is far away;

(4) When b1> b2 and b3> b2, indicating that the object starts turning back reversely, and preparing for touching the screen again;

(5) Further judgment is made when b1=b2=b3;

(5.1) when a1=a2=a3, explaining that the object is performing a right mouse button operation;

(5.2) when a1< a2< a3 or a1> a2> a3, explaining that the object is performing the operation of turning up the page by the wheel;

(5.3) when a1> a2> a3 or a1< a2< a3, explaining that the object is performing the scroll-down operation;

after the situation (2) occurs, two data a2 and b2 of the touch position abscissa information represented by the second row of the array are respectively stored in X and Y, and then whether the operation is double-click or not is judged; the judging mode of judging whether the operation is double-click is similar to the principle of judging whether the condition (2) appears in the object or not, namely, circularly sampling in a limited time, storing sampling data in an array of 3 rows and 2 columns, judging the second column information of the array through a data comparison mechanism in the array, if the condition (4) appears in the limited time, indicating that the object is reversely turned back after a turning back point of the condition (2) appears, namely, indicating that the object is about to touch the screen again, judging that the operation type is double-click, sending a double-click instruction to a CPU, and ensuring that the position information is in X and Y; if the situation (4) does not appear in the limited time, judging that the operation type is clicking after the limited time is over, sending a clicking instruction to the CPU, and positioning information is in X and Y; after the single click or double click instruction is sent, the program returns the main function to continue calculating the checksum on the warning line at the sampling frequency of 50hz, and the whole process operation closed loop is completed;

further judging after the occurrence of the condition (5);

when a1=a2=a3, indicating that the object is performing right mouse button operation, sending a right button instruction position coordinate in a1, b1 or a2, b2 or a3, b3 to the CPU;

when a1< a2< a3 or a1> a2> a3 indicates that the object is in the upward page turning operation of the roller, the upward page turning operation is sent to the CPU;

when a1> a2> a3 or a1< a2< a3 indicates that the object is performing the downward page turning operation of the roller, the downward page turning operation is sent to the CPU;

after the instruction is sent, the program returns to the main function to calculate the checksum on the warning line continuously at the sampling frequency of 50hz, and the whole process operation closed loop is completed;

if the situation (2) is out of the situation (5), the data of the second row is assigned to the first row, the data of the third row is assigned to the second row, and the number of the array rows minus one represents that the position information of the 'object' acquired in the next sampling is stored in the third row of the array; and after the next sampling, the number of the array lines is increased by one again and then is equal to three, and then the closed loop for circularly sampling and judging the operation type is re-entered until the situation (2) or the situation (5) occurs.

3. An optical imaging system for simulating a touch screen according to claim 1 or 2, wherein the optical structural component comprises 3 plane mirrors inclined by 45 degrees and a wide-angle lens with an observation angle of 160 degrees, the plane mirror 1 is positioned right in front of a camera of a notebook computer and forms an included angle of 45 degrees with the plane of the computer screen, and is used for turning the view direction of the camera of the computer to be parallel to the screen direction upwards; the plane mirror 2 is positioned on the plane mirror 1 upwards along the screen direction and is parallel to the plane mirror 1, and is used for turning the view direction of the computer camera back to be perpendicular to the direction of the computer screen; the plane mirror 3 is located forward along the direction vertical to the screen of the plane mirror 2 and is vertical to the plane mirror 2, and is used for converting the visual field direction of the computer camera into a direction parallel to the screen to be downward, and the wide-angle lens is located downward along the direction parallel to the screen of the plane mirror 3, and is used for increasing the observation visual angle.

4. An optical imaging system for simulating a touch screen as claimed in claim 1 or 2, wherein a circular through hole is formed on one of the clamping surfaces of the fixing clamp, the circular through hole is used for assisting a user in judging whether the clamping position of the fixing clamp is correct, the circular through hole is an optical path hole for connecting a camera of a notebook computer with the optical component, and the fixing clamp and the optical component are separated, so that the situation that the clamping position of the fixing clamp is blocked to the camera improperly or the angle for connecting the camera with the optical component is deviated can be avoided.

5. An optical imaging system for simulating a touch screen according to claim 1 or 2, wherein in the image preprocessing module, selecting skin or stylus color as reference color, setting a tolerance range, outputting result 1 if the camera acquires color information of pixels of the screen within the tolerance range, and outputting result 0 if the camera does not within the tolerance range; the judging method is to read the corresponding memory address of the pixel point, take out 24bit positions representing the color information of the pixel point in the corresponding memory address, obtain the corresponding values of red, green and blue three primary colors in the 24bit positions in a bit operation mode, obtain the corresponding value range of RGB three primary colors according to the set reference color tolerance range, and judge whether the corresponding values of red, green and blue three primary colors in the 24bit positions of the pixel point are in the tolerance range so as to finish the judgment.

6. The optical imaging system for simulating a touch screen according to claim 1 or 2, wherein in the image preprocessing module, the sampling frequency is set manually, when the notebook computer camera works, a section of memory area is applied to the CPU for storing the acquired picture information, the first address of the applied memory area is represented by a pointer x p, for a computer with 1920 x 1080 resolution, the acquired data of the camera is 1920 x 1080 pixels, each pixel is encoded according to the encoding mode of RGB888, that is, each pixel needs 24 bits of storage space for storing the acquired color information.

Images