CN112764544B - A method of combining eye tracker and asynchronous motor imaging technology to realize virtual mouse - Google Patents

Abstract

The present invention provides a method for realizing a virtual mouse by combining an eye tracker and an asynchronous motion imaging technology. The coordinates of the eye-movement fixation point, and data processing is performed on the coordinates of the user's eye-movement fixation point, and the pointer position of the virtual mouse is associated with the visual track of the user, so as to realize the movement function of the virtual mouse; Step S2, realize the movement of the mouse through motor imagination Left and right button click function: The user imagines the actions associated with the left and right buttons, and realizes the left and right button click function of the mouse by decoding the brainwave signals of the motor area. The invention combines the eye tracker and the asynchronous motion imaging technology to realize the target selection of the virtual mouse, so that the user can realize the remote control of the computer only by the eyes and the brain without taking any action.

Description

A method of combining eye tracker and asynchronous motor imaging technology to realize virtual mouse

technical field

The invention relates to the technical fields of brain science and cognitive science, in particular to a method for realizing a virtual mouse by combining an eye tracker and an asynchronous motion imagery technology.

Background technique

Brain computer interface (BCI) is an interactive system built between humans and machines. Generally, the signal acquisition of brain-computer interface can be divided into three types: non-invasive, semi-invasive and invasive. Among them, the non-invasive type may be applied to real life at the earliest in the future because it does not cause harm to the human body. According to whether the EEG signal is caused by external stimuli, brain-computer interface technology can be divided into induced and spontaneous. Common evoked brain-computer interface technologies include steady-state visual evoked potential (SSVEP) and event-related potential (ERP). SSVEP is mainly a method for judging the subject's intention when the subject's EEG is induced by the harmonic signal of the corresponding frequency when the subject looks at the visual stimulus that flashes in black and white at a fixed frequency. ERP is a long-latency evoked potential, which mainly reflects the neurophysiological changes of the brain during the cognitive process. Classical ERP can divide the main components into P1, N1, P2, N2, P3 according to the latency, P or N represent positive or negative, 1, 2 or 3 represent the peaks appear about 100ms, 200ms or 300ms after the visual stimulus, respectively. . The most classic application is the P300Speller proposed by Farewell and Donchin in 1988, whose stimulation paradigm is mainly composed of random flickering of target and non-target stimuli. About 300ms after the appearance of the target stimulus with low probability, a positive wave peak, called the P300 signal, can be observed in the subject's EEG. By detecting this signal, the subject's intention can be judged. The evoked brain-computer interface technology is usually limited by the stimulation paradigm. In addition, both SSVEP and ERP need to flicker the eyes of the subjects for a long time, which is easy to cause visual fatigue and does not conform to human operation habits. Therefore, the patent mainly adopts the spontaneous brain-computer interface technology, motor imagery (MI).

MI technology does not depend on external stimuli and is an endogenous spontaneous EEG signal. It is only necessary for the subject to imagine a specific movement task, without the need for the subject to perform the actual movement, to detect specific waveforms in different brain regions, thereby judging the subject's intention. When the subjects are imagining the movement of a certain limb of the body, it will cause changes in different rhythms of the brain's sensorimotor cortex (Sensorimotor Rhythms, SMRs). SMRs induced by motor imagery of different limbs have different spatiotemporal distribution characteristics, and their EEG (Electroencephalogram, brain waves) have high distinguishability. Therefore, the MI pattern in the EEG can be decoded by the pattern recognition method and converted into a control command, so as to realize the control of the external device. However, the MI technology is limited by the number of classifications. Currently, the hands, feet and tongue can be better classified. That is to say, only 5 categories can be output in one classification, and it is difficult to directly output more instructions to achieve more complex task. The reason why elicited brain-computer interface technology can achieve a larger number of target recognition is by arranging multitasking on the display. The spontaneous brain-computer interface is more flexible and more in line with human operating habits. This led us to think about how to build a transformation bridge between the limited output categories of MI and the assignment of tasks.

Eye tracker is an important instrument for basic research in psychology. It is usually used to record the characteristics of human eye movement when processing visual information. It is widely used in the research of attention, visual perception, reading and other fields. At present, many laptops are equipped with eye trackers as a selling point. For example, the Alienware 17R5 is equipped with Tobii's eye trackers. However, because it can only track the gaze trajectory of human vision and does not have functions such as confirmation, its practical use in daily life is very limited, and it is usually used in games or professional psychological analysis.

SUMMARY OF THE INVENTION

The present invention provides a method for realizing a virtual mouse by combining an eye tracker and asynchronous motion imagery technology. The practical use of the device is limited, and the eye tracker and asynchronous motor imagery technology are not applied to the technical problems of the virtual mouse.

In order to achieve the above purpose, the present invention provides a method for realizing a virtual mouse by combining an eye tracker and an asynchronous motor imagery technology, comprising the following steps:

Step S1, realize the moving function of the mouse through the eye movement signal: collect the user's visual track in real time through the eye tracker, obtain the coordinates of the user's eye movement fixation point, and perform data processing on the coordinates of the user's eye movement fixation point, and the virtual mouse's coordinates are processed. The position of the pointer is associated with the user's visual track to realize the movement function of the virtual mouse;

Step S2, realizing the function of clicking the left and right buttons of the mouse through motor imagination: the user imagines the actions associated with the left and right buttons, and realizes the function of clicking the left and right buttons of the mouse by decoding the brain wave signals of the motor area.

Preferably, step S2 specifically includes the following steps:

Step S21, when the user wants to click the left mouse button, imagine grasping or clicking with the left hand; when the user wants to click the right mouse button, imagine grasping or clicking with the right hand;

Step S22, collecting the brain wave signal of the motor area of the user's brain in real time through the brain electricity collection device;

Step S23 , by decoding the brain waves of the user, distinguishing the movement intention of the left hand or the right hand of the user, so as to realize the control of the function of the left and right buttons of the mouse.

Preferably, step S1 is specifically:

Step S11, obtain the coordinates of the user's original gaze point: after calibrating the user's eye movement information, obtain the user's original gaze point coordinates through a python software development kit;

Step S12, as the user's eyes move, obtain the coordinates of the user's eye movement gaze point;

Step S13, excluding the coordinates of the eye movement fixation point collected by the eye tracker including the coordinates of blinking and saccade, and obtaining the coordinates of the sampling point of the user's fixation target;

Step S14, writing the eye movement visual trajectory coordinates represented by the coordinates of the user sampling points into the buffer;

Step S15 , associating the eye movement visual track coordinates of the buffer with the pointer position of the virtual mouse.

Preferably, the step S13 is specifically:

Step 131. Eliminate the coordinates of eye blinks that cannot be captured by the eye tracker: calibrate the coordinates where the eye tracker cannot capture eye blinks as (-1, -1), and remove the points whose coordinates are less than 0, and the eye tracker captures less than the coordinates of the blink of an eye;

Step 132, remove the coordinates of the saccade captured by the eye tracker: according to the sampling rate of the eye tracker, obtain the distance between two sampling points, that is, obtain the eye movement speed; Sampling points with a certain threshold are set as saccade samples and eliminated, and the coordinates of the sampling points of the user's gaze target are retained;

Preferably, in the step S23, after feature extraction is performed on the collected brainwave signals, the brainwaves are divided into left-hand mouse button click signals and mouse right-hand button click signals by a classifier to realize the selection function of the virtual mouse.

Preferably, the step S23 specifically includes the following steps:

Step S231, obtaining a brain wave signal of at least 2000ms;

Step S232, adopt the form of sliding window, continuously collect brain wave signal data with a duration of 100ms or 200ms, and put it into the classifier;

Step S233, brain electrolysis calculation: use the BCI algorithm to process the intercepted brain wave signal, after the feature extraction algorithm is performed, the sample score is obtained by the stepwise discriminant analysis method, and the classifier is divided into left-hand MI and right-hand MI classification. There are three types of classifier, left-hand MI and idle state classifier, and right-hand MI and idle state classifier. Through the three classifiers, the final classification result is obtained by voting in the form of left-hand, right-hand or idle state; if it is left-hand state, Then proceed to step S234 and step S235 in turn; if it is an idle state, then go to step S236; if it is a right-hand state, when the virtual mouse right button function is a non-menu command, then go to step S235 to perform anti-shake processing, and execute the right-hand state command , when the virtual mouse right button function is a menu command, step S234 and step S235 are sequentially entered to execute the menu command at the target pointed by the virtual mouse;

Step S234, read the coordinates of the eye movement gaze point buffer that is judged to be left-handed state or right-handed state;

Step S235: Use the software de-jitter method to classify and re-judg the coordinates of the eye-movement gaze point buffer: that is, when a piece of EEG data is judged to be in a non-idle state, compared with the result of the next sliding window, if it is the same left hand or right-hand state, then confirm this state, and go to step S237; otherwise, it is considered to be a false touch, and it will be changed to an idle state, and go to step S236;

In step S236, after it is determined to be in an idle state, it is considered that the user has no operation requirement or does not find the target option, and the virtual mouse does not perform any operation, thereby realizing asynchronous;

In step S237, the coordinates of the eye-movement gaze point buffer are displayed in the result feedback area of the corresponding position in the display, and the user confirms whether the selected target is his desired target; If the device misjudges the user's intention, the result is changed to an idle state, and the process proceeds to step S236.

Preferably, the feature extraction algorithm in step S233 includes an empirical mode decomposition algorithm and a common space mode algorithm.

Preferably, in the step S233, when the virtual mouse right button function is a non-menu command, the right-hand state instruction is specifically: an instruction to delete the last selection of the user.

Preferably, in the step S235, if the classification results of two consecutive sliding windows are both left-handed, average the coordinates of 12 gaze points during the two classification periods to obtain the user's desired gaze point.

Preferably, the step S23 further includes the following steps:

Step S238, stay for 500ms without processing data, and then enter the next cycle of EEG data processing.

The invention can achieve the following beneficial effects: combining the eye tracker and the asynchronous motion imaging technology to realize the target selection of the virtual mouse, so that the user can realize the remote control of the computer only by the eyes and the brain without taking any action. Through the present invention, all functions of the mouse can be realized without any action by the user. Combined with the soft keyboard function that comes with the computer, users can remotely control all functions of the computer directly through their eyes and brain, including web search, writing documents, playing games, etc. It is of great practical significance that the disabled aid equipment for patients with movement disorders is no longer limited to the spelling of specific characters.

Description of drawings

1 is a schematic diagram of a preferred embodiment of a method for realizing a virtual mouse in combination with an eye tracker and asynchronous motion imagery technology of the present invention;

2 is a schematic diagram of an option background interface of a display according to a preferred embodiment of a method for implementing a virtual mouse in combination with an eye tracker and an asynchronous motion imaging technique according to the present invention;

FIG. 3 is a schematic diagram of pixel coordinates of an effective area of an option of a display according to a preferred embodiment of a method for realizing a virtual mouse by combining an eye tracker and an asynchronous motion imaging technology according to the present invention.

Detailed ways

In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

Aiming at the existing problems, the present invention provides a method for realizing a virtual mouse by combining an eye tracker and an asynchronous motion imaging technology. The embodiment of the shaking invention includes that the eye tracker collects the visual trajectory of the user in real time, and associates the position of the mouse pointer with the visual trajectory of the user through python (a computer programming language) to realize the movement and selection of the virtual mouse. The two tasks of left and right hands in motor imagery are used as the left and right buttons of the mouse, respectively.

Eye movement and motor imagery are two different modalities that achieve different functions. The eye movement signal mainly realizes the movement function of the mouse, that is, where the eyes look, the pointer of the mouse moves. Motor imagination is mainly to realize the function of the left and right buttons of the mouse.

For example, to open a file:

Method 1: Step 1. The user stares at the file, and the eye tracker will align the mouse pointer with the file. Step 2. The user imagines the movement of the left hand twice in a row. left-click to open the file.

Method 2: Step 1 is the same as above; Step 2, the user imagines the movement of the right hand, and the brainwave solves the user's intention to right-click the file to open the function menu; Step 3, the user moves his eyes to the "Open" function, and the eye tracker moves the mouse The pointer is aligned with "Open"; in step 4, the user imagines the movement of the left hand, thereby selecting the "Open" function.

If you want to implement the sliding bar function commonly used in web browsing, step 1, the user aligns his eyes with the sliding bar; step 2, the user imagines the movement of the left hand, which is equivalent to long pressing the left mouse button to select the sliding bar; step 3, through the eye movement, Pull the slider down or left or right to realize the slider function.

As shown in FIG. 1 , it is a schematic diagram of a preferred embodiment of a method for realizing a virtual mouse by combining an eye tracker and an asynchronous motor imagery technology of the present invention. FIG. 2 is a schematic diagram of an option background interface of a display; FIG. 3 is a schematic diagram of pixel coordinates of an effective area of an option of the display.

It includes the following steps:

Step S1, realize the moving function of the mouse through the eye movement signal: collect the user's visual track in real time through the eye tracker, obtain the coordinates of the user's eye movement fixation point, and perform data processing on the coordinates of the user's eye movement fixation point, and the virtual mouse's coordinates are processed. The position of the pointer is associated with the user's visual track to realize the movement function of the virtual mouse;

Step S2, realizing the function of clicking the left and right buttons of the mouse through motor imagination: the user imagines the actions associated with the left and right buttons, and realizes the function of clicking the left and right buttons of the mouse by decoding the brain wave signals of the motor area.

In step S1, a Tobii pro (a brand) eye tracker with a sampling rate of 60 Hz is used. After calibrating the user's eye movement information, the user's original gaze point coordinates can be obtained through a python SDK (Software Development Kit). Because the selection function requires the subject to gaze at the target, through the speed threshold recognition and classification algorithm, we set the sampling points with eye movement speed greater than a certain threshold as saccade samples and eliminate them, and only keep the sampling points where the user is gazing at the target.

In this embodiment, the original eye movement coordinates collected by the eye tracker include the coordinates of blinking, saccade and gaze. Because the eye tracker cannot capture the eye when blinking, the coordinates are calibrated as (-1,-1), so the coordinates of the blink calibration can be eliminated by eliminating the points whose coordinates are less than 0. The saccade, or saccade, has no practical significance in this embodiment. The sampling rate of the eye tracker is fixed at 60Hz, and a point is sampled about every 17ms, and the distance between the two sampling points can represent the speed of eye movement. By calculating the Euclidean distance of each sampling point relative to the previous sampling point, and removing the sampling points larger than a certain threshold, the user's eye movement sampling points during the saccade process can be eliminated. Therefore, only the user's gaze point is retained in this embodiment, so that the synchronous association between the mouse position and the eye-movement gaze point can be realized later by using python's pymouse (a python mouse module).

Step S1 is specifically:

Step S11, obtain the coordinates of the user's original gaze point: after calibrating the user's eye movement information, obtain the user's original gaze point coordinates through a python software development kit;

Step S12, as the user's eyes move, obtain the coordinates of the user's eye movement gaze point;

Step S13, excluding the coordinates of the eye movement fixation point collected by the eye tracker including the coordinates of blinking and saccade, and obtaining the coordinates of the sampling point of the user's fixation target; specifically:

Step 131. Eliminate the coordinates of blinks that cannot be captured by the eye tracker: set the coordinates of the eyes that cannot be captured by the eye tracker as (-1, -1), and remove the points whose coordinates are less than 0 to be captured by the eye tracker. less than the coordinates of the blink of an eye;

Step 132: Eliminate the coordinates of the saccade captured by the eye tracker; obtain the distance between two sampling points according to the sampling rate of the eye tracker, that is, obtain the eye movement speed; Sampling points with a certain threshold are set as saccade samples and eliminated, and the coordinates of the sampling points of the user's gaze target are retained;

Step S14, writing the eye movement visual trajectory coordinates represented by the coordinates of the user sampling points into the buffer;

Step S15 , associating the eye movement visual track coordinates of the buffer with the pointer position of the virtual mouse.

Step S2 specifically includes the following steps:

Step S21, when the user wants to click the left mouse button, imagine grasping or clicking with the left hand; when the user wants to click the right mouse button, imagine grasping or clicking with the right hand;

Step S22, collecting the brain wave signal of the motor area of the user's brain in real time through the brain electricity collection device;

Step S23 , by decoding the brain waves of the user, distinguishing the movement intention of the left hand or the right hand of the user, so as to realize the control of the function of the left and right buttons of the mouse.

In step S23, after feature extraction is performed on the collected brainwave signals, the brainwaves are divided into left-hand mouse button click signals and mouse right-hand button click signals by a classifier to implement a virtual mouse selection function.

In terms of EEG, in order to realize the asynchronous function, although it is a left/right hand MI classification task, it is also necessary to discriminate the idle state. That is, each classification is a three-classification task. Usually, in order to ensure a certain classification accuracy, each MI classification task requires a period of EEG data of at least 2000ms. At the same time, in order to ensure the real-time performance as much as possible, the EEG signal is processed in the form of sliding window, that is, on the premise that the total length of the data is 2000ms, the data with a duration of 100ms is slid backward each time and put into the classifier. The data length of the sliding window mainly depends on the processing speed of the algorithm. In order to improve the reliability of the system to a certain extent and reduce misjudgment, the software de-jitter method commonly used in keyboards in single-chip microcomputers is adopted. That is, when a piece of EEG data is determined to be in a non-idle state, this state is not immediately confirmed. But compared with the result of the last sliding window, if it is still in the same state, it is confirmed as this state, otherwise it is considered to be a false touch, and it is changed to an idle state.

The step S23 specifically includes the following steps:

Step S231, obtaining a brain wave signal of at least 2000ms;

Step S232, adopt the form of sliding window, continuously collect brain wave signal data with a duration of 100ms or 200ms, and put it into the classifier;

Step S233, brain electrolysis calculation: use the BCI (brain-computer interface, Brain-Computer Interface) algorithm to process the intercepted brain wave signal, and go through a feature extraction algorithm (the feature extraction algorithm includes empirical mode decomposition (empirical mode decomposition, EMD). algorithm and common spatial pattern (CSP) algorithm) after feature extraction, the sample score is obtained by stepwise linear discriminant analysis (SWLDA), and the classifier is divided into left-hand MI and right-hand MI classifier, left-hand MI There are three types of MI and idle state classifiers, as well as right-hand MI and idle state classifiers. The final classification result is obtained in the form of voting through the three classifiers as one of left-handed, right-handed or idle state; if it is left-handed state, go to step S234, step S235; if it is in the idle state, then go to step S236; if it is in the left hand state, then go to step S234 and step S235 in turn; if it is in the idle state, go to step S236; if it is in the right hand state, when the virtual mouse right button function is When it is not a menu command, then enter step S235 to perform anti-shake processing, and execute the right-hand state command. When the virtual mouse right button function is a menu command, then enter step S234 and step S235 in turn to execute the menu at the target pointed by the virtual mouse. Order.

When the function of the virtual mouse right button is a non-menu command, the right-hand state instruction is specifically: an instruction to delete the last selection of the user.

Step S234, read the coordinates of the eye movement gaze point buffer that is judged to be left-handed state or right-handed state;

Step S235: Use the software de-jitter method to classify and re-judg the coordinates of the eye-movement gaze point buffer: that is, when a piece of EEG data is judged to be in a non-idle state, compared with the result of the next sliding window, if it is the same left hand or right-hand state, then confirm this state, and go to step S237; otherwise, it is considered to be a false touch, and it will be changed to an idle state, and go to step S236;

If the classification results of two consecutive sliding windows are left-handed, average the coordinates of the 12 fixation points during the two classification periods to obtain the user's desired fixation point;

In step S236, after it is determined to be in an idle state, it is considered that the user has no operation requirement or does not find the target option, and the virtual mouse does not perform any operation, thereby realizing asynchronous;

In step S237, the coordinates of the eye-movement gaze point buffer are displayed in the result feedback area of the corresponding position in the display, and the user confirms whether the selected target is his desired target; If the device misjudged the user's intention, it changed the result to an idle state, and entered step S236;

Step S338, stay for 500ms without processing data, and then enter the next cycle of EEG data processing.

The user presses the function key of the virtual mouse through the asynchronous MI, records the coordinates of the current fixation point buffer, compares it with the target area, and selects the target corresponding to the target area where the fixation point coordinates are located, which is the target selected by the user. MI, which can only implement two functions originally, can be extended to any number of functions through the eye movement intermediary. The eye movement without the confirmation function also expands its usefulness.

In this embodiment, the computer detects the EEG of the user by using an EEG detection device and intercepts a segment of EEG signal with a window length of 2000ms every 100ms, and then uses the BCI algorithm to process the intercepted EEG signal , after feature extraction by empirical mode decomposition algorithm and common space mode algorithm, the sample scores are obtained by stepwise discriminant analysis method. Through the above algorithms, the classifiers are divided into left-hand MI and right-hand MI classifiers, left-hand MI and idle state classifiers, and right-hand MI classifiers. There are three types of MI and idle state classifiers, and the final classification result is obtained in the form of voting through the three classifiers as one of left-hand, right-hand or idle state. If the classification result is idle, it is considered that the subject has no need for operation or no target option is found, and the computer does not perform any operation, thereby realizing asynchronous. When the classification result is left-handed, the gaze point coordinates are read from the eye movement coordinate buffer and compared with the data after the sliding window for 100ms. If the classification results are all left-handed, the coordinates of the 12 gaze points during the two classification periods are averaged. Thereby, the desired gaze point of the user is obtained. Compare the desired fixation point with the target option area in Figure 3. If the fixation point falls within the target option area, it is considered that the option is the user's desired choice, and the result is displayed in the result feedback area to help the user confirm whether it is his or her choice. desired goal. If the gaze point falls within the invalid area, it is considered that the classifier misjudged the user's intention, and the result is changed to an idle state.

In this embodiment, the right mouse button is not very useful, so the case where the MI classification result is the right hand is used instead as the last option to delete the user. It should be noted that every time the classification result is not in an idle state, anti-jitter processing needs to be performed to avoid mis-touching by the user or misjudgment by the classifier.

Through step S335, through the pymouse package, each time the recognition result is non-idle, the left or right mouse button is kept pressed until the next recognition ends.

When MI is continuously solved as left-hand, combined with eye movement, the page can be slid left and right, or up and down, that is, the scroll wheel function of the mouse. As a result, all the functions of the mouse can be fully realized through the combination of eye movement and asynchronous motor imagination, breaking through the fact that the handicap equipment for patients with movement disorders in the past can only realize the spelling of fixed characters. Combined with the soft keyboard function that comes with the computer, or turning on the P300 Speller function, patients with movement disorders can use all the functions of the computer, which is of great practical significance.

When the user presses the function key of the virtual mouse through the asynchronous MI, the coordinates of the current gaze point are recorded and compared with the target area. The target corresponding to the target area where the gaze point coordinates are located is the target selected by the user. As a result, MI, which can only implement two functions originally, can be extended to any number of functions through the mediation of eye movement. The eye movement without the confirmation function also expands its usefulness.

Each time a task is successfully selected, or the deletion is successfully triggered, the data for 500ms will not be processed. On the one hand, it is to allow time for the user to find the next target, and on the other hand, it can prevent the user from being too late to respond and cause continuous deletion or continuous deletion. The case where the same target is selected appears.

The invention can achieve the following beneficial effects: combining the eye tracker and the asynchronous motion imaging technology to realize the target selection of the virtual mouse, so that the user can realize the remote control of the computer only by the eyes and the brain without taking any action. Through the present invention, all functions of the mouse can be realized without any action by the user. Combined with the soft keyboard function that comes with the computer, users can remotely control all functions of the computer directly through their eyes and brain, including web search, writing documents, playing games, etc. It is of great practical significance that the disabled aid equipment for patients with movement disorders is no longer limited to the spelling of specific characters.

The above are the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (8)

1. a method for realizing virtual mouse in conjunction with eye tracker and asynchronous motion imagery technology, is characterized in that, comprises the steps: Step S1, realizing the moving function of the mouse through the eye movement signal: collecting the user's visual trajectory in real time through the eye tracker, obtaining the coordinates of the user's eye-moving fixation point, and performing data processing on the coordinates of the user's eye-moving fixation point, and converting the virtual mouse The pointer position is associated with the user's visual track to realize the movement function of the virtual mouse; Step S2, realize the function of clicking the left and right buttons of the mouse through motor imagination: the user imagines the actions associated with the left and right buttons, and realizes the function of clicking the left and right buttons of the mouse by decoding the brain wave signals of the motor area; Step S2 specifically includes the following steps: Step S21, when the user wants to click the left mouse button, imagine grasping or clicking with the left hand; when the user wants to click the right mouse button, imagine grasping or clicking with the right hand; Step S22, collecting the brain wave signal of the motor area of the user's brain in real time through the brain electricity collection device; Step S23, by decoding the user's brain waves, distinguish the movement intention of the user's left hand or right hand, and realize the control of the function of the left and right buttons of the mouse; The step S23 specifically includes the following steps: Step S231, obtaining a brain wave signal of at least 2000ms; Step S232, adopt the form of sliding window, continuously collect brain wave signal data with a duration of 100ms or 200ms, and put it into the classifier; Step S233, brain electrolysis calculation: adopt BCI algorithm to process the intercepted brain wave signal, after feature extraction is carried out by feature extraction algorithm, obtain sample score by stepwise discriminant analysis method, and classify the classifier into left-hand MI and right-hand MI classification There are three types of classifier, left-hand MI and idle state classifier, and right-hand MI and idle state classifier. The final classification result is obtained by voting through the three classifiers as one of left-hand, right-hand or idle state; if it is left-hand state, Then enter step S234 and step S235 in turn; if it is in an idle state, then enter step S236; if it is in a right-hand state, when the virtual mouse right button function is a non-menu command, then enter step S235 to perform anti-shake processing, and execute the right-hand state instruction , when the function of the virtual mouse right button is a menu command, step S234 and step S235 are sequentially entered to execute the menu command at the target pointed by the virtual mouse; Step S234, read the coordinates of the eye movement gaze point buffer that is judged to be left-handed state or right-handed state; Step S235: Use the software de-jitter method to classify and re-judg the coordinates of the eye-movement gaze point buffer: that is, when a piece of EEG data is judged to be in a non-idle state, compared with the result of the next sliding window, if it is the same left hand or right-hand state, then confirm this state, and go to step S237; otherwise, it is considered to be a false touch, and it will be changed to an idle state, and go to step S236; In step S236, after it is determined to be in an idle state, it is considered that the user has no operation requirement or does not find the target option, and the virtual mouse does not perform any operation, thereby realizing asynchronous; In step S237, the coordinates of the eye-movement gaze point buffer are displayed in the result feedback area of the corresponding position in the display, and the user confirms whether the selected target is his desired target; If the device misjudges the user's intention, the result is changed to an idle state, and the process proceeds to step S236. 2. a kind of method combining eye tracker and asynchronous motion imagery technology to realize virtual mouse according to claim 1, is characterized in that, step S1 is specifically: Step S11, obtaining the coordinates of the user's original gaze point: after calibrating the user's eye movement information, obtain the user's original gaze point coordinates through a python software development kit; Step S12, as the user's eyes move, obtain the coordinates of the user's eye movement gaze point; Step S13, excluding the coordinates of the eye movement fixation point collected by the eye tracker including the coordinates of blinking and saccade, and obtaining the coordinates of the sampling point of the user's fixation target; Step S14, writing the eye movement visual trajectory coordinates represented by the coordinates of the user sampling points into the buffer; Step S15 , associating the eye movement visual track coordinates of the buffer with the pointer position of the virtual mouse. 3. a kind of method combining eye tracker and asynchronous motion imagery technology to realize virtual mouse according to claim 2, is characterized in that, described step S13 is specifically: Step 131. Eliminate the coordinates of eye blinks that cannot be captured by the eye tracker: calibrate the coordinates where the eye tracker cannot capture eye blinks as (-1, -1), and remove the points whose coordinates are less than 0, and the eye tracker captures less than the coordinates of the blink of an eye; Step 132, remove the coordinates of the saccade captured by the eye tracker: according to the sampling rate of the eye tracker, obtain the distance between two sampling points, that is, obtain the eye movement speed; Sampling points with a certain threshold are determined as saccade samples and are eliminated, and the coordinates of the sampling points of the user's gaze target are retained. 4. a kind of method combining eye tracker and asynchronous motion imagery technology to realize virtual mouse according to claim 1, is characterized in that, in described step S23, after feature extraction is carried out to the brainwave signal collected, by classification The device divides the brain waves into the click signal of the left mouse button and the click signal of the right mouse button, so as to realize the selection function of the virtual mouse. 5 . The method for realizing a virtual mouse by combining an eye tracker and an asynchronous motion imagery technology according to claim 1 , wherein the feature extraction algorithm in the step S233 includes an empirical mode decomposition algorithm and a common space mode algorithm. 6 . 6. a kind of method that combines eye tracker and asynchronous motion imagery technology to realize virtual mouse according to claim 1, it is characterized in that, in described step S233, when virtual mouse right key function is non-menu command, described right-hand state The instruction is specifically: an instruction to delete the last choice of the user. 7. a kind of method that combines eye tracker and asynchronous motion imagery technology to realize virtual mouse according to claim 1, is characterized in that, in described step S235, if the classification result of two consecutive sections of sliding windows is left-handed state , then average the coordinates of the 12 gaze points during the two classification periods to obtain the user's desired gaze point. 8. a kind of method combining eye tracker and asynchronous motion imagery technology to realize virtual mouse according to claim 1, is characterized in that, described step S23 also comprises the following steps: Step S238, stay for 500ms without processing data, and then enter the next cycle of EEG data processing.

Images