CN116824938A - A virtual remote experiment system and method based on bioelectricity sensing - Google Patents

Abstract

The application belongs to the technical field of Internet and discloses a virtual remote experiment system and a virtual remote experiment method based on bioelectric perception. According to the application, the gesture is identified by extracting the electromyographic signal characteristics, so that the prediction in advance is realized, and the remote efficiency is further improved. The visual and controllable remote operation is truly realized.

Description

A virtual remote experiment system and method based on bioelectricity sensing

Technical field

The invention belongs to the field of Internet technology, and in particular relates to a virtual remote experiment system and method based on bioelectricity sensing.

Background technique

Experimental class is an indispensable course for students to test their comprehensive ability. It not only requires students' systematic organizational ability, but also tests their hands-on operation ability. On the basis of a comprehensive examination of students, higher requirements are also put forward for experimental equipment. Traditional laboratories are based on physical equipment. Students make an appointment to use it, participate in the entire operation, and wait for the experimental results. Although this process is real and easy to observe, it also causes problems such as long time consumption, the need to arrive in the laboratory in person, and the ineffective use of experimental resources, which all trouble people.

In recent years, the rapid development of remote communication technology has made office learning more convenient online, which has triggered our thinking about remote experiments. Students can control the laboratory through a complete remote operating system, anytime and anywhere. and achieve the same sense of reality as offline. In fact, the function of remotely controlling the start and stop of equipment through software has appeared, but it has not yet achieved truly immersive operation. The real sense of experience not only helps students master experiments, but also improves the efficiency of resource space utilization.

Contents of the invention

The purpose of the present invention is to provide a virtual remote experiment method based on bioelectricity sensing to solve the above technical problems.

In order to solve the above technical problems, the specific technical solutions of a virtual remote experiment system and method based on bioelectricity sensing of the present invention are as follows:

A virtual remote experiment system based on bioelectricity sensing, including a myoelectric sensing module, a virtual display module, a remote information module, a control machine module, and an environment sensing module;

The myoelectric sensing module is used to collect the user's arm electromyographic signals, process and identify them, and output accurate hand posture modeling results in real time;

The virtual display module is used to display the laboratory video screen in three dimensions to the user;

The remote information module is used for information processing and communication between the virtual terminal and the real terminal, and conducts experimental evaluation;

The control machine module is used to perform the same actions following the gestures recognized by myoelectricity, and uses the smart skin to obtain the touch parameters of the experimental supplies;

The environment sensing module is used to sense the environmental parameters of the laboratory in multiple directions and provide operational responses and effect feedback.

Further, the electromyographic sensing module includes an electromyographic signal acquisition device, an inertial measurement unit, an angle sensor, a microprocessor, a touch generator and a wireless communication module. The electrical signal acquisition device, the inertial measurement unit and the angle sensor are connected Microprocessor, the microprocessor is connected to the touch generator, and the wireless communication module is connected to the microprocessor; the inertial measurement unit is installed at the center of the experimenter's elbow; the angle sensor is installed on the finger; The touch generator is installed on the fingertips of both hands of the experimenter; the myoelectric signal acquisition device is used to collect muscle electrical signals; the inertial measurement unit is used to measure the inertia of the moving joints; the angle sensor is used to measure the angular direction of the fingers; microprocessing The processor is used to process the signals of each module; the wireless communication module is used to transmit signals.

Further, the touch generator includes a micro heater, a vibrating piece, and a micro humidity generator. Further, the electromyographic signal acquisition device includes an 8-channel surface electrode, a high-pass filter circuit, a differential amplifier, a low-pass filter circuit, and an analog-to-digital converter. The 8-channel surface electrode is connected to the high-pass filter circuit, and the high-pass filter circuit is connected to the differential amplifier. , the differential amplifier is connected to the low-pass filter circuit, and the low-pass filter circuit is connected to the analog-to-digital converter. The 8-channel electromyographic signal extracts preliminary features through wavelet transformation, digs deep features through the autoencoder, and obtains the time domain through the long short-term memory model. Features, and finally achieve continuous gesture classification, combined with finger angle sensor data for accurate prediction. Further, the virtual display module includes holographic imaging glasses, headphones and wireless communication modules. Further, the remote information module is a processor and wireless communication module loaded with control, encryption and evaluation algorithms.

Further, the control machine module is at least one set of manipulator system devices equipped with smart skin, which can sense the material, hardness, weight and pressure touch parameters of the experimental equipment in real time.

Further, the environment sensing module is at least a system including a multi-directional binocular stereo camera, a microphone, sensors for measuring parameters such as temperature, concentration, and light intensity, and a wireless communication module.

Further, the working process of the system is as follows. The environment sensing module captures the real-time video of the laboratory and wirelessly transmits it to the virtual display module. The three-dimensional display is displayed to the user. The user conducts experiments based on the real-time three-dimensional picture. Myoelectric sensing The module collects the user's hand posture during movements for accurate hand modeling. After being encoded and encrypted by the remote information module, it is wirelessly transmitted to the control machine module. The machine device in it performs specified actions according to the decrypted instructions, and the touch on the smart skin is The information is transmitted back to the myoelectric sensing module to give the user a response. At the same time, the experimental effect information is fed back to the virtual display module in real time, and the experiment is evaluated through the target recognition of the remote information module. The invention also discloses a control method for a virtual remote experiment system based on bioelectricity sensing, which includes a forward information transmission process and a reverse sensing feedback process. The specific steps are as follows:

S1: Holographic imaging: The binocular camera in the environmental sensing module is fixed above the manipulator device to capture the depth map of the experimental platform, which is transmitted to the virtual display module in real time through wireless communication. The holographic imaging technology is used to perform real-time three-dimensional imaging of the video. The holographic glasses are displayed to the user, and the user performs virtual experimental operations based on the holographic images they view;

S2: Hand action recognition: When offline, the myoelectric sensing module acquires the action electrical signals of the arm muscles and filters them through the circuit to denoise, rectify and smooth them. Then it uses wavelet transform to extract various features and builds a pattern recognition network for training. Online During operation, the predicted muscle strength E and gesture categories G ¹ _i , G ² _i (i=1~40) are output. At the same time, an inertial measurement unit is installed in the center of the elbow to record the movement direction D, and combined with the angle sensor to obtain the wrist and ten The bending angle A _j of the finger (j=0~10) models the hand shape through fuzzy gestures and precise angles, thereby accurately predicting gesture movements;

S3: Remote information transmission: When the virtual terminal is mapped to the real terminal, the pose coordinate B of the manipulator elbow is converted from the direction D through the coordinate system, involving formula (1). During online operation, the virtual terminal needs to transmit to the real terminal The form of instruction m is @_t_D_G ¹ _i _G ² _i _E_A _j _#, where @ and # are the start and end characters, t is the time, G ¹ _i and G ² _i respectively represent the gesture categories of the left and right hands, and are transmitted remotely using the SCP protocol and through RSA The algorithm encrypts instructions, such as formula (2);

in, is the robot elbow coordinate B,/> are the elbow IMU direction D, representing acceleration and angular velocity respectively, R is the rotation matrix, and T is the translation matrix;

c＝m ^e mod N (2)

Among them, c is the encrypted instruction, m is the plaintext instruction before encryption, and the key pair (e, N) is randomly generated;

S4: Controlled operation of the real end: A dexterous five-finger manipulator is set up in front of the experimental bench to follow the experimental operation. The manipulator adopts a six-axis vertical 14-joint structure. After receiving the instructions transmitted remotely from the virtual end, it performs operations according to the decrypted elbow position B of the manipulator. Continuous trajectory control, decoding the hand model and controlling the attitude of the manipulator according to the gesture category G and the finger bending angle A _j , and obtaining the actual position C through the depth image obtained by the binocular camera, according to the current position C and the target position B The error between is adjusted using PID controller, such as formula (3);

S5: Experimental feedback and evaluation: Feedback includes operation feedback and effect feedback. The operation feedback mechanism uses intelligent skin covering on the ten ends of the manipulator to remotely transmit several physical quantities such as temperature T _S , pressure P and humidity H to the touch generation of the virtual end. The device provides real-time touch feedback to the user. The effect feedback mechanism is to perform target recognition on the depth video captured by the binocular camera, and combine the physical quantities transmitted by gas sensors, electrical signal sensors, etc. to feed back to the virtual display module and introduce it to the equipped evaluation system. Medium; The evaluation system monitors the operation and experimental effects of each step, and conducts final evaluation of the experiment according to the set scoring mechanism.

A virtual remote experiment system and method based on bioelectricity sensing of the present invention has the following advantages:

1. Experimenters do not need to rely on offline physical laboratories. Remote experiments not only improve learning efficiency, but also save time and effort and improve space utilization.

2. The three-dimensional holographic display provides the experimenter with a real experience as if he is in a real laboratory, provides all-round visual and auditory feedback, and creates an immersive experimental atmosphere.

3. By extracting electromyographic signal features to recognize gestures, advance prediction is achieved and further improves remote efficiency.

4. Efficient information encryption transmission not only enables rapid response and synchronized experiments, but also improves security.

5. The multi-sensor setting adds the function of operation and effect feedback, allowing users to grasp the laboratory's action response in a timely manner and obtain an immersive operating touch.

6. It truly realizes "visible and controllable" remote operation, which greatly facilitates experimenters to fully control the laboratory online.

7. Using a robot to follow the experiment can effectively protect the user's personal safety and avoid life safety problems caused by experimental errors.

8. For some experiments with a long period of time, there is no need to wait for a long time. The alarm system can be set up through the surveillance camera to remotely notify the user, which greatly improves time efficiency.

Description of the drawings

Figure 1 is a block diagram of the equipment of a remote experimental system based on bioelectric signals;

Figure 2 is a block diagram of the myoelectric sensing module;

Figure 3 is a schematic diagram of the remote following experimental operation based on bioelectric signals;

Figure 4 is a network diagram of continuous gesture recognition and prediction algorithm based on bioelectric signals;

Figure 5 is the performance evaluation and teaching flow chart of the remote experiment system.

Detailed ways

In order to better understand the purpose, structure and function of the present invention, a virtual remote experiment system and method based on bioelectrical sensing of the present invention will be described in further detail below in conjunction with the accompanying drawings.

A virtual remote experiment system based on bioelectricity sensing. The system includes a myoelectric sensing module, a virtual display module, a remote information module, a control machine module, and an environment sensing module;

The myoelectric sensing module is used to collect the user's arm electromyographic signals, process and identify them, and output accurate hand posture modeling results in real time. The electromyographic sensing module includes an electromyographic signal acquisition device, an inertial measurement unit, an angle sensor, a microprocessor, a touch generator and a wireless communication module. The electrical signal acquisition device, the inertial measurement unit, and the angle sensor are connected to the microprocessor. The processor is connected to the touch sensor generator, and the wireless communication module is connected to the microprocessor. The inertial measurement unit is installed at the center of the experimenter's elbow; the angle sensor is installed on the finger; the touch generator is installed on the fingertips of the experimenter's hands; the touch generator includes a micro heater, a vibrating plate, and a micro humidity generator. The myoelectric signal acquisition device is used to collect muscle electrical signals; the inertial measurement unit is used to measure the inertia of the moving joints; the angle sensor is used to measure the angular direction of the finger; the microprocessor is used to process the signals of each module; the wireless communication module is used to transmit signals . The electromyographic signal acquisition device includes an 8-channel surface electrode, a high-pass filter circuit, a differential amplifier, a low-pass filter circuit, and an analog-to-digital converter. The 8-channel surface electrode is connected to the high-pass filter circuit, the high-pass filter circuit is connected to a differential amplifier, and the differential amplifier is connected to the low-pass filter. circuit, the low-pass filter circuit is connected to the analog-to-digital converter. The 8-channel electromyographic signal extracts preliminary features through wavelet transform, mines deep features through autoencoders, and obtains time domain features through the long short-term memory (LSTM) model. Finally, continuous gesture classification is achieved, and accurate predictions are made based on finger angle sensor data.

The virtual display module includes holographic imaging glasses, earphones and a wireless communication module; it is used to display the laboratory video picture to the user in three dimensions.

The remote information module is a processor and wireless communication module loaded with control, encryption and evaluation algorithms; used for information processing and communication between the virtual end and the real end, and for experimental evaluation;

The control machine module is at least one set of manipulator system devices equipped with smart skin, which can sense touch parameters such as material, hardness, weight and pressure of experimental equipment in real time. The control machine module is used to perform the same actions following the gestures recognized by myoelectricity, and uses the smart skin to obtain the touch parameters of the experimental supplies;

The environment sensing module is at least a system including a multi-directional binocular stereo camera, a microphone, various sensors for measuring temperature, concentration, light intensity and other parameters, and a wireless communication module; the environment sensing module is used for multiple The orientation senses the environmental parameters of the laboratory and provides operational response and effect feedback.

The working process of the virtual remote experiment system based on bioelectricity sensing is as follows. The environment sensing module captures the real-time video of the laboratory and wirelessly transmits it to the virtual display module. The three-dimensional display is displayed to the user. The user conducts experiments based on the real-time three-dimensional picture. , the myoelectric sensing module collects the user's hand posture during movements for accurate hand modeling. After being encoded and encrypted by the remote information module, it is wirelessly transmitted to the control machine module. The machine device in it performs specified actions according to the decrypted instructions. The touch information received by the smart skin is transmitted back to the myoelectric sensing module to give the user a response. At the same time, the effect information of the experiment is fed back to the virtual display module in real time, and the experiment is evaluated through the target recognition of the remote information module.

A virtual remote experimental method based on bioelectricity perception not only includes the forward information transfer process, but also includes the reverse sensory feedback process. The specific steps are as follows:

(1) Holographic imaging: The binocular camera in the environmental sensing module is fixed above the manipulator device to capture the depth map of the experimental platform, and transmits it to the virtual display module in real time through wireless communication. The holographic imaging technology is used to perform real-time three-dimensional imaging of the video. It is displayed to the user through holographic glasses, and the user performs virtual experimental operations based on the holographic image they view;

(2) Hand action recognition: When offline, the myoelectric sensing module acquires the action electrical signals of the arm muscles, and after filtering, denoising, rectifying and smoothing the circuit, it uses wavelet transform to extract a variety of features, and builds a pattern recognition network for training. During online operation, the predicted muscle strength E and gesture categories G ¹ _i , G ² _i (i=1~40) are output. At the same time, an inertial measurement unit (IMU) is installed in the center of the elbow to record the movement direction D, and is obtained by combining with the angle sensor. The bending angle A _j of the wrist and ten fingers (j=0~10) models the hand shape through fuzzy gestures and precise angles to accurately predict gesture movements;

(3) Remote information transmission: When the virtual terminal is mapped to the real terminal, the pose coordinate B of the manipulator elbow is converted from the direction D through the coordinate system, involving formula (1). When operating online, the virtual terminal needs to transmit to the real terminal The form of the instruction m is @_t_D_G ¹ _i _G ² _i _E_A _j _#, (@ and # are the start and end characters, t is the time, G ¹ _i and G ² _i represent the gesture categories of the left and right hands respectively), and is transmitted remotely using the SCP protocol. Encrypt instructions through RSA algorithm, such as formula (2);

in, is the robot elbow coordinate B,/> are the elbow IMU direction D, representing acceleration and angular velocity respectively, R is the rotation matrix, and T is the translation matrix;

c＝m ^e mod N (2)

Among them, c is the encrypted instruction, m is the plaintext instruction before encryption, and the key pair (e, N) is randomly generated;

(4) Controlled operation of the real end: A dexterous five-finger manipulator is set up in front of the experimental bench to follow the experimental operation. The manipulator adopts a six-axis vertical 14-joint structure. After receiving the instructions remotely transmitted from the virtual end, according to the decrypted elbow position B of the manipulator Perform continuous trajectory control, decode the hand model and control the attitude of the manipulator according to the gesture category G and the finger bending angle A _j , and obtain the actual position C through the depth image obtained by the binocular camera. According to the current position C and the target position B The error between them is adjusted using PID controller, such as formula (3);

(5) Experimental feedback and evaluation: Feedback includes operation feedback and effect feedback. The operation feedback mechanism uses intelligent skin covering on the ten ends of the manipulator to remotely transmit several physical quantities such as temperature T _S , pressure P and humidity H to the touch sense of the virtual terminal. Generator to give users touch feedback in real time. The effect feedback mechanism is to perform target recognition on the depth video captured by the binocular camera, and combine the physical quantities transmitted by gas sensors, electrical signal sensors, etc. to feed back to the virtual display module and introduce it to the equipment evaluation. in the system. The evaluation system monitors the operation and experimental effects of each step, and conducts final evaluation of the experiment according to the set scoring mechanism.

Embodiment 1: Using the virtual remote experiment system based on bioelectric signals of the present invention to conduct basic experiments

As shown in Figure 1, the remote experiment system in this embodiment includes holographic glasses, headphones, myoelectric sensing module, touch generator, microprocessor, five wireless communication modules, smart skin, seven-degree-of-freedom manipulator system, dual camera, microphone and multi-directional sensors. As shown in Figures 2 and 3, the myoelectric signal acquisition device of the myoelectric sensing module is fixed on the experimenter's upper arm, the touch generator is fixed on the fingertips of the experimenter's hands, and the microprocessor is placed in the holographic glasses. , five wireless communication modules are respectively placed on the electromyographic signal collection device of both hands, the touch generator of both hands, the holographic glasses and the laboratory manipulator device. As shown in Figure 3, 1 is the myoelectric sensing module, 2 is the myoelectric collection electrode, 3 is the inertial measurement unit, 4 is the angle sensor, 5 is the touch generator, 6 is the binocular camera, and 7 is the manipulator.

The built-in binocular camera in the laboratory captures the depth video of the experimental platform and transmits it to the holographic glasses in real time. The user performs experimental operations according to the three-dimensional image. The myoelectric signal acquisition device on the arm sends the preprocessed muscle electrical signal to the remote information Accurate hand posture modeling is carried out in the processor of the module. The gesture recognition network used is shown in Figure 4. The 8-channel electromyographic signal is extracted from the preliminary features through wavelet transformation, and the deep features are mined through the autoencoder. Through long and short-term memory (LSTM) model obtains time domain features, ultimately achieves continuous gesture classification, and combines finger angle sensor data for accurate prediction. After encrypted wireless transmission in the processor to the manipulator, the manipulator responds according to the converted coordinates and decryption instructions to achieve remote control experiments. After the depth video obtained by the binocular camera is remotely fed back to the microprocessor, target recognition is performed frame by frame, and the YOLOv4 (You Only Look Once version 4) algorithm is used to identify the number of the watch needle, the color of the reagent or the placement of the experimental equipment, etc., and finally the results are Input into performance evaluation system.

Example 2: Performance evaluation of virtual remote experimental system based on bioelectrical signals

This embodiment uses the same remote experiment collection device as Embodiment 1, in which the microprocessor is used for holographic image generation and instruction encryption generation, information transmission, and evaluation systems. As shown in Figure 5, the specific steps to complete the experiment are as follows:

(1) Equipment readiness calibration: The manipulator returns to its position. The experimenter makes a classic gesture to observe the follow-up sensitivity of the manipulator before the experiment. When making a gesture indicating the start, the timer starts the total time of the experiment;

(2) Experimental operation: When performing experimental actions, if a malfunction occurs and the manipulator fails to respond in time, the yellow light will flash. At this time, the manipulator system turns on the automatic teaching function and repeats the previous step. If the recognition algorithm detects that the current operation If the risk coefficient is higher than the threshold, a red light alarm will be triggered and safe operations will be automatically performed;

(3) Experiment recording and judgment: Under the premise that the operation responds normally and no dangerous operations occur, the gesture recognition results and the binocular camera monitoring results are combined into a text record experimental report. When a gesture indicating termination is detected, the manipulator returns bit, the experiment is over;

(4) Score calculation and evaluation: Target detection is performed on images acquired by binocular cameras to extract effect information, including indications of various meters, objects grabbed by robots, color and volume changes caused by experiments, etc., and there is sound at the same time , temperature, gas and other multi-sensor assisted monitoring, and the evaluation system automatically gives a comprehensive score after the experiment;

(5) Correction teaching of the experiment: If the comprehensive score is not full score, the evaluation system will automatically call up the error correction mechanism, compare each step with the correct operation in the database, point out the wrong steps and demonstrate the correct operation method, and finally the equipment will stop and return to the original state. Bit.

It is understood that the present invention has been described through some embodiments. Those skilled in the art know that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, the features and embodiments may be modified to adapt a particular situation and material to the teachings of the invention without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed here, and all embodiments falling within the scope of the claims of the present application are within the scope of protection of the present invention.

Claims (10)

1. The virtual remote experimental system based on bioelectricity perception is characterized by comprising a myoelectricity sensing module, a virtual display module, a remote information module, a control machine module and an environment sensing module;

the myoelectricity sensing module is used for collecting, processing and identifying arm myoelectricity signals of a user and outputting accurate hand gesture modeling results in real time;

the virtual display module is used for displaying the video pictures of the laboratory to a user in three dimensions;

the remote information module is used for information processing and communication of the virtual end and the real end and experimental evaluation;

the control machine module is used for performing the same action along with the gesture recognized by myoelectricity and acquiring touch parameters of the experimental product by using intelligent skin;

the environment sensing module is used for sensing environmental parameters of a laboratory in multiple directions and giving operation response and effect feedback.

2. The virtual remote experimental system based on bioelectrical sensing according to claim 1, wherein the myoelectricity sensing module comprises an myoelectricity signal acquisition device, an inertia measurement unit, an angle sensor, a microprocessor, a touch sensor and a wireless communication module, wherein the electrical signal acquisition device, the inertia measurement unit and the angle sensor are connected with the microprocessor, the microprocessor is connected with the touch sensor, and the wireless communication module is in communication connection with the microprocessor; the inertial measurement unit is installed in the center of an elbow of an experimenter; the angle sensor is arranged on the finger; the touch feeling generator is arranged at the abdomen of the two fingers of the experimenter; the myoelectric signal acquisition device is used for acquiring myoelectric signals; the inertia measurement unit is used for measuring the inertia of the motion joint; the angle sensor is used for measuring the angle direction of the finger; the microprocessor is used for processing signals of all the modules; the wireless communication module is used for transmitting signals.

3. The virtual remote experimental system based on bioelectrical sensing as claimed in claim 2, wherein the tactile sensor comprises a micro-heater, a vibration plate, a micro-humidity generator.

4. The virtual remote experimental system based on bioelectrical sensing as claimed in claim 2, wherein the electromyographic signal acquisition device comprises an 8-channel surface electrode, a high-pass filter circuit, a differential amplifier, a low-pass filter circuit and an analog-to-digital converter, wherein the 8-channel surface electrode is connected with the high-pass filter circuit, the high-pass filter circuit is connected with the differential amplifier, the differential amplifier is connected with the low-pass filter circuit, the low-pass filter circuit is connected with the analog-to-digital converter, the electromyographic signal of the 8-channel is subjected to wavelet transformation to extract preliminary characteristics, the deep characteristics are mined through a self-encoder, the time domain characteristics are obtained through a long-short-term memory model, continuous gesture classification is finally realized, and accurate prediction is performed by combining finger angle sensor data.

5. The virtual remote experimental system based on bioelectrical perception according to claim 1, wherein the virtual display module comprises holographic imaging glasses, headphones and a wireless communication module.

6. The virtual remote experimental system based on bioelectrical perception according to claim 1, wherein the telematics module is a processor and wireless communication module loaded with control, encryption and evaluation algorithms.

7. The virtual remote experimental system based on bioelectrical sensing as claimed in claim 1, wherein the control machine module is at least one set of manipulator system device provided with intelligent skin, and senses touch parameters of materials, hardness, weight and pressure of the experimental appliance in real time.

8. The virtual remote experimental system based on bioelectrical sensing according to claim 1, wherein the environmental sensing module is at least one system comprising a multi-azimuth binocular stereo camera, a microphone, a sensor for measuring parameters of temperature, concentration and light intensity, and a wireless communication module.

9. The virtual remote experiment system based on bioelectrical sensing according to claim 1, wherein the working process of the system is as follows, the environment sensing module captures real-time video of a laboratory and transmits the real-time video to the virtual display module in a wireless mode, the real-time video is displayed to a user in a three-dimensional mode, the user performs experiments according to real-time three-dimensional pictures, the myoelectric sensing module acquires hand gestures of the user during actions to perform accurate hand modeling, the remote information module codes and encrypts the hand gestures and then transmits the hand gestures to the control machine module in a wireless mode, the machine device performs specified actions according to decrypted instructions, touch information received by intelligent skin is reversely transmitted to the myoelectric sensing module to give responses to the user, meanwhile, effect information of the experiments is fed back to the virtual display module in real time, and the target recognition of the remote information module is used for performing experimental evaluation.

10. A method for controlling a virtual remote experimental system based on bioelectrical sensing as claimed in any one of claims 1 to 9, comprising a forward information transfer process and a reverse sensing feedback process, comprising the following specific steps:

s1: holographic imaging: the binocular camera in the environment sensing module is fixed above the manipulator device and used for capturing a depth image of the experiment table, the depth image is transmitted to the virtual display module in real time through wireless communication, a holographic imaging technology is adopted to carry out real-time three-dimensional imaging on a video, the video is displayed to a user through holographic glasses, and the user carries out virtual experiment operation according to the observed holographic image;

s2: hand motion recognition: when offline, the myoelectricity sensing module acquires action electric signals of arm muscles, and after circuit filtering, denoising, rectifying and smoothing, various characteristics are extracted by wavelet transformation, a mode recognition network is built for training, and when online operation, predicted muscle force E and gesture types are outputG group ¹ _i 、G ² _i (i=1 to 40), and at the same time, the elbow center is provided with an inertial measurement unit to record the movement direction D, and the bending angle a of the wrist and ten fingers is obtained by combining an angle sensor _j (j=0 to 10), modeling the hand morphology through the fuzzy gesture and the accurate angle, so as to accurately predict the gesture motion;

s3: remote information transmission: when the virtual end is mapped to the real end, the pose coordinate B of the elbow of the manipulator is converted from the direction D through a coordinate system, and relates to a formula (1), and when the robot is in online operation, the instruction m which is required to be transmitted to the real end by the virtual end is in the form of @ -t-D-G ¹ _i _G ² _i _E_A _j _#, where @ and # are start and stop symbols, t is time, G ¹ _i 、G ² _i Respectively representing gesture types of left and right hands, remotely transmitting by using SCP protocol, and encrypting the instruction by RSA algorithm, as shown in formula (2);

wherein ,for the robot elbow coordinate B, +.>The direction D of the elbow IMU is respectively represented by acceleration and angular velocity, R is a rotation matrix, and T is a translation matrix;

c＝m ^e mod N (2)

wherein c is an encrypted instruction, m is a plaintext instruction before encryption, and the key pair (e, N) is randomly generated;

s4: controlled operation of the real end: the smart five-finger manipulator is arranged in front of the experiment table to carry out following experiment operation, the manipulator adopts a six-axis vertical 14-joint structure, after receiving a command of virtual end remote transmission, continuous track control is carried out according to the decrypted manipulator elbow position and posture B, and the continuous track control is carried out according to gesture typesG and finger bending angle A _j Decoding a hand model, performing gesture control on a manipulator, acquiring an actual position C through a depth image obtained by a binocular camera, and adjusting by using a PID (proportion integration differentiation) controller according to an error between the current position C and a target position B, wherein the actual position C is shown in a formula (3);

s5: experimental feedback and evaluation: the feedback comprises operation feedback and effect feedback, wherein the operation feedback mechanism adopts intelligent skin coverage to ten ends of the manipulator to cover the temperature T _S The physical quantities of pressure P and humidity H are remotely transmitted to a touch sensor at a virtual end so as to be fed back to a user in real time, and an effect feedback mechanism is to perform target recognition on a depth video shot by a binocular camera, and the physical quantities transmitted by a gas sensor, an electric signal sensor and the like are combined to be fed back to a virtual display module and introduced into an equipped evaluation system; the evaluation system monitors the operation and experimental effect of each step, and finally evaluates the experiment according to the set scoring mechanism.