CN107643695B - Human/unmanned aerial vehicle cluster formation VR simulation method and system based on electroencephalogram - Google Patents

Abstract

The invention relates to the development of a simulation platform in the aerospace field, in order to propose a simulation platform capable of supporting the verification of a manned/unmanned aerial vehicle swarm formation control algorithm. EEG-based VR simulation method and system for manned/unmanned aerial vehicle swarm formation, real-time simulation Simulink Real Time target machine is responsible for the real-time simulation function of the Matlab simulation program, and is connected to the simulation platform management computer through Ethernet, and the simulation platform management computer integrates the MATLAB program Downloaded to the target computer, the target computer runs the MATLAB program, and sends the simulation results to the simulation platform management computer in real time, and displays and saves data on the computer; each Simulink Real Time target computer + embedded controller represents a UAV; the embedded controller runs the UAV control algorithm and calculates the control signal. The present invention is mainly used in aerospace simulation occasions.

Description

EEG-based VR simulation method and system for manned/unmanned aerial vehicle swarm formation

technical field

The invention relates to a simulation platform development problem in the aerospace field, and is a simulation platform development problem for verifying a manned/unmanned aerial vehicle swarm formation control method. Specifically, it involves a VR simulation system of manned/unmanned aerial vehicle swarm formation based on EEG equipment.

Background technique

In modern air combat, with the increasingly complex battlefield environment and combat tasks, only a single UAV or manned aircraft will not be able to meet the operational needs. The coordinated operation of manned aircraft and UAVs will be an important form of future air combat. The decision-making assignment of manned/unmanned aerial vehicle coordination is to study the decision-making control level of manned aircraft and unmanned aerial vehicles in different situations. It combines the advantages of human experience, intuition, flexibility, etc. with the advantages of strong computer data processing ability and fast speed, so that the two can complement each other and achieve better decision-making results. The manned aircraft as the lead aircraft leads the unmanned aerial vehicle to fly in a mixed formation, which will not only greatly improve the overall efficiency of the system and the redundancy performance during mission execution, but also make up for the low recognition accuracy of the simple unmanned aerial vehicle formation in complex environments. This problem has become a new research hotspot in the development of manned/UAV hybrid formation and its application technology. At present, it has been carried out in many aspects such as formation control, cooperative operations, information interaction processing and specific implementation verification. Extensive research. With the continuous deepening of research, the dissimilarity of the set system operating environment and formation model has brought greater challenges to the verification methods and verification methods. Human resources, lengthy work cycles, and expensive experimental equipment and material consumption. Therefore, virtual simulation verification, as a flexible and low-cost verification method, is suitable for most theoretical pre-research.

The manned/unmanned aerial vehicle formation control system is different from the common space vehicle. The function of the system is completed by multiple aircrafts in cooperation with each other, so there is a complex information interaction. This puts forward new requirements for aircraft position and attitude control, aircraft task management and scheduling control. The method based on the integration of simulation models in a single field is not conducive to the expansion of the system, and the confidence of the system simulation results after integration is low. The model cannot be described by equations; the simulation system can be flexibly constructed based on the heterogeneous simulation environment, effectively improve the simulation efficiency, and integrate the existing research results; the distributed modeling and simulation scheme has certain flexibility and scalability, reducing network data Redundancy ensures maximum data consistency, and can integrate real simulation and virtual simulation into a unified simulation environment.

The manned/unmanned formation simulation platform adopts computer real-time simulation, embedded software and hardware technology and virtual reality technology to simulate and verify the control method of manned/unmanned formation. Its biggest feature is that the simulation model can be quickly and efficiently transplanted to the platform. And watch the simulation in real time in a virtual reality environment. In line with the application requirements and development trends of manned/unmanned formation simulation.

Through a search of the prior art, no similar patents were found. Especially for the real-time verification of manned/unmanned formation control algorithms, there is no effective simulation verification platform.

SUMMARY OF THE INVENTION

In order to overcome the deficiencies of the prior art, the present invention aims to provide a simulation platform capable of supporting the verification of the manned/unmanned aerial vehicle swarm formation control algorithm. The technical scheme adopted in the present invention is that a human/unmanned aerial vehicle cluster formation VR simulation system based on EEG, a real-time simulation Simulink Real Time target machine is responsible for the real-time simulation function of the Matlab simulation program, and is connected with the simulation platform management computer through Ethernet, and the simulation The platform management computer downloads the MATLAB program to the target computer, the target computer runs the MATLAB program, and sends the simulation results to the simulation platform management computer in real time, and displays and saves data on the computer; each Simulink Real Time target computer + embedded The type controller represents a UAV, and the target machine runs the UAV digital simulation simulink model; the embedded controller runs the UAV control algorithm, calculates the control signal, and transmits it to the target machine through the serial port to realize the UAV control. Circuit; EEG equipment + human-computer interaction device simulates a man-machine, the brain-electric device collects the experimenter's brain signal, converts it into the corresponding formation control command, simulates the operation of the man-machine pilot, and the human-computer interaction handle receives the human gestures, simulating the pilot's movement. The operation of the manned aircraft; the simulation platform management computer receives these two control signals and sends the manned aircraft commands to the control drone; the visual computer receives the position and attitude angle information generated by the simulation, and conducts real-time viewing on the large-screen display system. scene presentation.

The platform management software runs on the platform management computer, including a digital simulation module, a real-time simulation module, a data management module, a network communication module, a brain-computer interface module and a human-computer interaction module, as follows:

(1) Digital simulation module: establish a reusable digital simulation loop in the simulation software Matlab/Simulink environment, and divide the digital simulation loop into a manned/unmanned aerial vehicle model library, a formation library, and task assignment based on the idea of modularity There are five modules: method library, trajectory optimization method library and formation control method library. Among them, the manned/unmanned aerial vehicle model library provides four-rotor and hexa-rotor aircraft models, and the formation formation library provides information including attack formation data, reconnaissance team The database of formation data, cruise formation data and task assignment method provides dynamic and static task assignment algorithms including swarm algorithm, market algorithm and clustering algorithm; the trajectory optimization method library provides the trajectory including adaptive genetic algorithm and improved particle swarm algorithm. Optimization method; formation control method library provides formation control methods including PID, sliding mode, backstepping, in a reusable digital loop, select different models, formation formation, task allocation method, trajectory in different libraries Optimization method and control algorithm to generate digital simulation dlm file; in the process of loop modular design, the I/O interface for interaction between modules is unified, making it universal and providing convenience for subsequent development and expansion;

(2) Real-time simulation module: analyze the digital simulation model composed of the above five modules based on the Simulink Real Time function library, realize simulation data monitoring, and develop model connection, program download, start simulation, and end simulation function modules to manage real-time simulation of cluster formations The whole process; the real-time simulation data display is based on the drawing control Teechart to develop a curve drawing sub-module to realize the display of key data of real-time position information, attitude angle and speed of the manned/unmanned aerial vehicle formation;

(3) Data management module: Based on the function development kit API provided by the database Mysql, a data management sub-module is developed to realize the management function of real-time simulation data of a group of manned/unmanned aerial vehicles, including data storage, data modification, data deletion, Data comparison and data export function modules make full use of the data generated by the simulation process;

(4) Network communication module: develop a network communication module based on the user datagram protocol UDP, transmit the real-time simulation data of the manned/unmanned aerial vehicle formation to the visual presentation software, and drive the visual presentation software for dynamic display;

(5) Brain-computer interface module: It includes a signal acquisition sub-module and a signal conversion sub-module, and collects user EEG signals based on the EEG acquisition system Neuracle. Based on NeuroCube, a supporting software development unit based on EEG equipment, developed with Neuracle supporting software, converted into corresponding man-machine control instructions, and sent to the UAV controller in real time to complete the virtual flight verification of manned/unmanned aerial vehicles based on brain control;

(6) Human-computer interaction module: including gesture recognition sub-module. Based on the gesture recognition technology Orion, it recognizes the operator's actions and converts them into corresponding manned-aircraft control commands, controls the drones to fly in formation, conducts manned/unmanned coordinated flight simulation verification, and realizes the "human-in-the-loop" simulation.

The visual presentation software runs in the visual computer: design the three-dimensional virtual scene module, the human-computer interaction module and the network communication module, as follows:

(1) 3D virtual scene module: It includes the design and implementation of VR models, scenes and special effects. Through the 3D modeling software SketchUp and 3D max modeling, a manned/unmanned aerial vehicle model library including a variety of fixed wings and rotors is established, and The land scene library in the mountains, plains and forest environments is imported into the graphics development engine Unity to establish an application resource group. Based on the physics engine Physics, the explosion and fragmentation effects generated when the cluster formation strikes military targets is realized. Based on Unity, it is released across platforms. function to output the virtual scene to the VR head display device for display;

(2) Human-computer interaction module: It includes gesture recognition sub-module, user graphical interface GUI sub-module, and head tracking sub-module. Based on Orion gesture recognition technology, users can interact with the program through gestures in VR scene demonstrations. , Based on the 3D graphical interface library Hovercast, the GUI interface in the VR scene is designed to realize the user's viewing of simulation information, sand table demonstration, and scene configuration; based on the head tracking component OpenVR, the VR perspective switching function is realized;

(3) Network communication module: Develop network communication module based on UDP protocol, receive real-time simulation data of formation, drive the movement of manned/unmanned aerial vehicle in virtual scene, and demonstrate the process of formation and maintenance of manned/unmanned formation.

The EEG-based VR simulation method of manned/unmanned aerial vehicle swarm formation uses the Simulink Real Time target computer to be responsible for the real-time simulation function of the Matlab simulation program, and is connected to the simulation platform management computer through Ethernet, and the simulation platform management computer downloads the MATLAB program to the target. In the machine, the target machine runs the MATLAB program, and sends the simulation results to the simulation platform management computer in real time, and displays and saves data on the computer; each Simulink RealTime target machine + embedded controller represents a UAV, The target computer runs the UAV simulink model; the embedded controller runs the UAV control algorithm, calculates the control signal, and transmits it to the target computer through the serial port to realize the UAV control loop; EEG equipment + human-computer interaction device simulates a human being The EEG device collects the brain signals of the experimenter, converts them into corresponding formation control instructions, simulates the operation of the man-machine pilot, and the human-computer interaction handle receives the gestures of the person, simulating the pilot's operation of the man-machine; the simulation platform management computer receives this Two control signals are sent to the man-machine command to control the unmanned aerial vehicle; the visual computer receives the position and attitude angle information generated by the simulation, and performs a real-time visual demonstration on the large-screen display system.

The characteristics and beneficial effects of the present invention are:

Social benefits: This invention is of great significance to the research on manned/unmanned aerial vehicle formation control systems. This invention is at the international advanced level. It can not only improve the simulation credibility in the early stage of the development of the manned/unmanned aerial vehicle formation control system, but also greatly shorten the research cycle; at the same time, it reduces the cost of platform development and supports the simulation of different formation control methods. The verification effectively improves the theoretical research level and simulation test level of manned/unmanned aerial vehicle formation control, and lays a good test platform foundation for the research and development of manned/unmanned aerial vehicle control system.

Economic benefits: The manned/unmanned aerial vehicle formation combines the advantages of manned and unmanned aerial vehicle operations, which can simplify combat missions, expand the observation field, and improve system robustness. The adaptability of the formation to the task has high economic value, and it has great potential applications in the military. The simulation and verification platform is designed for the manned/unmanned aerial vehicle formation control system, which can not only provide more comprehensive and complex simulation and verification functions for the future development of manned/unmanned aerial vehicle formation flight control systems; The rapid prototyping platform of the formation flight control system solves the problems of simulation optimization design and control strategy verification, which is fast, efficient and cost-saving.

Description of drawings:

Figure 1 is a schematic diagram of the hardware structure of the manned/unmanned aerial vehicle formation simulation platform.

Figure 2 is a flow chart of the co-simulation method of man-machine and unmanned aerial vehicle based on EEG equipment.

Figure 3 is a schematic diagram of the software structure of the platform management software.

Figure 4. Visual demonstration software structure diagram.

Fig. 5 Simulink Real Time programming realizes the flow chart.

Figure 6 Simulink Real Time target serial communication flow chart.

Detailed ways

The purpose of the present invention is to provide a simulation platform capable of supporting the verification of the manned/unmanned aerial vehicle swarm formation control algorithm.

The invention makes full use of the Simulink Real Time real-time simulation technology to construct a real-time simulation module, and is based on the Simulink Real Time host machine-target machine architecture. Advantages, it solves the problem that traditional digital simulation cannot adjust parameters online, and realizes fast C coding and real-time simulation of Matlab/Simulink simulation program. It effectively improves the theoretical research level and simulation test level of manned/unmanned aerial vehicle formation control, and lays a good test platform foundation for the research and development of manned/unmanned aerial vehicle formation collaborative control system.

The invention designs a manned and unmanned formation controller module based on the embedded controller PcDuino, adopts the Linux embedded operating system to support the development of multi-threaded tasks and application programs, simulates the communication between manned/unmanned formations through Wifi wireless network technology, and can be extended The communication topology is easy to reconstruct, and the hardware platform of the present invention supports the later development of various application programs.

The present invention designs a manned/unmanned formation virtual reality experience environment module based on the head display Htc Vive, and builds manned/unmanned formation virtual reality software based on Unity to realize the virtual reality experience of manned/unmanned formation simulation.

The invention designs a manned/unmanned formation brain control simulation experiment module based on the Neuracle brain control device, and converts the brain signals collected by the brain control device into corresponding control instructions. Simulate human-machine control commands to perform "human-in-the-loop" simulation.

The functions and features of the present invention are as follows:

(1) The simulation platform supports the real-time simulation function based on Simulink Real Time, that is, given the expected trajectory, attitude parameters, and power configuration of the manned/unmanned formation, it can calculate the manned/unmanned aerial vehicle formation at any simulation time and attitude data.

(2) The simulation system structure supports the simulation verification of the manned/unmanned aerial vehicle swarm formation control algorithm. The invention designs the flight controller module structure to support the compilation of the studied Simulink model into C code for real-time simulation verification.

(3) The simulation platform supports the visual simulation of manned/unmanned formations based on virtual reality VR technology, and the VR head-mounted display provides an immersive experience environment. Demonstration effect of flight vision of manned/unmanned formation.

(4) The simulation platform supports the co-simulation of man-machine and unmanned aerial vehicle based on EEG equipment, and uses EEG equipment to convert brain signals into electrical signals that can be recognized by computers, thereby simulating man-machine control instructions, and simulating man-machine control instructions through interactive equipment. Machine operation to achieve "human-in-the-loop" simulation.

The simulation platform of the present invention can improve the authenticity and credibility of the simulation, and through the combination of real-time simulation and the software and hardware of the controller, the real-time performance and portability of the system control strategy are enhanced, and the formation control of manned/unmanned aerial vehicles can be realized. The system development saves costs and lays the foundation for the industrialization of the domestic manned/unmanned formation control system.

Manned/UAV formation simulation and verification platform, the hardware includes real-time simulation module, flight control module, VR module, platform management module, brain control module, and large-screen display module. The software design mainly includes platform management software and visual VR simulation software.

The hardware design scheme of the present invention is as follows:

The invention is developed based on the Simulink Real Time real-time simulation technology. The Simulink Real Time real-time simulation target machine is mainly responsible for the real-time simulation function of the Matlab simulation program, and is connected to the simulation platform management computer through the Ethernet, and the simulation platform management computer downloads the MATLAB program to the target machine. In the MATLAB program, the target computer runs the MATLAB program, and sends the simulation results to the simulation platform management computer in real time, and displays and saves the data on the computer, which is convenient for later analysis and processing. Each Simulink Real Time target machine + embedded controller represents a UAV, and the target machine runs the UAV simulink model. The embedded controller UAV control algorithm calculates the control signal and transmits it to the target machine through the serial port to realize the UAV control loop. EEG equipment + human-computer interaction device simulates a man-machine. The EEG device collects the brain signals of the experimenter and converts them into corresponding formation control instructions, simulating the operation of the manned-aircraft pilot, and the man-machine interaction handle receives the human gestures, simulating the pilot's operation of the manned-aircraft. The simulation platform management computer receives these two control signals and sends the man-machine command to the UAV. The visual computer receives the position, attitude angle and other information generated by the simulation, and performs real-time visual demonstration on the large-screen display system.

The platform management software runs on the platform management computer, including a digital simulation module, a real-time simulation module, a data management module and a network communication module. The technical scheme is as follows:

(1) Digital simulation module: establish a reusable digital simulation loop in the Matlab/Simulink environment, and divide the digital simulation loop into a manned/unmanned aerial vehicle model library, a formation library, and a task assignment method library based on the idea of modularity , trajectory optimization method library and formation control method library five modules, built-in rich airframe model, formation formation and formation method. Among them, the manned UAV model library provides aircraft models such as quadrotors and hexacopters, the formation formation library provides formation data such as attack formation, reconnaissance formation, and cruise formation, and the task allocation method library provides information such as Dynamic and static task allocation algorithms such as swarm algorithm, market-based algorithm, clustering algorithm, etc. The trajectory optimization method library provides trajectory optimization methods such as adaptive genetic algorithm and improved particle swarm algorithm. The formation control method library provides formation control methods such as PID, sliding mode, and backstepping. In a reusable digital loop, different models, formations, task allocation methods, trajectory optimization methods and control algorithms are selected from different libraries to generate digital simulation dlm files. In the process of loop modular design, the I/O interface for interaction between modules is unified, making it universal and providing convenience for subsequent development and expansion.

(2) Real-time simulation module: Based on the Simulink Real Time function library, the digital simulation model composed of the above modules is analyzed, the simulation data is monitored, and functional modules such as model connection, program download, start simulation, and end simulation are developed to manage the real-time simulation of the cluster formation. The real-time simulation data display is based on the Teechart control to develop a curve drawing sub-module to realize the display of real-time position information, attitude angle, speed and other key data of the manned/unmanned aerial vehicle formation.

(3) Data management module: Based on the API provided by the Mysql database, a data management sub-module is developed to realize the management function of real-time simulation data of a group of manned/unmanned aerial vehicles, including data storage, data modification, data deletion, data comparison, Data export and other functional modules make full use of the data generated in the simulation process.

(4) Network communication module: The network communication module is developed based on the UDP protocol, and uses its advantages of low resource consumption and fast processing speed to transmit the real-time simulation data of the manned/unmanned aerial vehicle formation to the visual demonstration software, and drive the visual demonstration software. for dynamic display.

(5) Brain-computer interface module: including signal acquisition sub-module and signal conversion sub-module. Based on the Neuracle EEG acquisition system, the user's EEG signals are collected. Based on the NeuroCube unit, it is developed with Neuracle supporting software, converted into corresponding manned-machine control commands, and sent to the UAV controller in real time to complete the virtual flight verification of manned/unmanned aerial vehicles based on brain control.

(6) Human-computer interaction module: including gesture recognition sub-module. Based on Orion gesture recognition technology, it recognizes the operator's actions and converts them into corresponding manned-machine control commands, controls the drones to fly in formation, conducts manned/unmanned coordinated flight simulation verification, and realizes "human-in-the-loop" simulation.

The visual presentation software runs in the visual computer: design a three-dimensional virtual scene module, a human-computer interaction module, a network communication module and an EEG device module. The technical solutions are as follows:

(1) 3D virtual scene module: including the design and implementation of VR models, scenes and special effects. Through SketchUp and 3Dmax modeling, build a manned/unmanned aerial vehicle model library including a variety of fixed-wing and rotary-wing models, as well as land scene libraries in various environments such as mountains, plains, and forests, and import them into the Unity engine to create application resources. Group. Based on the Physics engine, the special effects of explosion and debris generated when the cluster formation strikes the military target are realized. Based on the function of Unity's cross-platform release, the virtual scene is output to the VR head display device for display.

(2) Human-computer interaction module: including gesture recognition sub-module, GUI interface sub-module, and head tracking sub-module. Based on the Orion gesture recognition technology, the user can interact with the program through gestures in the VR scene demonstration. Based on the Hovercast 3D graphical interface library, the GUI interface in the VR scene is designed to realize the user's viewing of simulation information, sand table demonstration, scene configuration and other functions; based on the OpenVR head tracking component, the VR perspective switching function is realized.

(3) Network communication module: Develop network communication module based on UDP protocol, receive real-time simulation data of formation, drive the movement of manned/unmanned aerial vehicle in virtual scene, and demonstrate the process of formation and maintenance of manned/unmanned formation.

The present invention will be further described in detail with reference to the accompanying drawings.

Referring to Figure 1, it is the hardware structure diagram of the manned/unmanned aerial vehicle control method simulation and verification platform, which mainly includes: the simulation master computer running the platform management software, the Simulink Real Time real-time simulation computer responsible for real-time simulation, and the formation control. The embedded controller, the visual computer responsible for visual demonstration, the perception element that provides human-computer interaction functions, the brain-controlled device that collects EEG signals, etc.

Simulation main control computer: run the simulation platform management software, responsible for downloading the formation configuration control software, Matlab/Simulink manned/unmanned aerial vehicle simulation model, data communication model, etc. to the Simulink Real Time real-time simulation computer, forming a distributed structure. /UAV formation control real-time simulation environment, and realize the functions of simulation process monitoring and database management of formation flight control system.

Simulink Real Time real-time simulation computer: Based on the DOS environment, a real-time simulation environment is constructed to simulate the manned/unmanned aerial vehicle formation model. The control model is run on the simulation machine, and the model information is sent to each formation controller. At the same time, the controller sends the calculated information to the real-time simulation computer. The two form a closed loop of manned/unmanned aerial vehicle control.

Embedded controller: Based on the Linux embedded operating system, it runs the manned/unmanned aerial vehicle position and attitude control module to realize the formation control of manned/unmanned aerial vehicles, receives the formation control instructions through the wireless Wifi communication module, calculates the control signal, and passes The serial port is passed to the corresponding simulated slave to realize the manned/unmanned aerial vehicle formation control loop.

Visual simulation computer: Based on Windows embedded operating system, it runs manned/unmanned aerial vehicle formation visual software, and puts video signals on the large screen and VR head-mounted display to display the manned/unmanned formation flight status in real time.

VR headset: Based on Htc Vive, it receives video signals of manned/unmanned formation flight, providing the observer with an immersive experience effect.

Interactive equipment: Gesture capture based on the capture device Leap Motion, collects human motion information, and converts it into human-machine operation instructions for virtual manned/unmanned aerial vehicle formation simulation interaction.

EEG equipment: including signal acquisition sub-module and signal conversion sub-module. Based on the neural EEG acquisition system, the user's EEG signals are collected. Based on the NeuroCube unit, it is developed with the neurocle supporting software, converted into corresponding control instructions, commanded the UAV, and completed the virtual simulation of manned/unmanned formation.

Referring to Figure 2, it is a flow chart of the co-simulation method of man-machine and unmanned aerial vehicle based on EEG equipment. The simulation process is started through the main control platform, the initialization conditions of the formation task are provided, and the real-time simulation environment of the master-slave structure is constructed; the simulation master control The computer runs the main control software, the EEG device collects the simulated man-machine commands and sends the control commands to the controller through wireless communication. The controller sends control signals to the simulated slave machine through the serial port through the simulation calculation, and the simulated target machine simulates the manned/unmanned aerial vehicle movement. The mechanical model and the environment model provide the state information of the manned/unmanned aerial vehicle and form a control loop with the UAV controller. The simulation data is sent back to the main control platform for unified management and analysis through the database. The main control computer sends the simulation position and attitude angle information to the visual VR platform for visual simulation.

Referring to Figure 3, it is a schematic diagram of the structure of the platform management software. The function of the platform management software is to realize the whole process of the manned/unmanned aerial vehicle formation control system from controller scheme design to offline simulation, real-time simulation, and finally simulation analysis. The technical layer of the simulation platform adopts a variety of technologies to support the realization of the simulation function. It uses the multi-threaded Matlab engine technology to support the simulation model developed based on Matlab/Simulink and the formation control software, and allows offline simulation in advance. Use the RTI driver provided by Matlab for Simulink Real Time target to complete the automatic conversion of the simulation program to the C language code, and realize the download of the real-time code. The simulation parameter analysis technology is used to obtain the corresponding relationship between the simulation parameter variables of the C code program and the simulation parameters written in Matlab, so that the simulation data information can be monitored in real time. Using database technology, both offline simulation data and real-time simulation data are saved in the data form corresponding to the database. and perform performance evaluation.

Referring to Figure 4, it is a structural diagram of the visual presentation software. The software is developed based on Unity and is divided into three modules. The specific implementation methods are as follows:

3D virtual scene module: Based on SketchUp, 3D max modeling and development resource station Unity AssetStore, Google3D Warehouse and other resources, establish a rich model library, scene library, and realize a variety of 3D special effects. The model library includes a variety of fixed-wing/rotor-wing manned/unmanned aerial vehicles, missiles, various military strike targets, buildings, etc. 3D models; the land scene library includes a variety of terrains, such as mountains, plains, rivers, forests, etc.; based on Unity The integrated Physics engine realizes the special effects of explosion and debris when hitting the target, and realizes the special effects such as tail flame and wake generated during flight based on the particle system Shuriken.

Human-computer interaction module: Leap Motion provides the Unity version of the gesture recognition software package Orion, and integrates it into the visual presentation software for secondary development. The realization of interactive functions includes the guidance prompt function, the graphical interface function, the simulation information viewing function, the viewing angle switching function, the sand table function, and the large-screen multi-perspective split-screen display function.

Network communication module: send user service requests to the main control unit, receive simulation data, and drive the operation of the visual demonstration program. Based on the TCP and UDP protocols, the application layer communication protocol with the host computer platform management software is designed. Based on the Socket socket API of the .NET version, the client communication program is developed to realize the packing of the sent data packets and the unpacking of the received data packets.

Referring to Fig. 5, it is the programming flow chart of the Simulink Real Time master control platform, using the Simulink Real Time API function library programming provided by Matlab to realize the control function of the Simulink Real Time target machine. First, the Simulink Real Time application needs to be registered with the application registration function to establish the communication connection between the main control platform and the Simulink Real Time simulator; then, the formation control software developed based on Matlab and the manned/unmanned aerial vehicle simulation model are loaded Go to the corresponding Simulink Real Time simulator; start real-time simulation, simulation monitoring and online parameter adjustment can be performed during the simulation process; judge whether the simulation is over, if not, enter the simulation monitoring loop, if the simulation is over, disconnect the host machine and the target The connection of the machine is completed to complete the simulation process. After the simulation is completed, the obtained real-time simulation data is saved in the database for later analysis and processing.

Referring to Figure 6, it is a flow chart of serial communication of Simulink Real Time target. The serial communication of Simulink Real Time simulation model is provided by Matlab\Simulink with serial communication module with FIFO buffer, reading serial data module according to the data header, packing The data frame module is composed of the unpacked data frame module. By judging the interrupt type and whether there is new data, enter the write buffer; search through the preset header of the data to determine whether the data is required, if it meets the requirements, output the data, otherwise continue to retrieve the data header.

Claims (3)

1. a kind of people/unmanned aerial vehicle swarm formation VR simulation system based on EEG, it is characterized in that, the real-time simulation SimulinkReal Time target machine is responsible for the real-time simulation function of Matlab simulation program, is connected with simulation platform management computer by ethernet, simulation platform The management computer downloads the MATLAB program to the target computer, the target computer runs the MATLAB program, and sends the simulation results to the simulation platform management computer in real time, and displays and saves data on the computer; each Simulink Real Time target computer + embedded The controller represents a UAV, and the target machine runs the UAV digital simulation simulink model; the embedded controller runs the UAV control algorithm, calculates the control signal, and transmits it to the target machine through the serial port to realize the UAV control loop ;The EEG device + human-computer interaction device simulates a man-machine, and the EEG device collects the experimenter's brain signal and converts it into the corresponding formation control command to simulate the operation of the man-machine pilot. The operation of the man-machine; the simulation platform management computer receives the two control signals and sends the man-machine command to the control UAV; the vision computer receives the position and attitude angle information generated by the simulation, and performs real-time viewing on the large-screen display system Demonstration; among them, the platform management software runs on the platform management computer, including a digital simulation module, a real-time simulation module, a data management module, a network communication module, a brain-computer interface module and a human-computer interaction module, as follows: (1) Digital simulation module: establish a reusable digital simulation loop in the simulation software Matlab/Simulink environment, and divide the digital simulation loop into a manned/unmanned aerial vehicle model library, a formation library, and task assignment based on the idea of modularity There are five modules: method library, trajectory optimization method library and formation control method library. Among them, the manned/unmanned aerial vehicle model library provides four-rotor and hexa-rotor aircraft models, and the formation formation library provides information including attack formation data, reconnaissance team The database of formation data, cruise formation data and task assignment method provides dynamic and static task assignment algorithms including swarm algorithm, market algorithm and clustering algorithm; the trajectory optimization method library provides the trajectory including adaptive genetic algorithm and improved particle swarm algorithm. Optimization method; formation control method library provides formation control methods including PID, sliding mode, backstepping, in a reusable digital loop, select different models, formation formation, task allocation method, trajectory in different libraries Optimization method and control algorithm to generate digital simulation dlm file; in the process of loop modular design, the I/O interface for interaction between modules is unified, making it universal and providing convenience for subsequent development and expansion; (2) Real-time simulation module: analyze the digital simulation model composed of the above five modules based on the Simulink Real Time function library, realize simulation data monitoring, and develop model connection, program download, start simulation, and end simulation function modules to manage real-time simulation of cluster formations The whole process; the real-time simulation data display is based on the drawing control Teechart to develop a curve drawing sub-module to realize the display of key data of real-time position information, attitude angle and speed of the manned/unmanned aerial vehicle formation; (3) Data management module: Based on the function development kit API provided by the database Mysql, a data management sub-module is developed to realize the management function of real-time simulation data of a group of manned/unmanned aerial vehicles, including data storage, data modification, data deletion, Data comparison and data export function modules make full use of the data generated by the simulation process; (4) Network communication module: develop a network communication module based on the user datagram protocol UDP, transmit the real-time simulation data of the manned/unmanned aerial vehicle formation to the visual presentation software, and drive the visual presentation software for dynamic display; (5) Brain-computer interface module: It includes a signal acquisition sub-module and a signal conversion sub-module. It collects user EEG signals based on the EEG acquisition system Neuracle, and is based on the NeuroCube unit, a supporting software development unit for EEG equipment. Corresponding man-machine control instructions are sent to the UAV controller in real time to complete the virtual flight verification of manned/unmanned aerial vehicles based on brain control; (6) Human-computer interaction module: It includes a gesture recognition sub-module, based on the gesture recognition technology Orion, recognizes the operator's actions, and converts them into corresponding manned-machine control instructions, controls the drones to fly in formation, and conducts manned/unmanned coordination Flight simulation verification to achieve "human-in-the-loop" simulation. 2. the human/unmanned aerial vehicle swarm formation VR simulation system based on EEG as claimed in claim 1, is characterized in that, visual presentation software runs in visual computer: design three-dimensional virtual scene module, human-computer interaction module and Network communication module, as follows: (1) 3D virtual scene module: It includes the design and implementation of VR models, scenes and special effects. Through the 3D modeling software SketchUp and 3D max modeling, a manned/unmanned aerial vehicle model library including a variety of fixed wings and rotors is established, and The land scene library in the mountains, plains and forest environments is imported into the graphics development engine Unity to establish an application resource group. Based on the physics engine Physics, the explosion and fragmentation effects generated when the cluster formation strikes military targets is realized. Based on Unity, it is released across platforms. function to output the virtual scene to the VR head display device for display; (2) Human-computer interaction module: It includes gesture recognition sub-module, user graphical interface GUI sub-module, and head tracking sub-module. Based on Orion gesture recognition technology, users can interact with the program through gestures in VR scene demonstrations. , Based on the 3D graphical interface library Hovercast, the GUI interface in the VR scene is designed to realize the user's viewing of simulation information, sand table demonstration, and scene configuration; based on the head tracking component OpenVR, the VR perspective switching function is realized; (3) Network communication module: Develop network communication module based on UDP protocol, receive real-time simulation data of formation, drive the movement of manned/unmanned aerial vehicle in virtual scene, and demonstrate the process of formation and maintenance of manned/unmanned formation. 3. A kind of human/unmanned aerial vehicle swarm formation VR simulation method based on EEG, it is characterized in that, utilize Simulink RealTime target machine to be responsible for the real-time simulation function of Matlab simulation program, through Ethernet and simulation platform management computer connection, simulation platform management The computer downloads the MATLAB program to the target computer, the target computer runs the MATLAB program, and sends the simulation results to the simulation platform management computer in real time, and displays and saves data on the computer; each Simulink Real Time target computer + embedded control The device represents a UAV, and the target computer runs the UAV simulink model; the embedded controller runs the UAV control algorithm, calculates the control signal, and transmits it to the target computer through the serial port to realize the UAV control loop; EEG The equipment + human-computer interaction device simulates a man-machine, and the EEG device collects the brain signals of the experimenter and converts them into corresponding formation control instructions, simulating the operation of the man-machine pilot, and the man-machine interaction handle receives the gestures of the man, simulating the pilot's response to the man-machine. Operation; the simulation platform management computer receives the two control signals and sends the man-machine command to the control UAV; the visual computer receives the position and attitude angle information generated by the simulation, and performs a real-time visual demonstration on the large-screen display system.

Images