CN114743433B - Multi-channel alarm presenting method and device for simulating threats in flight training environment - Google Patents

Abstract

The invention provides a multi-channel warning presentation method and device for simulating threats in a flight training environment, and relates to the technical field of human-computer interaction and the related technical fields of multi-modal presentation. Calculate the user's field of view; according to whether the threat target is within the user's field of vision, divide the threat target into the threat target within the user's field of vision and the threat target outside the user's field of vision; Push and display threat information to the user; for threat targets outside the user's field of vision, use the audio-visual dual-channel presentation method to prompt the user with threat information. The present invention can present threat target information to the user in real time as the user's head position changes, improves the user's ability to perceive the environment situation, shortens the time for the user to discover and recognize the threat target, and improves the user's interaction in the simulated flight training process efficiency.

Description

Threat multi-channel warning presentation method and device in simulated flight training environment

technical field

The present invention relates to the technical field of human-computer interaction and the relevant technical field of multi-modal presentation, in particular to a multi-channel warning presentation method and electronic device for threatening targets in a simulated flight training environment.

Background technique

Simulated flight training is a complex human-computer interaction task, which has high requirements on the user's perception and processing capabilities. In the simulated flight training environment, users are faced with a variety of threats. If the threat warning system can prompt the user with information about the threat target through natural interaction, it will greatly help to improve the user's awareness of the surrounding situation and enable the user to quickly act. respond and take appropriate countermeasures to assist users in completing training tasks with high quality and efficiency. Threat target presentation technology has a wide range of applications in various scenarios. For example, during simulated flight training, the system can inform the user of the location, type, and threat level of the threat target by presenting threat target information such as enemy aircraft and enemy ships, and assist the user to grasp more comprehensive environmental situation information and improve the success of the training mission. rate and completion efficiency. For example, in a simulated driving training environment, the system can improve the user's comprehensive awareness of the road situation by presenting threatening targets that affect driving safety, such as adjacent vehicles and pedestrians with collision risks, and assist users to take corresponding safety measures. So as to achieve the purpose of safe driving. When presenting relevant threat information, the system needs to do so without affecting the user's ability to complete the primary task. Most existing threat target presentation methods use the method of directly presenting threat data, ignoring the overall perception centered on the user ontology, resulting in unnatural presentation of threat target information and easily causing high cognitive load for users.

At present, threat warning presentation methods can be divided into two categories. The first category is two-dimensional top view image presentation, and the second category is three-dimensional stereo model presentation. Among them, the two-dimensional top view image presentation method is based on the vehicle where the user is located as the center, and the view obtained by the top-down orthographic projection; while the three-dimensional threat presentation method is user-centered, where the threat is relative to the user. In the azimuth, the alarm information is presented. The two-dimensional top view threat presentation method has the following disadvantages: 1) The location information of the threat warning is lost: due to the need to present the threat information in the three-dimensional space on a two-dimensional plane, there are limitations in the presentation of part of the location information of the threat warning; 2) it has High cognitive load: Two-dimensional top-down images are used to present threat warning information. When users perceive the location of threat warning information, they use alien-centered spatial perception; while the main scene tasks of users are based on user ontology-centered spatial perception. For example, in simulated flight training, if the user uses the two-dimensional top view threat information presentation method to perceive the threat target, the user needs to convert the alien center space coordinates to the body center space coordinates, resulting in additional cognitive load. The three-dimensional threat presentation method has the following deficiencies: 1) The visible area of existing display devices is limited: virtual reality head-mounted display devices, augmented reality head-mounted display devices, vehicle-mounted or airborne head-up displays, etc. The effective display area of the device is too small to support the user to fully observe the environmental information. 2) Complex 3D model manipulation methods: Usually, the user needs to drag, zoom, and pan the 3D presentation model in order to fully and accurately recognize the warning information presented by the model, but it is difficult for the user to adjust it correctly while driving the vehicle. The rendering angle of the 3D model.

To sum up, the existing presentation methods and systems cannot fully reflect the current situation to the user, resulting in the user's perception of the situation is not comprehensive enough, and has a high cognitive load in the corresponding task scenario. How to design a user-centered adaptive threat target presentation method to provide users with a high-efficiency and low-load threat presentation system has become an urgent problem to be solved.

Contents of the invention

In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a multi-channel warning presentation method and electronic device for threatening targets in a simulated flight training environment, based on the user's spatial perception ability, fully utilizing the advantages of combining visual and auditory channels, It can present the threat target information to the user in real time with the change of the user's head posture, improve the user's perception of the external environment, shorten the time for the user to discover and recognize the threat target, reduce the user's cognitive load, and improve the user's experience in flight simulation. Interactive efficiency during training.

In order to achieve the above object, the technical scheme adopted in the present invention is as follows:

A method for presenting a multi-channel warning of a threat in a simulated flight training environment, comprising the following steps:

1) Calculate the user's field of view;

2) According to whether the threat target is located in the user's field of vision, the threat target is divided into the threat target within the user's field of vision and the threat target outside the user's field of view;

3) For threatening targets within the user's field of vision, push and display threat information to the user in the central area of the user's field of vision;

4) For threatening targets outside the user's field of view, an audio-visual dual-channel presentation method is used to prompt the user with threat information.

Further, the user's visual range is calculated according to the user's current head posture and visual range.

Further, the method for obtaining the current head posture of the user is: using the pose sensor in the head-mounted device to obtain six degrees of freedom posture parameters of the user's head in three-dimensional space, and the six degrees of freedom posture parameters include The X, Y, and Z three-axis position with the pose sensor as the coordinate origin, as well as the heading angle, pitch angle, and roll angle.

Further, the visible range refers to the maximum display range provided by the head-mounted display device used by the user.

Further, the central area of the user's field of view refers to a circular area delineated with the center of the user's field of view as the center and one-fifth of the height of the field of view as the radius.

Further, the audio-visual dual-channel presentation method is as follows: presenting one or more pointers indicating the orientation of the threat target to the user through the head-mounted display, and presenting a warning sound through the head-mounted stereo earphone, and the warning sound is passed through the sound Features represent the orientation and threat level information of a threat target.

Further, for multiple threatening targets outside the user's field of vision, the alarm prompts of the multiple threatening targets are played in turn through virtual surround sound.

Further, for the threat target within the user's field of view, it is displayed by a solid line icon, and displayed at the position of the threat target.

Further, for a threat target outside the user's field of vision, it is displayed by a dotted line icon, and the offset direction and distance of the actual position of the threat target relative to the current position are displayed by an arrow icon.

A multi-channel warning presentation device for simulating threats in a flight training environment, comprising a head posture acquisition device, a user input device, a graphic display device, an audio output device, a memory and a processor, the head posture acquisition device contains A head-mounted device for a position sensor, the graphic display device is a head-mounted display, the audio output device is used to output an alarm prompt sound, a computer program is stored in the memory, and the processor executes the computer program When implementing the steps of the above method.

Compared with prior art, the beneficial effect of the present invention is:

1) The present invention uses an auditory and visual dual-channel presentation method, fully mobilizes multiple sensory abilities of the user, can effectively help enhance the user's rapid discovery and recognition of threat targets outside the field of vision, improve the efficiency of completing training tasks, and reduce the user's time spent on Cognitive load during training.

2) The present invention provides natural and effective presentation schemes for threatening targets within and outside the visual range respectively. The two schemes described in the present invention can help users quickly distinguish the urgency of threatening targets, and at the same time have the ability to reduce the enhanced display Beneficial effects of interface complexity.

Description of drawings

Fig. 1 is a schematic diagram of a multi-channel warning presentation device for threats in a simulated flight training environment according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of an exemplary user's initial field of view adjustment interface.

Fig. 3 is a flowchart of a multi-channel warning presentation method for threats in a simulated flight training environment according to an embodiment of the present invention.

Fig. 4 is a schematic diagram of a multi-channel warning presentation device for threats in a simulated flight training environment according to another embodiment of the present invention.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings. It should be noted that the described embodiments are for the purpose of illustration only, and do not limit the scope of the present invention.

This embodiment discloses a multi-channel warning presentation method and device for simulating threats in a flight training environment, and provides a user-centered three-dimensional threat presentation method. The method perceives the user's field of view and head posture, and combines Threat information presented to users through multiple channels. For example, in a simulated flight training scenario, the threat at the rear left of the user's field of vision can be mapped to the left side of the user's field of vision, and a specially designed icon is used to indicate that the threat is not in the direction corresponding to the icon within the field of vision , but located in the left rear outside the visual range, the specific location of the threat is indicated under the presentation of more visual elements and auditory channels, so that the user can directly and naturally perceive the threat in the scene.

FIG. 1 shows a multi-channel warning presentation device for simulating threats in a flight training environment. The device 101 may be any device with computing capabilities, such as a personal computer, server, smart phone, game console, embedded computer, and the like. In this embodiment, the apparatus 101 is depicted as an onboard embedded computer in simulated flight training, in other embodiments, the apparatus 101 may be any suitable type of equipment.

The device 101 may have input and output functions such as touch, keyboard and mouse or other physical interaction interfaces, gestures, eye movement, voice, and other structures that allow communication between the device 101 and its users. In addition, the device 101 should have the function of collecting the user's head posture, which can be provided by the device 101, or be transmitted to the device 101 after being collected by other equipment; the device 101 should have a spatial audio output function, and this function can be provided by the device 101, or Via an external device such as a virtual surround sound capable sound card, headphones or surround sound speakers.

The graphic display device 102 may be any head-mounted device with a display function, including but not limited to: a head-mounted augmented reality or virtual reality screen, a large screen, a spherical projection screen, and the like. The threat information is presented in the user's current field of view 103, which is calculated by combining the user's head posture and the initial field of view. Among them, the icon of threat 104 is represented as outside the field of vision, indicating that the threat target is located in the upper left position; since the position of the threat 104 icon is obtained by the relative head posture mapping method, the actual position of the threat 104 is not in the threat 104 icon The location is represented by a dotted line. The arrow icon 105 is used to indicate the offset direction and distance of the actual location of the threat 104 relative to the current location. The threat 106 icon presents threats within the user's field of vision. The icon of the threat 106 represents an air threat, and the border of the icon is drawn with a solid line, indicating that the threat indicated by the threat 106 icon is in the corresponding position within the field of vision. In addition, the device 101 will play the audio alarms of the threats outside the field of view in turn. In the exemplary scenario shown in FIG. 1 , the virtual surround sound alarms corresponding to the threat 104 and the threat 107 will be played in turn.

Fig. 2 shows an exemplary user's initial field of view adjustment interface. The field of view 103 is determined by the upper left icon 201 , the upper right icon 202 and the lower icon 203 . The user adjusts the positions of the icon 201 , the icon 202 and the icon 203 through the input device of the above-mentioned device 101 , so that the visual range 103 matches the real visual range of the user.

Fig. 3 shows an exemplary process in which threat information is presented to a user by means of a multi-channel warning presentation method simulating threats in a flight training environment. Before turning to the description of FIG. 3, it should be noted that the flowchart of FIG. 3 is described as an example with reference to the construction shown in FIGS. 1 and 2, but the process of FIG. The scene shown in Figure 1 and Figure 2. Furthermore, the flowchart in Figure 3 shows examples where the processes are performed in the particular order indicated by the arrows, but the stages shown in these figures can be adapted in any order or in any combination or sub-combination.

At 301, the user's current head posture is acquired; at 302, the head posture information is combined with the user's initial field of view to obtain the user's real-time field of view. At 304, the threat warning information (block 303) is divided into two parts based on the user's real-time field of view (block 302), namely threats within the field of view (block 304) and threats outside the field of view (block 306). For threat information within the field of view (block 304), the icon is directly displayed in the corresponding direction within the user’s field of vision, that is, the original position is pushed and displayed (block 308); , voice, voice broadcast and other auditory channels to prompt (block 309). In addition, threats outside the field of vision are further visually monitored after being mapped to the user's field of vision by a mapping method based on the user's head posture (block 307). Presentation on (block 310).

That is to say, at block 308, the user can intuitively perceive the situation information within the range of his field of vision without the need to transform the spatial coordinates of alien perception to the spatial coordinates of proprioceptive perception, thus completing the process with a relatively low cognitive load. Situational awareness and assessment. After the coordinate mapping performed at block 307, the threats outside the user's field of view are mapped into the user's field of view, and push display on the visual channel (block 310), so that the user can make full use of the visual channel to perceive his field of view Threat information out of range without the need for additional visual searches, thereby reducing the associated load. In order to further reduce the cognitive load, block 309 presents the threat information outside the user's field of vision to the user through the auditory channel: through direct audio presentation, the spatial coordinate mapping in block 307 to the spatial mapping of threats outside the field of view has a negative effect on the user's perception bring the load.

Fig. 4 shows a multi-channel warning presentation device for simulating threats in a flight training environment according to another embodiment of the present invention. The device 400 includes one or more processors 404 , and one or more data storage components, that is, memory 405 . The device 400 may be any device with certain computing capabilities, such as a head-mounted mixed reality all-in-one device, an situational computer, a mobile notebook computer, a server, a tablet computer, a set-top box, an embedded computer, a smart mobile phone, and the like. Typical examples of processor(s) 404 are microprocessors, such as those found in personal computers, mobile laptops, servers, headsets, and mixed reality devices. The (one or more) data storage components 405 are components capable of short-term or long-term storage of data, such as including hard disks, removable disks, internal memory, cache memory, magnetic tape, etc. (one or more) data storage components are equipment scale storage Examples of media. The device 400 may include a user input device 407 and an output device 406, such as a wearable mixed reality all-in-one machine, which includes a user input device 407, such as gesture, eye movement, voice and other acquisition devices, and also includes an augmented reality screen and a dual-channel earphone As the output device 406; the device 400 can also be connected with one or more user input devices 407 and output devices 406 through external connection or modification, such as connecting a mouse and a keyboard as the user input device 407, and connecting a headset that includes headphones A personal computer with a virtual reality display as the output device 406 . It should be pointed out that the output device 406 mentioned above is only an example, and any one or more devices capable of displaying and sound output are within the scope of the subject of the present invention, including but not limited to earphones, stereos, etc., and head-mounted Augmented reality display, display screen, large screen, dome screen, etc. Likewise, the user input devices mentioned above are merely examples, and any one or more devices that can support user input are within the scope of the subject matter of the present invention, including but not limited to keyboards, mice, eye trackers, gesture acquisition equipment, wearable physiological information collection equipment, myoelectric collection equipment, brain-computer interface, etc.

Software computer programs utilized by apparatus 400 may be stored in data memory means(s) 405 and may be executed on processor(s) 404 . An example of such software is adaptive threat presentation software 403 that may implement some or all of the functionality described above in conjunction with Figures 1-3, but any type of software could be used.

It should be noted that the interfaces depicted in Figures 1 and 2, and the system architecture depicted in Figure 4 are just examples. Any suitable design can visually, audibly communicate the threat's current situational status, and adapt based on the user's head position. Any such designs are within the scope of the subject matter in this invention. For example, the interface described in the present invention is applied in the scene of simulated flight training.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that subject matter defined in the appended claims does not necessarily reside in the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example implementations of what is claimed. The scope of protection of the present invention should be defined by the claims.

Claims (6)

1. A multi-channel warning presentation method of threats under a simulated flight training environment, characterized in that, comprising the following steps: The user's field of view is calculated according to the user's current head posture and visual range; the acquisition method of the user's current head posture is: using the pose sensor in the head-mounted device to obtain 6 free positions of the user's head in three-dimensional space Degree attitude parameters, the six degrees of freedom attitude parameters include the X, Y, and Z three-axis positions with the attitude sensor as the coordinate origin, and the heading angle, pitch angle, and roll angle; According to whether the threat target is located in the user's field of vision, the threat target is divided into the threat target within the user's field of vision and the threat target outside the user's field of view; For the threat target within the user's field of vision, it is displayed by a solid line icon, and displayed at the position of the threat target, and the threat information is pushed to the user in the central area of the user's field of vision; For threat targets outside the user's field of view, they are mapped to the user's field of view through a mapping method based on the user's head posture, displayed with a dotted line icon, and the actual location of the threat target relative to the current location is displayed with an arrow icon The offset direction and distance of the azimuth, using the audio-visual dual-channel presentation method to prompt the user with threat information. 2. The multi-channel warning presentation method for threats in a simulated flight training environment according to claim 1, wherein the visible range refers to the maximum display range provided by the head-mounted display device used by the user. 3. The multi-channel warning presentation method of threats under the simulated flight training environment as claimed in claim 1, wherein the central area of the user's field of view refers to the center of the user's field of view as the center, and the fifth point of the field of view height range. One is a circular area delimited by a radius. 4. The multi-channel warning presentation method of threats under the simulated flight training environment according to claim 1, wherein the audio-visual dual-channel presentation method is: present one or more threat targets to the user through a head-mounted display The azimuth pointer presents a warning tone through the headset stereo earphone, and the warning tone represents the azimuth and threat level information of the threat target through the sound characteristics. 5. The multi-channel warning presentation method of threat under the simulated flight training environment as claimed in claim 4, characterized in that, for a plurality of threat targets outside the user's field of vision, the plurality of threat targets are played in turn by means of virtual surround sound alarm tone. 6. A multi-channel warning presentation device threatening under a simulated flight training environment, characterized in that it comprises a head position acquisition device, a user input device, a graphics display device, an audio output device, a memory and a processor, and the head The pose acquisition device is a head-mounted device containing a pose sensor, the graphic display device is a head-mounted display, the audio output device is used to output an alarm sound, and a computer program is stored in the memory, and the processing When the computer executes the computer program, the steps of the multi-channel warning presentation method for threats in the simulated flight training environment described in claim 1 are realized.

Images