KR102263227B1 - Method and apparatus for control unmanned vehicle using artificial reality with relative navigation information - Google Patents

Abstract

A method and apparatus for controlling an unmanned moving object based on augmented reality using relative navigation information are disclosed. An unmanned vehicle control method according to an embodiment includes the steps of: receiving navigation information of an unmanned vehicle; generating relative navigation information between the unmanned vehicle and a pilot based on the navigation information; The method includes: configuring an AR screen by reflecting the relative navigation information on a pilot image photographed in front by an image processing device; and outputting the AR screen to the image processing device.

Description

A method and apparatus for controlling an unmanned moving object based on augmented reality using relative navigation information {METHOD AND APPARATUS FOR CONTROL UNMANNED VEHICLE USING ARTIFICIAL REALITY WITH RELATIVE NAVIGATION INFORMATION}

The following embodiments relate to a method and apparatus for controlling an unmanned moving object based on augmented reality using relative navigation information.

Unmanned vehicles can be classified into unmanned aerial vehicles (UAVs), unmanned ground vehicles (UGVs), and unmanned underwater vehicles (UUVs) according to usage environments. The unmanned vehicle may perform a pre-entered mission, or a pilot may give control commands while looking directly at the aircraft from the outside.

While the unmanned vehicle is performing its mission, it is possible to receive information from the unmanned vehicle on the ground using the ground control system to understand the mission performance and to monitor the status of the unmanned vehicle. However, since the conventional ground control system is operated in a computer on the ground or inside a dedicated vehicle, it is impossible to see the aircraft directly and the screen output by the system at the same time.

In many cases, the pilot controls the aircraft while actually watching the aircraft, but due to the aforementioned drawback, there is a limit that the pilot cannot monitor the ground control system at the same time as the pilot.

In order to solve these shortcomings, the pilot is mainly in charge of the operation, and a dedicated monitoring person is placed next to him to perform the mission while notifying the pilot of the current situation. However, this method is also inefficient because it cannot deliver all the large amount of information required for operation of an unmanned moving vehicle, and it is difficult to respond immediately to the situation.

In addition, when the unmanned moving object moves away from the pilot, the pilot cannot visually identify the position or posture of the unmanned moving object, so intuitive control becomes difficult.

Embodiments may provide a technology capable of increasing user convenience by combining an HMD device supporting augmented reality with the use of an unmanned moving object and ground control equipment that manages it, so that a pilot can control the unmanned moving object and monitor the situation at the same time.

An unmanned vehicle control method according to an embodiment includes the steps of: receiving navigation information of an unmanned vehicle; generating relative navigation information between the unmanned vehicle and a pilot based on the navigation information; The method includes: configuring an AR screen by reflecting the relative navigation information on a pilot image photographed in front by an image processing device; and outputting the AR screen to the image processing device.

The method may further include configuring an FPV screen by reflecting the navigation information or the relative navigation information in an unmanned moving object image photographed in front of the unmanned moving object, and outputting the FPV screen to the image processing device. can

The generating may include generating the relative navigation information by processing the navigation information based on the positions, speeds, and postures of the pilot and the unmanned moving object.

The step of constructing the AR screen includes converting the position in the navigation coordinate system of the unmanned moving object into the image coordinate system of the image capturing device included in the image processing device, and mapping the converted position on the pilot image to cover a rectangular frame. The method may include displaying the unmanned moving object and displaying the relative navigation information on the lower right side of the AR screen.

The configuring of the AR screen may further include displaying the velocity vector of the unmanned moving object as an arrow.

The method may further include configuring the FPV screen in the lower left corner of the AR screen.

The method may further include switching any one of the AR screen and the FPV screen to a main screen based on the relative distance between the unmanned moving object and the pilot.

The method may further include switching any one of the AR screen and the FPV screen to a main screen based on whether the visibility vector between the unmanned moving object and the pilot is maintained.

An apparatus for controlling an unmanned vehicle according to an embodiment includes a memory storing instructions and a processor executing the instructions, and when the instructions are executed, the processor receives the navigation information of the unmanned vehicle. Receive, generate relative navigation information between the unmanned mobile body and the pilot based on the navigation information, and configure an AR screen by reflecting the relative navigation information in the pilot image photographed in front by the image processing device worn by the pilot, and , and output the AR screen to the image processing device.

The processor may configure an FPV screen by reflecting the navigation information or the relative navigation information in an unmanned moving object image photographed in front of the unmanned moving object, and output the FPV screen to the image processing device.

The processor may generate the relative navigation information by processing the navigation information based on the position, speed, and posture of the pilot and the unmanned moving object.

The processor converts the position in the navigation coordinate system of the unmanned moving object into the image coordinate system of the image capturing device included in the image processing device, maps the converted position on the pilot image, and covers the unmanned moving object with a rectangular frame, The relative navigation information may be displayed in the lower right corner of the AR screen.

The processor may display the velocity vector of the unmanned moving object as an arrow.

The processor may configure the FPV screen in the lower left corner of the AR screen.

The processor may switch any one of the AR screen and the FPV screen to the main screen based on the relative distance between the unmanned moving object and the pilot.

The processor may switch any one of the AR screen and the FPV screen to the main screen based on whether the visibility vector between the unmanned vehicle and the pilot is maintained.

The device may be implemented in at least one of the unmanned moving object, the image processing device, and a ground control system that manages the unmanned moving object and receives the navigation information.

1 shows an augmented reality-based unmanned mobile ground control system according to an embodiment.
2 is a diagram for explaining an operation of generating relative navigation information and displaying it on a screen.
3 shows an example of a screen provided by an augmented reality-based unmanned mobile ground control system to a user's AR device.
4 is a flowchart for explaining screen switching, and FIG. 5 is a diagram for explaining a criterion used for screen switching.
6 is a schematic block diagram of an apparatus for generating a steering image according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

The terms used in the examples are used for description purposes only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The terms are only for the purpose of distinguishing one element from another element, for example, without departing from the scope of rights according to the concept of the embodiment, a first element may be named as a second element, and similarly The second component may also be referred to as the first component.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 shows an augmented reality-based unmanned mobile ground control system according to an embodiment.

The unmanned moving object ground control system 10 includes an unmanned moving object 100 , a ground control system (GSC) 200 , a manipulating image generating apparatus 300 , an image processing apparatus 400 , and a manipulating apparatus 500 . includes The unmanned mobile ground control system 10 may refer to an augmented reality-based unmanned mobile ground control system.

The unmanned mobile vehicle 100 may include an unmanned aerial vehicle (UAV), an unmanned ground vehicle (UGV), an unmanned underwater vehicle (UUV), etc. depending on the usage environment. The unmanned moving object 100 may perform a mission input in advance, or a pilot may transmit a steering command through the control device 500 while directly looking at the unmanned moving object 100 from the outside.

While the unmanned mobile body 100 performs a mission, the ground control system 200 on the ground manages the unmanned mobile body 100 and receives information from the unmanned mobile body 100 to monitor the mission performance of the unmanned mobile body 100 . It is possible to identify and monitor the state of the unmanned moving object 100 .

The ground control system 200 may receive navigation information and an image of the unmanned moving object from the unmanned moving object 100 . The ground control system 200 may transmit navigation information to the manipulation image generating apparatus 300 . Also, the ground control system 200 may transmit an image of an unmanned moving object to the control image generating apparatus 200 .

The navigation information may be information related to the navigation of the unmanned mobile body 100 in flight. The navigation information may include attitude information, location information, speed information, and altitude information of the unmanned moving object 100 . In addition, the navigation information may further include information on the amount of energy remaining and the available flight time of the unmanned moving object 100 .

The unmanned moving object image may be an image captured by an image capturing device included in the unmanned moving object 100 . The unmanned moving object image may be an image of the front of the unmanned moving object 200 .

The pilot image generating apparatus 300 may generate relative navigation information between the pilot and the unmanned moving object 100 based on the navigation information and the pilot information. For example, the pilot information may include the position and attitude information of the pilot, and the relative navigation information may include information on the relative distance, relative speed, relative position, and the like between the pilot and the unmanned moving object 100 . The relative navigation information may refer to relative three-dimensional navigation information between the pilot and the unmanned vehicle 100 with respect to the pilot.

The control image generating device 300 implements augmented reality (Augmented Reality (AR)), virtual reality (Virtual Reality (VR)), mixed reality (Mixed Reality (MR)), or extended reality (eXtended Reality (XR)). can

The pilot image generating apparatus 300 may configure the AR screen by reflecting the relative navigation information on the pilot image. The pilot image may be an image captured by an image capturing device included in the image processing device 400 worn by the pilot. The pilot image may be an image taken from the front of the image processing apparatus 400 . The AR screen may be a third-person control screen.

The control image generating apparatus 300 may configure a First Person View (FPV) screen using the unmanned moving object image. Also, the manipulation image generating apparatus 300 may configure the FPV screen by reflecting the navigation information or the relative navigation information on the unmanned moving object image. The FPV screen may be a first-person control screen.

The manipulation image generating device 300 may transmit the AR screen and/or the FPV screen to the image processing device 400 .

The pilot may control the unmanned moving object 100 by using the control device 500 while looking at the unmanned moving object 100 by wearing the image processing apparatus 400 .

The image processing device 400 may implement augmented reality (Augmented Reality (AR)), virtual reality (Virtual Reality (VR)), mixed reality (Mixed Reality (MR)), or extended reality (eXtended Reality (XR)). have.

The image processing apparatus 400 may be implemented as a portable electronic device. For example, the image processing apparatus 100 may be implemented as a smart device. The smart device may include a head mounted display (HMD), smart glasses, a smart watch, and a smart band. That is, the image processing apparatus 100 may be a wearable device that a user can wear or is suitable for wearing.

The image processing apparatus 400 may output the relative navigation information or the AR screen and/or the FPV screen in which the navigation information is reflected to the pilot through the display included in the image processing apparatus 400 .

The pilot wears the image processing device 400 and while looking at the unmanned moving object 100 , the pilot uses the control device 500 to steer and monitor various situations at the same time. By utilizing the advantages of AR technology that can float various information on a real situation, the pilot can independently monitor various information necessary for the operation of the unmanned mobile body 100 without the help of external personnel without being obstructed by the operation. In addition, various interfaces for the pilot's convenience are displayed on the screen of the image processing apparatus 400 to enable safe and efficient mission performance.

In FIG. 1 , the manipulation image generating device 300 is illustrated as a separate device distinct from the ground control system 200 , the image processing device 400 , and the manipulation device 500 , but the present invention is not limited thereto. According to an example, the manipulation image generating device 300 may be implemented in the ground control system 200 , the image processing device 400 , or the manipulation device 500 .

FIG. 2 is a diagram for explaining an operation of generating relative navigation information and displaying it on a screen, and FIG. 3 shows an example of a screen provided by the augmented reality-based unmanned mobile ground control system to the user's AR device.

The manipulation image generating apparatus 300 may generate an AR screen and an FPV screen. The pilot image generating device 300 provides an AR screen so that the pilot can wear the image processing device 400 to monitor the various situations at the same time while using the control device 500 to control the unmanned moving object 100 while looking at the unmanned moving object 100 . and/or the FPV screen may be transmitted to the image processing apparatus 400 .

The AR screen may display relative navigation information along with the pilot image. In order to facilitate the identification of the unmanned moving object 100 with the naked eye when a plurality of unmanned moving objects 100 are operated or the unmanned moving object is too small, etc., it is difficult to identify the unmanned moving object 100 with a square display on the AR screen. ) can be indicated. In addition, the speed vector is indicated by arrow marks, and various information is displayed next to it so that it can be helpful for steering.

The manipulation image generating apparatus 300 may generate relative navigation information by processing the navigation information based on the three-dimensional position, speed, and posture of the pilot and the unmanned moving object 100 .

In order to display information (eg, relative navigation information) on the unmanned moving object 100 displayed on the AR screen, it is necessary to calculate at which point the unmanned moving object is in the image captured by the AR camera. To this end, in consideration of the relative position and attitude between the unmanned mobile body 100 and the pilot, the process of converting the navigation position information of the unmanned mobile body 100 calculated in the navigation coordinate system into the image coordinate system of the image processing device 400 of the pilot is required. Do.

)를 화상 좌표계상의 점

로 변환시키는 과정은 homogeneous 좌표로 다음 수학식 1과 같이 나타낼 수 있다.As shown in FIG. 2 , the unmanned moving object 100 may be projected onto the image plane of the camera. The position of the unmanned moving object 100 calculated on the navigation coordinate system (

) to a point on the image coordinate system

The process of transforming into ? can be expressed as the following Equation 1 in homogeneous coordinates.

행렬은 영상 처리 장치(400)에 포함된 영상 촬영 장치(예를 들어, 카메라)의 파라미터 설정에 따라 결정되는 행렬로서,

는 초점 거리,

는 주점,

는 비대칭 계수일 수 있다.

행렬은 항법 좌표계를 영상 처리 장치(400)의 화상 좌표계로 변환시키기 위한 회전/병진이동변환 행렬일 수 있다.here

The matrix is a matrix determined according to parameter settings of an image capturing apparatus (eg, a camera) included in the image processing apparatus 400 ,

is the focal length,

is a tavern,

may be an asymmetry coefficient.

The matrix may be a rotation/translation transformation matrix for converting the navigation coordinate system into the image coordinate system of the image processing apparatus 400 .

The control image generating apparatus 300 processes the relative navigation information based on the three-dimensional position/velocity/position of the pilot and the unmanned moving object 100 and converts the relative navigation information into valid information in the two-dimensional image coordinate system, thereby converting the relative navigation information to the AR screen. It can be provided as effective flight assistance information to

The control image generating apparatus 300 maps the position of the unmanned moving object 100 on the pilot image to increase the identification of the aircraft by covering the rectangular frame, and shows an arrow in the moving direction of the unmanned moving object 100 to the image processing apparatus 400 . By displaying the velocity vector of the unmanned moving object 100 as an arrow on the two-dimensional coordinates of the included image photographing device, it is possible to increase the maneuverability. The pilot image generating apparatus 300 may be provided as collision avoidance information to the pilot by generating relative speed information with an obstacle such as a nearby drone or a building using the two-dimensional position and speed information. In addition, the manipulation image generating apparatus 300 may display a relative distance and/or relative speed at which the unmanned moving object 100 and the pilot are approaching or moving away from each other on the screen.

The pilot image generating apparatus 300 may configure the AR screen so that detailed navigation information is displayed at the lower right of the AR screen output to the pilot. Detailed navigation information includes information including attitude indicator, aircraft position, speed, altitude, temperature, pressure, navigation mode, motor output, gas battery status, mission progress status, etc. It can enable detailed management of tasks.

In addition, the manipulation image generating apparatus 300 may configure the FPV screen so that an unmanned moving object image is displayed in the lower left corner of the AR screen. The pilot can also check the image of the unmanned moving object taken from a first-person view of the front of the unmanned moving object 100 through the AR screen.

The manipulation image generating apparatus 300 may also display the above-described relative navigation information on the FPV screen.

Through this, it is possible to operate the unmanned mobile body 100 with less manpower and the convenience of controlling the unmanned mobile body 100 can be improved. The pilot can control the unmanned moving object 100 while focusing on it, and at the same time, can check various mission situations implemented in augmented reality within the same field of view and aircraft information of the unmanned moving object 100 at a glance. In addition, in case the unmanned moving object 100 is out of view, the aircraft position is indicated on the unmanned moving object 100 with a square mark so that the pilot does not miss the aircraft, and various information is displayed next to it to help control do.

4 is a flowchart for explaining screen switching, and FIG. 5 is a diagram for explaining a criterion used for screen switching.

The manipulation image generating apparatus 300 may output by switching between the AR screen and the FPV screen.

When the unmanned moving object 100 departs from the pilot's field of view after the unmanned moving object 100 departs to perform the mission, the pilot is performing the mission rather than the screen captured by the image processing device 400 having the same field of view as the pilot. It may be more useful to view the FPV screen of the unmanned moving object 100 .

When the unmanned moving object 100 deviates from the visibility environment due to a predetermined relative distance or obstacle in consideration of the pilot's visibility situation, the pilot image generating device 300 switches the FPV screen displayed on the lower left to the main screen, The AR screen (eg, a pilot image) provided by the processing device 400 may be displayed in a small size in the lower left corner to provide the pilot with a screen more helpful in performing the mission.

In the case of the FPV screen, a virtual reality (VR) environment may be created as if the pilot was riding on the unmanned moving object 100 to increase the convenience of operation. When the unmanned mobile vehicle 100 comes into the pilot's visibility environment to return and land after completing the mission, the two screens may be switched back to each other to facilitate the landing procedure from the pilot's point of view. When the FPV screen is output as the main screen, the control image generating apparatus 300 may receive the unmanned moving object image directly from the unmanned moving object 100 to configure the FPT screen and output it.

The manipulation image generating apparatus 300 may perform the above-described screen switching when one is satisfied using two criteria.

For example, the relative distance based on the location information of the unmanned moving object 100 and the pilot may be a criterion for determination. The manipulation image generating apparatus 300 may perform screen switching when it deviates from a predefined relative distance.

As another example, whether to maintain a line-of-sight vector between the unmanned mobile vehicle 100 and the pilot in conjunction with the surrounding terrain map information may be a determination criterion. The pilot image generating apparatus 300 may perform switching by determining whether to maintain a visibility vector between the unmanned mobile vehicle 100 and the pilot based on the surrounding terrain map information.

6 is a schematic block diagram of an apparatus for generating a steering image according to an exemplary embodiment.

The manipulation image generating apparatus 300 includes a transceiver 310 , a processor 330 , and a memory 350 .

The transceiver 310 may perform communication between the unmanned moving object 100 , the ground control system 200 , and the image processing apparatus 400 and the manipulation image generating apparatus 300 .

The memory 350 may store instructions (or programs) executable by the processor 330 . For example, the instructions may include instructions for executing an operation of the processor 330 and/or an operation of each component of the processor 330 .

The processor 330 may process data stored in the memory 350 . The processor 330 may execute computer readable code (eg, software) stored in the memory 350 and instructions induced by the processor 330 .

The processor 330 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program.

For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

The processor 330 may control the overall operation of the video title identification apparatus 300 . For example, the processor 330 may control the operation of each of the components 310 and 350 of the manipulation image generating apparatus 300 .

The processor 330 may perform the operations for generating relative navigation information, generating an AR screen, and generating an FPV screen, which have been described with reference to FIGS. 2 and 3 . Also, the processor 330 may perform the screen switching operation described with reference to FIGS. 4 and 5 .

That is, the processor 330 may substantially perform the operation of the manipulation image generating apparatus 300 described with reference to FIGS. 1 to 5 .

As described above, the embodiments can increase user convenience by combining an unmanned moving object and a device supporting augmented reality to the use of a ground control system that manages it, so that a pilot can control the unmanned moving object and monitor the situation at the same time. have.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

Receiving navigation information of an unmanned vehicle;
generating relative navigation information between the unmanned vehicle and a pilot based on the navigation information;
constructing an AR screen by reflecting the relative navigation information in a pilot image photographed in front by the image processing device worn by the pilot; and
outputting the AR screen to the image processing device
including,
The generating step is
converting the position of the unmanned moving object into image coordinates of the AR screen by transforming the position of the unmanned moving object based on a matrix based on a focal length and an asymmetry coefficient of an image capturing apparatus included in the image processing apparatus,
The configuring step is
Comprising the step of mapping the unmanned moving object on the pilot image based on the image coordinates,
An unmanned moving object control method.
According to claim 1,
constructing an FPV screen by reflecting the navigation information or the relative navigation information on an unmanned moving object image photographed in front of the unmanned moving object; and
outputting the FPV screen to the image processing device
An unmanned moving object control method further comprising a.
According to claim 1,
The generating step is
generating the relative navigation information by processing the navigation information based on the position, speed, and posture of the pilot and the unmanned moving object;
An unmanned moving object control method comprising a.
4. The method of claim 3,
The step of constructing the AR screen comprises:
converting the position on the navigation coordinate system of the unmanned moving object into an image coordinate system of an image capturing apparatus included in the image processing apparatus;
mapping the converted position on the pilot image and displaying the unmanned moving object by covering a rectangular frame; and
Displaying the relative navigation information on the lower right of the AR screen
An unmanned moving object control method comprising a.
5. The method of claim 4,
The step of constructing the AR screen comprises:
Displaying the velocity vector of the unmanned moving object as an arrow
An unmanned moving object control method further comprising a.
3. The method of claim 2,
Composing the FPV screen at the lower left of the AR screen
An unmanned moving object control method further comprising a.
7. The method of claim 6,
Switching any one of the AR screen and the FPV screen to the main screen based on the relative distance between the unmanned moving object and the pilot
An unmanned moving object control method further comprising a.
7. The method of claim 6,
Switching any one of the AR screen and the FPV screen to the main screen based on whether the visibility vector between the unmanned moving object and the pilot is maintained
An unmanned moving object control method further comprising a.
In the device for controlling an unmanned vehicle,
a memory for storing instructions; and
a processor that executes the instructions
including,
When the instructions are executed, the processor:
Receives the navigation information of the unmanned vehicle, generates relative navigation information between the unmanned vehicle and the pilot based on the navigation information, and adds the relative navigation information to the pilot image captured by the image processing device worn by the pilot in front. By reflecting, constructing an AR screen, outputting the AR screen to the image processing device, and transforming the position of the unmanned moving object based on a matrix based on the focal length and asymmetry coefficient of the image capturing device included in the image processing device Converting to image coordinates of the AR screen, and mapping the unmanned moving object on the pilot image based on the image coordinates
Device.
10. The method of claim 9,
The processor is
An apparatus for configuring an FPV screen by reflecting the navigation information or the relative navigation information on an unmanned moving object image taken in front of the unmanned moving object, and outputting the FPV screen to the image processing device.
10. The method of claim 9,
The processor is
An apparatus for generating the relative navigation information by processing the navigation information based on the position, speed, and posture of the pilot and the unmanned moving object.
12. The method of claim 11,
The processor is
The position on the navigation coordinate system of the unmanned vehicle is converted into the image coordinate system of the image capturing device included in the image processing device, the converted position is mapped on the pilot image, and a rectangular frame is covered to display the unmanned vehicle, and the relative navigation method A device that displays information in the lower right corner of the AR screen.
13. The method of claim 12,
The processor is
A device for displaying the velocity vector of the unmanned moving object as an arrow.
11. The method of claim 10,
The processor is
A device for configuring the FPV screen at the lower left of the AR screen.
15. The method of claim 14,
The processor is
An apparatus for switching any one of the AR screen and the FPV screen to a main screen based on the relative distance between the unmanned moving object and the pilot.
15. The method of claim 14,
The processor is
An apparatus for switching any one of the AR screen and the FPV screen to a main screen based on whether the visibility vector between the unmanned vehicle and the pilot is maintained.
10. The method of claim 9,
The device is
A device implemented in at least one of the unmanned moving object, the image processing device, and a ground control system that manages the unmanned moving object and receives the navigation information.

Images