KR102525912B1 - Unmanned Air Vehicle and Control Method Thereof - Google Patents

Abstract

The present invention relates to an unmanned aerial vehicle and a method for controlling the same, and the unmanned aerial vehicle according to the present invention includes an FPV camera for acquiring a first-person view front image, a 3D depth sensor for obtaining 3D depth map data, and a position and attitude data of the unmanned aerial vehicle. a position and attitude acquisition unit to acquire, a storage unit to store 3D depth map data and the position and attitude data in correspondence with an acquisition time, and a control unit to operate the unmanned aerial vehicle according to a manual control command transmitted from a remote control device in a first control mode; and a communication unit that transmits the FPV image, the acquisition time of the FPV image, and residual information on manual control commands that have not been processed in the unmanned aerial vehicle to the remote control device, and receives the manual control command from the remote control device.

Description

Unmanned air vehicle and its control method {Unmanned Air Vehicle and Control Method Thereof}

The present invention relates to an unmanned vehicle and a control method thereof, and more particularly, to an unmanned air vehicle supporting both a manual control mode and a semi-autonomous control mode and a control method thereof.

Unmanned aerial vehicles (UAVs), also known as drones, are becoming popular in the military and civil industries because they are most utilized for search, surveillance, and reconnaissance in places where manned flight is difficult or dangerous.

As a method for piloting a drone in an area where the drone cannot be seen visually, the FPV video recorded by the first person view camera (hereinafter referred to as the FPV camera) mounted on the drone is wirelessly transmitted and the operator checks it. A method of remotely controlling a drone using a remote control device while doing so is used. However, due to problems such as communication delay or video delay, it is difficult for an operator to issue an appropriate control command while viewing an image.

In the case of search, surveillance, and reconnaissance, it is often efficient for the operator to manually operate the drone while grasping the on-site situation through FPV video, so a plan to enable stable manual flight control while overcoming problems such as communication delay or video delay Research on this is actively progressing.

Therefore, the technical problem to be solved by the present invention is to provide an unmanned aerial vehicle and a control method thereof supporting both a manual control mode and a semi-autonomous control mode.

An unmanned aerial vehicle according to the present invention for solving the above technical problem is an FPV camera for acquiring a front-facing image from a first person perspective, a 3D depth sensor for obtaining 3D depth map data, and a location for acquiring position and attitude data of the unmanned aerial vehicle A posture acquisition unit, a storage unit that stores the 3D depth map data and the position and attitude data in correspondence with acquisition time, a control unit that operates the unmanned aerial vehicle according to a manual control command transmitted from a remote control device in a first control mode, and and a communication unit configured to transmit the FPV image, an acquisition time of the FPV image, and residual information on manual control commands not processed in the unmanned aerial vehicle to the remote control device, and to receive the manual control command from the remote control device.

The remote control device displays the FPV image and the unprocessed manual control command residual information on a screen, and the unmanned aerial vehicle operates in response to a semi-autonomous control command received from the remote control device in a second control mode, and the 3D Collisions with obstacles can be avoided using depth map data.

When the second control mode is selected, the remote control device checks the manual control command residual information, confirms that there is no unprocessed residual manual control command in the unmanned aerial vehicle, and then receives a semi-autonomous control command to receive a semi-autonomous control command. You can activate the input interface.

The semi-autonomous control command may be a command to autonomously fly the unmanned aerial vehicle while avoiding a collision with an obstacle in a specified moving target point or a specified direction.

The control unit may move the unmanned aerial vehicle in response to a semi-autonomous control command received from the remote controller, and avoid collision with an obstacle by using 3D depth data obtained in real time from the 3D depth sensor.

The command to move the target point may include 2D image coordinates of the moving target point selected from the FPV image displayed on the remote controller and an image acquisition time of the FPV image of the moving target point.

The controller extracts 3D depth map data corresponding to the image acquisition time of the FPV image selected at the moving target point and position and attitude data of the unmanned aerial vehicle from the storage unit, and the controller extracts 2D image coordinates of the moving target point, the extracted 3D position coordinates of the moving target point may be acquired based on 3D depth map data and position and attitude data of the unmanned aerial vehicle.

When a search target location confirmation command is received from the remote control device, 3D location information of the search target point may be checked and transmitted to the remote control device.

The search target point location confirmation command may include 2D image coordinates of a search target point selected from the FPV image displayed on the remote control device and an image acquisition time of the FPV image of the search target point selected.

The control unit extracts 3D depth map data corresponding to the image acquisition time of the FPV image selected at the search target point and position and attitude data of the unmanned aerial vehicle from the storage unit, and the control unit extracts 2D image coordinates of the search target point, the extracted The 3D location coordinates of the search target point may be confirmed using the 3D depth map data and the position and attitude data of the unmanned aerial vehicle.

A method for controlling an unmanned aerial vehicle according to the present invention for solving the above technical problem includes storing 3D depth map data obtained from the unmanned aerial vehicle and position/attitude data of the unmanned aerial vehicle in correspondence with an acquisition time in the unmanned aerial vehicle. , Transmitting the FPV image from the unmanned aerial vehicle, the acquisition time of the FPV image, and residual information of the unprocessed manual control command from the unmanned aerial vehicle to a remote control device, the FPV image and the unprocessed manual control command from the remote control device Displaying residual information on a screen, operating the unmanned aerial vehicle according to the manual control command received from the remote control device, checking the manual control command residual information when the second control mode is selected in the remote control device activating a semi-autonomous pilot command input interface for receiving a semi-autonomous pilot command after confirming that there are no unprocessed manual pilot commands remaining in the unmanned aerial vehicle; a semi-autonomous pilot command input through the semi-autonomous pilot command input interface; transmitting from the remote control device to the unmanned aerial vehicle, and operating the unmanned aerial vehicle in response to a semi-autonomous control command received from the remote control device while avoiding a collision with an obstacle using the 3D depth map data. includes

According to the present invention, the operator can appropriately select a manual control mode and a semi-autonomous control mode according to field conditions to minimize the effect of communication delay or video delay while using drones to perform search, surveillance, and reconnaissance missions. there is

1 is a diagram showing the configuration of an unmanned aerial vehicle control system according to an embodiment of the present invention.
2 is a block diagram showing the configuration of an unmanned aerial vehicle according to an embodiment of the present invention.
3 is a block diagram showing the configuration of a remote control device according to an embodiment of the present invention.

Then, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention.

1 is a diagram showing the configuration of an unmanned aerial vehicle control system according to an embodiment of the present invention.

Referring to FIG. 1 , the unmanned aerial vehicle control system 1 includes an unmanned aerial vehicle (UAV) 100 and a remote control device 200 that remotely controls the unmanned aerial vehicle 100 . The unmanned aerial vehicle 100 and the remote controller 200 may exchange various data and commands with each other through wireless communication such as Wi-Fi, 4G, and 5G mobile communication.

The unmanned aerial vehicle 100 may be a helicopter or a multi-copter capable of vertical take-off and landing.

2 is a block diagram showing the configuration of an unmanned aerial vehicle according to an embodiment of the present invention.

Referring to FIG. 2 , the unmanned aerial vehicle 100 includes an FPV camera 110, a 3D depth sensor 120, a position and attitude acquisition unit 130, a storage unit 140, a control unit 150 and a communication unit 160 can do.

The FPV camera 110 may obtain a first-person view front image (hereinafter referred to as FPV image) corresponding to the direction in which the unmanned aerial vehicle 100 moves.

The 3D depth sensor 120 may be implemented with a device capable of acquiring 3D depth map data, such as a 3D depth camera, a lidar sensor, a radar sensor, an optical flow sensor, or a Kinect.

The FPV camera 110 or the 3D depth sensor 120 may be mounted on the unmanned aerial vehicle 100 by a gimbal (not shown) or directly mounted on the unmanned aerial vehicle 100.

The position and attitude acquisition unit 130 may be composed of sensor devices such as a 3-axis accelerometer, a 3-axis gyroscope, a magnetometer, a barometer, a GPS sensor, and a distance measurer, and includes position information and attitude information of the unmanned aerial vehicle 100. Position and attitude data can be obtained.

The storage unit 140 may be implemented as a data storage device such as a ROM, RAM, or flash memory, and may store 3D depth map data and position/attitude data in correspondence with acquisition time of the corresponding data.

The controller 150 controls the overall operation of the unmanned aerial vehicle 100 and may operate the unmanned aerial vehicle 100 according to a remote control command transmitted from the remote control device 200 .

The controller 150 may operate the unmanned aerial vehicle 100 according to a manual control command transmitted from the remote control device 200 in the manual control mode.

Here, the manual control command is input by manually manipulating a joystick lever provided in the remote control device 200 by an operator to make the unmanned aerial vehicle 100 ascend, descend, move forward, backward, move left, move right, rotate, etc. It is a command. Manual control commands may be transmitted from the remote control device 200 at regular intervals.

The controller 150 may sequentially execute manual control commands received from the remote control device 200 . In addition, the control unit 150 may generate unprocessed manual control command residual information from the unmanned aerial vehicle and transmit it to the remote control device 200 . The manual steering command residual information may include whether there are remaining unprocessed manual steering commands and/or the remaining number of unprocessed manual steering commands.

In the semi-autonomous control mode, the control unit 150 operates the unmanned aerial vehicle 100 in response to a semi-autonomous control command received from the remote control device 200, and avoids a collision with an obstacle using 3D depth map data. .

The semi-autonomous control command is a command for autonomously flying the unmanned aerial vehicle while avoiding a collision with an obstacle in a specified moving target point or in a specified direction. The semi-autonomous steering command may include a command to move to a target point, a forward command to avoid obstacles, a reverse command to avoid obstacles, a command to move left to avoid obstacles, a command to move right to avoid obstacles, a command to descend to avoid obstacles, and a command to ascend to avoid obstacles.

The command to move to the target point is a command for autonomously flying the unmanned aerial vehicle 100 to the moving target point while avoiding collision with an obstacle.

The obstacle avoidance forward command, the obstacle avoidance backward command, the obstacle avoidance move command to the left, the obstacle avoidance move command to the right, the obstacle avoidance descend command, and the obstacle avoidance ascend command, respectively, the unmanned aerial vehicle 100 autonomously flies forward while avoiding a collision with an obstacle, It is a command to move backward, move left, move right, descend, ascend, etc. For example, through an obstacle avoidance forward command, the operator can make the UAV advance through an open window if there is a building wall with an open window in the forward direction, or prevent the UAV 100 from colliding with the tunnel wall in a curved tunnel. You can move it to the entrance of the tunnel without doing it.

The difference between the forward command input as the manual control command described above and the forward command to avoid obstacles is that the former continues to move forward even if the unmanned aerial vehicle 100 collides with an obstacle, whereas the latter moves the unmanned aerial vehicle 100 while avoiding collision with an obstacle. There is a difference in moving according to the command.

The control unit 150 processes 3D depth map data and / or FPV images to recognize and avoid obstacles in the moving direction, or to recognize an area through which an unmanned aerial vehicle can pass, such as a window, and implement an algorithm to move to the corresponding area. software may be included. The algorithm can be implemented through a deep learning model, a machine learning model, and the like.

The control unit 150 corresponds to the target point selected by the operator based on the 2D image coordinates (u, v) corresponding to the target point selected from the FPV image by the operator in the remote control device 200 and the acquisition time information at which the corresponding FPV image was obtained. 3D location coordinates (latitude, longitude, altitude) can be obtained. Here, the 2D image coordinates may be coordinates (u, v) corresponding to the location of the target point based on the UV coordinate system of the FPV image.

Specifically, the controller 150 may extract 3D depth map data corresponding to the FPV image acquisition time from the storage 140 and extract points matching 2D image coordinates (u, v) from the 3D depth map data. . A technique of matching FPV images obtained from the FPV camera 110 and 3D depth sensor 120 with 3D depth map data is well known, and thus a detailed description thereof will be omitted. In addition, the control unit 150 extracts the position and attitude data of the unmanned aerial vehicle corresponding to the FPV image acquisition time from the storage unit 140, and then uses this to obtain 3D position coordinates (latitude, longitude, altitude) of the point extracted from the 3D depth map data. can be calculated.

In the case of a target point movement command, the controller 150 may autonomously fly the unmanned aerial vehicle 100 to move it to the target point after acquiring 3D position coordinates corresponding to the target point.

Further, in the case of a search target point confirmation command, the controller 150 may obtain 3D position coordinates corresponding to the target point and transmit the acquired 3D position coordinates to the remote control device 200 .

The communication unit 160 may be implemented as a communication device supporting wireless communication between the unmanned aerial vehicle 100 and the remote controller 200 .

The communication unit 160 may transmit the FPV image, acquisition time of the FPV image, residual information of manual control commands not processed in the unmanned aerial vehicle 100, and 3D location information of the search target point to the remote control device 200.

The communication unit 160 may receive a manual control command, a semi-autonomous control command, a search target location confirmation command, and the like from the remote control device 200 .

3 is a block diagram showing the configuration of a remote control device according to an embodiment of the present invention.

Referring to FIG. 3 , the remote control device 200 according to the present invention may include a display module 210, a command input module 220, a memory module 230, a control module 240 and a communication module 250. can

The display module 210 performs a function of displaying various information related to the control of the unmanned aerial vehicle 100, and can be implemented as a display means such as LCD or LED.

The display module 210 may display the FPV image transmitted from the unmanned aerial vehicle 100 and the unprocessed manual pilot command residual information on the screen.

In the manual control mode, the operator can stand by without inputting a manual control command in situations where real-time is important, such as a fork in the road or passing through a door. Thereafter, when the unmanned aerial vehicle 100 is in a hovering state after processing all of the previously transmitted manual control commands, there is no delay between the FPV image checked by the operator and the FPV image currently acquired by the unmanned aerial vehicle 100 . Therefore, the operator can control the unmanned aerial vehicle 100 in a state in which the effect of video delay or time delay is minimized or has no effect by starting to input the manual control command again after confirming that there is no unprocessed manual control command residual information.

The command input module 220 may include a manual steering command input interface for receiving a manual steering command and a semi-autonomous steering command input interface for receiving a semi-autonomous steering command.

The manual control command input interface may be implemented through a physical device such as a joystick lever, or a user interface corresponding to the joystick lever may be implemented on a screen displayed by the display module 210 in a touch screen manner.

The semi-autonomous control command input interface may be implemented as a physical button corresponding to the semi-autonomous control command, or displayed as a button corresponding to each semi-autonomous control command on the screen displayed by the display module 210 in a touch screen manner. there is. Also, among the semi-autonomous control commands, the command to move the target point can be implemented in a manner in which a specific point is selected as the moving target point from the operator in the FPV image displayed on the screen.

On the other hand, the search target location confirmation command can also be implemented in a way that a specific point is selected as the search target point from the operator in the FPV video displayed on the screen, similar to the target point movement command.

A method of dividing and inputting a point selected by an operator into a moving target point and a search target point in the FPV video may be implemented differently depending on the embodiment. For example, in a state where a search target point selection button and a moving target point selection button are displayed on the screen, if an operator selects a specific point in the FPV video after pressing the search target point selection button, the selected point can be treated as a search target point. .

The memory module 230 may be implemented as a data storage device such as ROM, RAM, or flash memory, and may store various information and data related to control of the unmanned aerial vehicle 100 . The memory module 230 may permanently or temporarily store some or all of the data transmitted from the unmanned aerial vehicle 100 .

The control module 240 controls the overall operation of the remote control device 200 .

When the manual control mode is selected by an operator, the control module 240 may transmit a manual control command input through a manual control command input interface to the unmanned aerial vehicle 100 at predetermined intervals.

When the semi-autonomous control mode is selected by the operator, the control module 240 checks residual manual control command information transmitted from the unmanned aerial vehicle 100 . The control module 240 may also be implemented to activate a semi-autonomous pilot command input interface for receiving a semi-autonomous pilot command after confirming that there are no remaining unprocessed manual pilot commands in the unmanned aerial vehicle 100 . For example, a semi-autonomous control command input interface may be activated and displayed on the screen displayed on the display module 210 . Alternatively, if the semi-autonomous control command input interface is implemented as a physical button, the corresponding button may be activated to receive a command from an operator.

When the moving target point is selected from the FPV image by the operator, the control module 240 generates a command to move the target point including the 2D image coordinates of the FPV image of the corresponding moving target point and the image acquisition time of the corresponding FPV image, thereby moving the unmanned aerial vehicle 100. can be sent to

Meanwhile, when a search target point is selected from the FPV image by the operator, the control module 240 generates a search target point confirmation command including the 2D image coordinates of the FPV image of the corresponding search target point and the image acquisition time of the corresponding FPV image to unmanned air vehicle ( 100) can be transmitted.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Claims (10)

delete in unmanned aerial vehicles,
An FPV camera that acquires a first-person view forward-looking image (hereinafter referred to as FPV image);
a 3D depth sensor acquiring 3D depth map data;
a position and attitude acquisition unit for obtaining position and attitude data of the unmanned aerial vehicle;
a storage unit for storing the 3D depth map data and the position/attitude data in correspondence with an acquisition time;
A controller for operating the unmanned aerial vehicle according to a manual control command transmitted from a remote control device in a first control mode; and
A communication unit configured to transmit the FPV image, the acquisition time of the FPV image, and manual control command residual information not processed in the unmanned aerial vehicle to the remote control device, and to receive the manual control command from the remote control device.
including,
The remote control device displays the FPV image and the unprocessed manual control command residual information on a screen;
The unmanned aerial vehicle operates in response to a semi-autonomous control command received from the remote control device in a second control mode, but avoids collision with an obstacle using the 3D depth map data;
The remote control device,
When the second control mode is selected, the unmanned aerial vehicle activates a semi-autonomous pilot command input interface for receiving a semi-autonomous pilot command after confirming that there is no unprocessed manual pilot command remaining in the unmanned aerial vehicle by checking the residual manual pilot command information. aircraft. In paragraph 2,
The semi-autonomous steering command,
An unmanned aerial vehicle, which is a command for autonomously flying the unmanned aerial vehicle while avoiding obstacles and collisions in a designated moving target point or in a designated direction. In paragraph 3,
The control unit,
An unmanned aerial vehicle that moves the unmanned aerial vehicle in response to a semi-autonomous control command received from the remote controller and avoids a collision with an obstacle by using 3D depth data obtained in real time from the 3D depth sensor. In paragraph 3,
The semi-autonomous control command includes 2D image coordinates of a moving target point selected from the FPV image displayed on the remote control device and an image acquisition time of the FPV image in which the moving target point is selected,
The controller extracts 3D depth map data corresponding to the image acquisition time of the FPV image selected at the moving target point and position and attitude data of the unmanned aerial vehicle from the storage unit, and the controller extracts 2D image coordinates of the moving target point, the extracted An unmanned aerial vehicle that obtains 3D position coordinates of the moving target point based on 3D depth map data and position and attitude data of the unmanned aerial vehicle. In paragraph 2,
When receiving a search target location confirmation command from the remote control device, the 3D position coordinates of the search target point are checked and transmitted to the remote control device;
The search target location confirmation command includes 2D image coordinates of a search target point selected from the FPV image displayed on the remote control device and an image acquisition time of the FPV image in which the search target point is selected,
The control unit extracts 3D depth map data corresponding to the image acquisition time of the FPV image selected at the search target point and position and attitude data of the unmanned aerial vehicle from the storage unit, and the control unit extracts 2D image coordinates of the search target point, the extracted An unmanned aerial vehicle that checks the 3D position coordinates of the search target point using 3D depth map data and position and attitude data of the unmanned aerial vehicle. delete In the control method of the unmanned aerial vehicle,
storing 3D depth map data obtained from the unmanned aerial vehicle and position/attitude data of the unmanned aerial vehicle in correspondence with an acquisition time in the unmanned aerial vehicle;
Transmitting an FPV image from the unmanned aerial vehicle, an acquisition time of the FPV image, and residual manual control command information not processed in the unmanned aerial vehicle to a remote control device;
displaying the FPV image and the unprocessed manual control command residual information on a screen in the remote control device;
Operating the unmanned aerial vehicle according to a manual control command received from the remote control device;
When the second control mode is selected in the remote control device, the manual control command residual information is checked to confirm that there is no unprocessed residual manual control command in the unmanned aerial vehicle, and then a semi-autonomous control command is input to receive a semi-autonomous control command. activating the interface;
Transmitting a semi-autonomous control command input through the semi-autonomous control command input interface from the remote control device to the unmanned aerial vehicle; and
Operating the unmanned aerial vehicle in response to a semi-autonomous control command received from the remote control device and avoiding a collision with an obstacle using the 3D depth map data
including,
The semi-autonomous steering command,
A control method of an unmanned aerial vehicle, which is a command for autonomously flying the unmanned aerial vehicle while avoiding collision with an obstacle in a designated moving target point or in a designated direction. In paragraph 8,
The semi-autonomous control command includes 2D image coordinates of a moving target point selected from the FPV image displayed on the remote control device and an image acquisition time of the FPV image in which the moving target point is selected,
The unmanned aerial vehicle extracts 3D depth map data corresponding to the image acquisition time of the FPV image at the moving target point and position and attitude data of the unmanned aerial vehicle from a storage unit, and 2D image coordinates of the moving target point and the extracted 3D depth map A method of controlling an unmanned aerial vehicle for acquiring 3D position coordinates of the moving target point based on data and position and attitude data of the unmanned aerial vehicle. In paragraph 8,
When the unmanned aerial vehicle receives a search target position confirmation command from the remote control device, the 3D position coordinates of the search target point are checked and transmitted to the remote control device;
The search target location confirmation command includes 2D image coordinates of a search target point selected from the FPV image displayed on the remote control device and an image acquisition time of the FPV image in which the search target point is selected,
The unmanned aerial vehicle extracts 3D depth map data corresponding to the image acquisition time of the FPV image at which the search target point is selected and position and attitude data of the unmanned aerial vehicle from a storage unit, and 2D coordinate information of the search target point and the extracted 3D depth map A method of controlling an unmanned aerial vehicle for checking 3D position coordinates of the search target point using data and position and attitude data of the unmanned aerial vehicle.

Images