KR102843953B1 - System and Method for Monitoring Unmanned Aircraft - Google Patents

Abstract

The present invention relates to a drone control system and a drone control method, and the drone control system according to an embodiment of the present invention may include a drone that stores map data or simulation data of metaverse data that enables judgment of the possibility of collision with another drone in relation to a flight path, and performs autonomous flight by measuring a flight position and altitude from the (pre-)stored simulation data using sensor data of a sensor provided by the drone itself when a control signal received from an external device is not detected, and a drone control service device that implements the real world in which the drone flies as a virtual world of the metaverse to control the movement of the drone on a screen of the metaverse, and displays the analysis results of the sensor data of the video, photo, or sensor provided by the drone on the metaverse screen together to perform control.

Description

System and Method for Monitoring Unmanned Aircraft

The present invention relates to a drone control system and a drone control method, and more specifically, to a drone control system and a drone control method that include a simulation section, such as a metaverse program environment related to the operation of a drone, in the drone itself, and in the event of a communication interruption, etc., corrects the position and height in a simulation environment using the drone's own computer (e.g., a flight controller (or flight computer) (FC)) to find a path.

An unmanned aerial vehicle (UAV), or drone, is an aircraft without a pilot on board. It is remotely controlled from the ground, flies autonomously or semi-autonomously along a pre-programmed route, or is equipped with artificial intelligence to perform missions based on its own environmental judgment. A drone system encompasses the entire system, including the Ground Control Station (GCS), communications equipment, and support equipment. Drones operate independently or in conjunction with space and terrestrial systems. Depending on the application, they can be equipped with various devices (e.g., optical, infrared, and radar sensors) to perform missions such as surveillance, reconnaissance, precision attack weapon guidance, communication/information relay, and decoys. They are also being developed and commercialized as precision weapons, loaded with explosives, attracting attention as a key economic and military force of the future. Furthermore, drones are being used for diverse purposes beyond military applications. Examples include surveying, filming, rescue, pipeline/electrical inspection, wildlife observation, relief supplies delivery to quarantined areas, and product delivery. To move and store drones, commercial or mission drones are mounted on vehicles, moved to the point of use, flown at a designated location, and then returned to the original location or vehicle.

Low-altitude unmanned aerial vehicle (UAV) traffic management (UTM) technology utilizing drones has recently been gaining attention. UTM's primary function is to receive flight plans from UAS operators, gather terrain, weather, and surveillance information from external data providers, receive operational requirements from the flight information management system, plan routes, collision avoidance strategies, and airspace use, and display the aircraft's location on a map in real time through a surveillance system. By transmitting and displaying video data acquired on-site in real time to a remote control center via LTE or 5G, it ensures on-site visibility. Furthermore, by providing real-time flight information, such as the drone's location, altitude, longitude, speed, and battery level, on a GIS-based platform, it enhances intuitiveness. JCH DroneRTS excels in real-time drone control and monitoring.

However, conventional UTM systems suffer from system failures in the event of LTE, 5G, or ground control equipment (RC) failures, compromising the safety of drone operations. This necessitates the development of technologies to ensure the safety of drone operations.

Korean Patent Publication No. 10-2254491 (May 14, 2021) Korean Patent Publication No. 10-2259855 (May 27, 2021) Korean Patent Publication No. 10-2500221 (February 10, 2023) Korean Patent Publication No. 10-2024-0095646 (June 26, 2024)

The technical problem to be achieved by the present invention is to provide a drone control system and a drone control method that includes a simulation section such as a metaverse program environment related to the operation of the drone itself, and in the event of a communication cutoff, etc., corrects the position and height in a simulation environment using the drone's own computer (e.g., flight controller (or flight computer) (FC)) to find a path.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to achieve the above object, the drone control system according to an embodiment of the present invention stores simulation data of map data or metaverse data that enables judgment of the possibility of collision with another drone in relation to a flight path, and performs autonomous flight by measuring a flight position and altitude from the stored simulation data using sensor data of a sensor provided by the drone itself when a control signal received from an external device is not detected, and a drone control service device that implements the real world in which the drone flies as a virtual world of the metaverse to control the movement of the drone on a screen of the metaverse, and displays the analysis results of the sensor data of the captured video, photo, or sensor provided by the drone together on the metaverse screen to perform control.

The above drone can receive a control signal from a ground control device that controls the drone on the ground as an external device or a comprehensive control device controlled by a control agent that remotely controls the drone from a long distance in the event of an emergency situation in which the control signal is not detected, and fly a designated route.

The above drone control system further includes a comprehensive control device that implements metaverse data provided by the drone control service device on a screen to control the flight location and flight environment of the drone on the screen of the metaverse, and the drone control service device can operate to implement the paths of a plurality of drones equipped with an artificial intelligence (AI) flight controller (FC) that performs the autonomous flight on the screen of the metaverse so that they are controlled by the comprehensive control device.

The above drone control system may further include a station device that allows the drone to take off and land and rapidly charge the drone to a speed higher than a reference speed to enable a task transition when the drone is used for the purpose of forest fire surveillance for forest fire prevention.

The above drone processes the captured video or sensor data through the external device when the above-mentioned unexpected situation does not occur, so that the analysis result is reflected on the screen of the metaverse, and when the above-mentioned unexpected situation occurs, the data according to the operation of the AI FC is directly transmitted to the drone control service device, so that the analysis result of the data is reflected on the screen of the metaverse.

In addition, the drone control system according to an embodiment of the present invention for achieving the above-mentioned task includes a step of storing simulation data of map data or metaverse data that enable the drone to determine the possibility of collision with another drone in relation to the flight path, and performing autonomous flight by measuring the flight position and altitude from the stored simulation data using sensor data of a sensor provided in the drone itself when a control signal received from an external device is not detected, and a step of allowing the drone control service device to implement the real world in which the drone flies into a virtual world of the metaverse and control the movement of the drone on a screen of the metaverse, and displaying the analysis results of the sensor data of the captured video, photo, or sensor provided by the drone together on the metaverse screen to perform control.

The above drone control method may further include a step of receiving a control signal from a ground control device that controls the drone on the ground as an external device or a comprehensive control device controlled by a control agent that remotely controls the drone from a long distance when an unexpected situation in which the control signal is not detected does not occur, and flying the drone along a designated path.

The above drone control method may further include a step of allowing the integrated control device to implement metaverse data provided by the drone control service device on a screen to control the flight location and flight environment of the drone on the screen of the metaverse, and a step of allowing the drone control service device to implement the paths of a plurality of drones equipped with an artificial intelligence (AI) flight controller (FC) performing the autonomous flight on the screen of the metaverse so that the paths are controlled by the integrated control device.

The above drone control method may further include a step in which the station device causes the drone to take off and land and rapidly charge the drone to a speed higher than a reference speed to enable a task transition when the drone is used for the purpose of forest fire surveillance for forest fire prevention.

The above drone control method may further include a step of processing video or sensor data captured by the drone through the external device when the unexpected situation does not occur so that the analysis result is reflected on the screen of the metaverse, and a step of directly transmitting data according to the operation of the AI FC when the unexpected situation occurs to the drone control service device so that the analysis result of the data is reflected on the screen of the metaverse.

According to an embodiment of the present invention, when an unmanned aerial vehicle (UAV) deviates from (or departs from) the position or altitude of the designated route due to unintended wind or gusts while flying a designated route in accordance with a control command from an external device, or when communication is impossible due to a control command not being received from an external device, the UAV can perform autonomous flight using map data or metaverse data related to the flight route installed internally and capable of preventing collisions between multiple UAVs, thereby enabling data processing with a cloud server, etc.

In addition, according to an embodiment of the present invention, when a drone performs autonomous flight in an unintended situation, data is processed directly with a cloud server or the like, that is, one (metaverse) platform and two digital twins (e.g., control room and drone), that is, two-way (twin) operation is possible, so that computer error duplication can be checked, and in case of hardware or software failure, stable drone operation can be possible through another path.

Furthermore, embodiments of the present invention can simplify and diversify the command system and data transmission, thereby promoting the stabilization of the transmission system.

In addition, in an embodiment of the present invention, when a drone performs autonomous flight due to an unexpected situation, an optimized path can be generated based on the analysis results of internal map data, metaverse data, and sensor data acquired from sensors, and autonomous flight can be performed using the optimized path.

FIG. 1 is a drawing showing an unmanned aerial vehicle control system according to an embodiment of the present invention.
Figure 2 is a drawing showing a schematic representation of the system of Figure 1.
Figure 3 is a schematic diagram showing an urban air mobility (UAM) system that applies the system of Figure 1.
Figure 4 is a diagram illustrating an AI forest fire prediction drone station.
Figures 5a and 5b are examples of control screens provided by the drone control service device of Figure 1.
FIG. 6 is a drawing for explaining a drone control process according to an embodiment of the present invention.
Figure 7 is a block diagram illustrating the detailed structure of the drone of Figure 1.
Figure 8 is a flowchart showing the driving process of the drone of Figure 1.

The embodiments of the present invention described below are provided to more clearly explain the present invention to a person having ordinary skill in the art, and the scope of the present invention is not limited by the following embodiments, and the following embodiments may be modified in various other forms.

The terminology used herein is used to describe particular embodiments and is not intended to limit the present invention. The singular forms used herein may include the plural forms unless the context clearly dictates otherwise. In addition, the terms "comprise" and/or "comprising" used herein specify the presence of a stated feature, step, number, operation, element, element, and/or group thereof, but do not exclude the presence or addition of one or more other features, steps, numbers, operations, elements, elements, and/or groups thereof. In addition, the term "connection" used herein not only means that certain elements are directly connected, but also includes a concept that indirectly connects elements by interposing another element between them.

In addition, when it is said in this specification that a certain element is located "on" another element, this includes not only cases where a certain element is in contact with another element, but also cases where another element exists between the two elements. The term "and/or" as used in this specification includes any one of the listed items and any and all combinations of one or more of them. In addition, terms of degree such as "about", "substantially", etc. as used in this specification are used to mean a range of or close to the numerical value or degree, taking into account inherent manufacturing and material tolerances, and are used to prevent infringers from unfairly using the disclosure that mentions exact or absolute numbers provided to help the understanding of this specification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. The sizes and thicknesses of areas or parts illustrated in the attached drawings may be somewhat exaggerated for clarity and convenience of explanation. Like reference numbers designate like components throughout the detailed description.

FIG. 1 is a diagram showing a drone control system according to an embodiment of the present invention, FIG. 2 is a diagram showing a schematic representation of the system of FIG. 1, FIG. 3 is a diagram briefly showing an urban air traffic control system (UAM) applying the system of FIG. 1, FIG. 4 is a diagram showing an AI forest fire prediction drone station as an example, and FIGS. 5a and 5b are exemplary diagrams of control screens provided by the drone control service device of FIG. 1.

As illustrated in FIGS. 1 and 2, the drone control system (90) according to an embodiment of the present invention is a system that can be applied to, for example, UTM or UAM (Urban Air Mobility), and forest fire forecasting, and includes part or all of a drone (100), a ground control device (110), a communication network (120), a drone control service device (130), and an integrated control device (140). It may further include a third-party device such as a weather service server that provides weather information, etc.

Here, “including some or all” means that some components, such as the ground control device (110), may be omitted to configure the drone control system (90) of FIG. 1, or some components, such as the drone control service device (130), may be integrated into a network device (e.g., gateway, wireless switching device, etc.) that constitutes the communication network (120), and is described as including all in order to help sufficient understanding of the invention.

The unmanned aerial vehicle (100) can be called by various names such as an unmanned aerial vehicle (UAV) or an unmanned aerial vehicle (UAV). The unmanned aerial vehicle (100) refers to an aircraft that performs a mission by remotely controlling it from the ground or flying automatically (semi-automatically) or autonomously according to a pre-programmed route without a pilot directly riding the aircraft. Of course, the unmanned aerial vehicle (100) according to the embodiment of the present invention may include a drone as commonly referred to. The unmanned aerial vehicle (100) can be utilized in various industrial fields, and has recently been attracting attention worldwide by being applied to urban air traffic systems. With regard to UAM, this is well illustrated in FIG. 3. UAM is a general term for aircraft, operation infrastructure, and services that transport passengers and cargo by air within urban areas (e.g., densely populated and congested ground traffic areas). The aircraft used in UAM is limited to eVTOL (Electric Vertical Take-off & Landing) that does not require a runway.

Looking into it a little more, the categories in which unmanned aerial vehicles (100) are used can be broadly categorized into future air vehicles (AAVs) and advanced air mobility (AAM). AAVs can be further divided into personal air vehicles (PAVs), such as air taxis or drone taxis, and eVTOLs, such as flying cars or flying taxis. AAM can also be divided into urban air mobility (UAM) and regional air traffic systems. A UAV system refers to the entire system, including the aircraft, which is a UAV, ground control equipment, communication equipment, and support equipment. Unmanned aircraft system (UAS) is the official name for ensuring engineering safety, such as the development, manufacturing, and operation of the system, as an aircraft rather than an aircraft. A remotely piloted aerial vehicle (RPV) refers to an unmanned aerial vehicle that is remotely controlled via radio communication from the ground. It was used when UAVs first appeared, but is no longer used today. RPAS stands for Remotely Piloted Aircraft System. Regional Air Mobility (RAM) refers to aircraft, operational infrastructure, and services that transport passengers and cargo between cities or satellite cities centered around major metropolitan areas. Both eVTOLs and eCTOLs are capable of operation, but certain requirements must be met to enter urban areas. AAM refers to an industry encompassing both UAM and RAM. NASA first coined the term, while EASA in Europe calls it Innovative Air Mobility (IAM).

The drone (100) according to an embodiment of the present invention can be used for monitoring forest fires, landslides, and pests, for example, when used for UTM. In this case, the drone (100) can fly according to the control command of the ground control unit (110), but it can also be operated by remote control of the integrated control unit (140) of the control center that performs the corresponding monitoring operation. For example, the integrated control unit (140) can transmit and receive control data related to drone path flight, 3D altitude flight, etc., through communication with the drone (100). Accordingly, the drone (100) can provide in real time the captured video, the detected sensing data, or the photos taken while flying the corresponding path to the drone control service unit (130) of FIG. 1, and the drone control service unit (130) can generate the analysis result of the corresponding data and provide it to the integrated control unit (140). Of course, these operations can also be applied to UAM. As shown in Fig. 3, a UAM operator can be installed at each vertiport where a UAV (100) operates, and each operator can communicate (or contact) with the UAV (100). During this process, route weather information, operational safety information, real-time aircraft operation information, and aircraft key status information can be exchanged. Of course, such flight information, surveillance information, weather information, and aviation information can be provided to the UAV (100) by a separate operational support information provider. For example, weather information can be provided from the Korea Meteorological Administration's server.

Above all, the drone (100) according to an embodiment of the present invention may include a flight controller (FC) that operates based on artificial intelligence (AI). The FC may also be referred to as a flight computer. Through this, the drone (100) may be able to automatically find a flight path, etc., by using its own map data or metaverse data when communication with the ground control unit (110), drone control service unit (130), or integrated control unit (140) is cut off, and may generate an optimized path and operate. Of course, since the optimized path is operated by multiple drones, it may be generated by taking into account collision avoidance, etc. For example, the drone (100) may fly along a designated path under the control of the ground control unit (110) or integrated control unit (140). For example, it may perform forest fire surveillance. During this process, if an unintended strong gust of wind is encountered, the aircraft may deviate from the designated path. In this case, communication can be performed with the ground control unit (110) or the integrated control unit (140). When communication is cut off, the aircraft can recover the deviation path by executing a simulation program related to flight on its own and continue the flight again. An artificial intelligence program for this type of autonomous flight can be applied. It finds the path by correcting the position and height through artificial intelligence actions. This AI action can be performed during takeoff, landing, and flight, and it is also entirely possible for the AI to check multiple flight paths and generate an optimized path to fly. Of course, checking multiple flight paths and generating an optimized path can be considered to be based on the learning results of pre-learned learning data.

Figure 4 illustrates a station (device) where an unmanned aerial vehicle (100) takes off and lands when used for forest fire surveillance, etc. The unmanned aerial vehicle (100) can be an AI drone, and the station can also be used to perform operations for early detection of forest fires and prevention of their spread. The station can form a space where the unmanned aerial vehicle (100) takes off and lands. For example, when a unmanned aerial vehicle (100) approaches, the stations can communicate with each other to guide the unmanned aerial vehicle (100) to land stably in a designated direction or posture. To respond to forest fires that occur year-round, a year-round forest fire surveillance system is required, and in this regard, unmanned aerial vehicles (100) and stations can be utilized. For example, drone stations can be installed in forests of each local government. In addition, a forest fire surveillance system can be established using a ground control system (GCS). For example, an AI drone operating as an unmanned aerial vehicle (100) can perform high-speed charging via the station, enabling rapid task transitions. Thanks to the TEC air conditioner, which cools the drone battery, rapid charging and takeoff are possible. Charging from 10% to 90% takes only about 25 minutes, making it useful for forest fire surveillance. By installing and utilizing an AI drone station capable of day and night monitoring, a system for early detection and early response to forest fires can be established.

The ground control unit (110) of Fig. 1 may include a remote control unit (RC) or a tab (or tablet) PC, etc. For example, the ground control unit (110) may monitor the flight environment of the drone (100) stored in advance through a monitor in real time and command a designated operation. The drone (100) may provide the ground control unit (110) with the captured video taken under the control of the ground control unit (110), but may also transmit it to the drone control service unit (130) so that control can be performed at the control center by a control agent, etc. If the ground control unit (110) controls the drone (100) on a 1:1 basis and confirms its operation, the integrated control unit (140) of Fig. 1 can be viewed as managing a plurality of drones (100) currently in operation in an integrated manner. More precisely, the integrated control device (140) can monitor events of incidents or accidents related to multiple unmanned aerial vehicles (100), and since notifications of such incidents or accidents can be provided from the ground control device (110), the embodiment of the present invention will not be particularly limited to any one form. For example, in a UAM environment such as FIG. 3, the ground control device (110) may refer to various devices operated by a UAM operator. Here, the UAM operator's devices may refer to computers, etc., owned by a manager, and may also include communication devices for communication.

The communication network (120) can be configured in various forms. The communication network (120) can include both wired and wireless communication networks. For example, a wired or wireless Internet network can be used or linked as the communication network (120). Here, the wired network includes an Internet network such as a cable network or a public switched telephone network (PSTN), and the wireless communication network includes a public or private wireless network based on CDMA, WCDMA, GSM, EPC (Evolved Packet Core), LTE (Long Term Evolution), Wibro network, and LPWA (Low Power Wide Area) recently used for applying Internet of Things (IoT) technology. Of course, the communication network (120) according to the embodiment of the present invention is not limited thereto, and can be used as an access network of a next-generation mobile communication system to be implemented in the future, for example, a cloud computing network under a cloud computing environment, a 5G network, a 6G network, etc. For example, if the communication network (120) is a wired communication network, an access point within the communication network can connect to a telephone exchange, etc., but if it is a wireless communication network, data can be processed by connecting to an SGSN or GGSN (Gateway GPRS Support Node) operated by a communication company, or data can be processed by connecting to various relays such as a BTS (Base Transceiver Station), NodeB, or e-NodeB.

The communication network (120) may include an access point. The access point may include a small base station such as a femto or pico base station, which is often installed inside a building. Here, the femto or pico base station may be classified according to the maximum number of unmanned aerial vehicles (100) or ground control devices (110) of FIG. 1 that can be connected to the small base station. Of course, the unmanned aerial vehicles (100) or ground control devices (110) may include a short-range communication module for performing short-range communication such as Zigbee and Wi-Fi. The access point may use TCP/IP or RTSP (Real-Time Streaming Protocol) for wireless communication. Here, short-range communication may be performed in various standards such as Bluetooth, Zigbee, infrared (IrDA), radio frequency (RF) such as UHF (Ultra High Frequency) and VHF (Very High Frequency) in addition to Wi-Fi, and ultra-wideband communication (UWB). Accordingly, the access point can extract the location of the data packet, designate the best communication path for the extracted location, and forward the data packet along the designated communication path to the next device, such as the drone control service device (130). The access point can share multiple lines in a typical network environment, and may include, for example, a router, a repeater, and a repeater.

The drone control service device (130) may include, for example, a cloud server, and may be configured to include a database (120a) linked to the server. The server and the database (120a) are connected via a dedicated network such as an intranet, and data processed by the server may be systematically classified in the database (120a) based on identification information of the drone (100), regional information where the drone (100) is used, and data such as captured videos or photos may be stored and managed. Of course, the analysis results of the data may be generated as a control screen (or inserted into the control screen) and provided so that control may be performed in the integrated control device (140) of FIG. 1. The drone control service device (130) may perform control service operations for integrated management of drones (100) utilized for various purposes, for example. Of course, the control screen may be formed in various ways, and thus the present invention will not be particularly limited to any one form. In other words, the control screens when the drone control service device (130) is used for forest fire forecasting and when it is used in a UAM environment as shown in FIG. 3 may be different. As shown in FIG. 3, the UAM traffic management service provider may include a control server that constitutes the drone control service device (130) of FIG. 1.

The drone control service device (130) according to an embodiment of the present invention can implement the environment in which the drone (100) of FIG. 1 operates as a metaverse environment and provide it to the comprehensive control device (140) of FIG. 1. For example, in order to build a metaverse environment, the drone control service device (130) can perform photography of an area where forest fire forecasting is performed, or, if there is a previously photographed image, can create a virtual space of the metaverse using the photographed image. The metaverse environment, or virtual space, means a conversion of a real environment into a virtual space, and the trends of drones (100) currently operating or flying within the virtual space can be displayed on the screen. Of course, this can be easily controlled by the comprehensive control device (140) of FIG. 1 by displaying location information, such as latitude and longitude information, on the virtual space of the metaverse or on map data when the drones (100) fly. Fig. 5a illustrates an example of a control screen, and Fig. 5b illustrates an object on the screen of an actual drone (100) in flight. The drone object on the metaverse screen related to the drone (100) can be easily configured by the control agent to facilitate control. For example, when comparing drones in Seoul and Gyeonggi-do, or when comparing drones from different manufacturers, different styles of drone objects can be designated and controlled to distinguish between regions or manufacturers, facilitating control.

Of course, the drone control service device (130) according to the embodiment of the present invention can operate in various forms with the drone (100), ground control device (110), and integrated control device (140) of FIG. 1. For example, the drone control service device (130) can receive weather information from a third-party device including a server such as the Korea Meteorological Administration, generate information on the sky over which the drone (100) is currently flying on the screen, and enable the integrated control device (140) to perform control. In addition, when the ground control device (110) requests remote control to the integrated control device (140) due to a disconnection in communication with the drone (100), the integrated control device (140) can display information related to an incident or accident event on the control screen. Accordingly, the integrated control device (140) can communicate with the drone (100) and remotely control takeoff, landing, and flight. If there is no control command from the ground control unit (110) or the integrated control unit (140) due to an unintended situation such as a communication cutoff, the drone (100) can operate in an autonomous mode and execute an AI program installed inside to perform takeoff and landing, flight, and check multiple flight paths and generate an optimized path to perform flight. Of course, during this process, the drone control service unit (130) of FIG. 1 transmits real-time video footage or location information related to the current flight, so that the manager of the ground control unit (110) of FIG. 1 or the control agent of the integrated control unit (140) of FIG. 1 can easily understand the operating status of the drone (100). The drone (100) can perform any number of operations such as takeoff and landing, flight, flight path check, and optimized path generation by utilizing map data or metaverse data related to the flight environment installed inside. Of course, this process can also be performed based on previously learned learning data. For example, a drone (100) can be controlled to fly along a path of A-B-C-D, and when an unintended communication failure occurs at point B, it can use pre-learned learning data to know that the next path is C, or it can use information included in map data or metaverse data to know, and of course, in the process, if a specific location or altitude must be maintained, the location or altitude can be found by analyzing sensor data from various types of sensors installed internally. Of course, it is also possible to use map data or the metaverse environment of the flight environment registered in advance in the process. In other words, if the objects in the video currently being filmed by the drone (100) and the objects in the map data or the metaverse environment are the same, the drone (100) can be judged to be flying at a normal location or altitude, and also flying a normal path.

The drone control service device (130) according to an embodiment of the present invention can control and track multiple drones, and this can be done through the AI FC according to an embodiment of the present invention mounted on the drone (100). For example, control can be done only for drones mounted with the AI FC. In addition, since two digital twins are possible on a single platform that provides metaverse services, in other words, two-way (or bidirectional) operation of the drone (100) and the digital twin is possible, and also, two-way operation of the digital twin is possible through a communication network (120) such as 5G or LTE with the integrated control device (140), so that duplicate checks for computer errors, i.e. inspection, are possible, and stable operation of the drone (100) is possible through another path in the event of a hardware or software failure. In addition, by operating two digital twins, the command system and data transmission can be simplified and diversified, thereby promoting the stabilization of the transmission system. In the event of a failure of the ground control unit (110) or communication network (120), the digital twin of the metaverse can be operated by the AI FC mounted on the drone (100). This can minimize the delay time of the digital twin. While the communication network (120) utilizes a base station of a specific telecommunications company, it is entirely possible to communicate directly with the drone control service device (130) via a LoRA communication module mounted on the drone (100).

The core technologies of digital twins include 3D modeling, data collection and analysis, and prediction and optimization. 3D modeling includes drones, GIS, CAD, and BIM, while data collection and analysis includes IoT sensors, big data, and AI. Prediction and optimization also include simulation and AR/VR/MR. This allows the real world to be transformed into a virtual world. The real and virtual worlds can be synchronized and optimized using 5G networks, the cloud, edge computing, and security. In other words, the drone control service device (130) according to an embodiment of the present invention can display images of the location or region where the drone (100) is flying in a virtual space and monitor the flight of the drone (100) in that virtual space. For example, the movement trends of the drones (100) can be displayed in the virtual space. This facilitates control from the integrated control device (140).

The integrated control unit (140) may refer to a computer, such as a control agent, that performs control operations at the control center. Of course, the control center may be equipped with an electronic display board or monitor to comprehensively control the entire flight of drones (100), and integrated management may be performed through the monitor. In addition, the control agents can judge emergency situations, etc. of the drones (100) while looking at the monitor, and can also determine the current flight status. If remote control is required, it may be possible to remotely control the drones (100). For example, in the case where 24-hour forest fire surveillance is required using an AI drone station as shown in FIG. 4, it may be difficult for a manager using the ground control unit (110) of FIG. 1 to be permanently stationed at the relevant location. In this case, forest fire surveillance can be facilitated by remotely controlling and monitoring the drones (100) that take off and land at the station from the integrated control unit (140). For example, the drone (100) may be able to provide operation data related to the station's operation status. Of course, the station itself may be equipped with a communication module so that its operation status (e.g., failure, etc.) may be periodically provided to the drone control service device (130) so that control may be performed by the integrated control device (140). However, it is also possible to save on system construction costs, etc., by having the station's operation status controlled through the drone (100). For example, when the drone (100) lands, the station's operation data may be wirelessly transmitted so that the data may be transmitted to the drone control service device (130) through the drone's (100) communication module.

In addition to the above, the drone (100), ground control device (110), communication network (120), drone control service device (130), and integrated control device (140) of FIG. 1 can perform various operations, and since related contents will be continuously covered later, those contents will be replaced.

Figure 6 is a block diagram illustrating the detailed structure of the drone of Figure 1.

As illustrated in FIG. 6, the unmanned aerial vehicle (100) according to an embodiment of the present invention may include an AI drone used for forest fire surveillance, an unmanned aerial vehicle used for UAM as in FIG. 3, and includes part or all of a communication interface unit (600), a monitoring unit (610), a control unit (620), an AI FC unit (630), and a storage unit (640). Of course, the unmanned aerial vehicle (100) may further include various mechanical (or hardware) components such as a propeller and a motor for driving the propeller, but in the embodiment of the present invention, only the driving mechanism related to the technical idea will be examined.

Here, “including some or all” means that some components, such as the storage unit (640), may be omitted to form the drone (100) of FIG. 1, or some components, such as the AI FC unit (630), may be integrated into other components, such as the control unit (620), and so on. In order to help a sufficient understanding of the invention, it is described as including all.

The communication interface unit (600) can communicate directly (e.g., P2P, etc.) with the ground control unit (110), or can communicate with the drone control service unit (130) and the integrated control unit (140) via the communication network (120) of FIG. 1. Of course, although not separately illustrated in the drawing, it can also communicate with GPS. To this end, the communication interface unit (600) includes a first communication module to a third communication module, and can communicate with the ground control unit (110) via each communication module, or can communicate with the drone control service unit (130) or the integrated control unit (140) via the communication network (120). For example, GPS communication can be performed via the GPS module to generate location information on the current flying location of the drone (100). In the process of performing communication, the communication interface unit (600) can perform operations such as modulation/demodulation, encoding/decoding, etc. This is obvious to those skilled in the art, so further explanation will be omitted.

The communication interface unit (600) can provide the ground control unit (110) with images or photos taken by a camera while moving along a designated path according to a control command from the ground control unit (110) under the control of the control unit (620), and sensor data acquired through various types of sensors. Accordingly, the manager of the ground control unit (110) can check the flight status or route of the drone (100) he or she is controlling on the metaverse screen displayed on the monitor. Of course, the data can also be provided to the drone control service unit (130) of FIG. 1 according to a method designated in the program, and the ground control unit (110) can also check the flight trend of the drone (100) through the metaverse screen provided by the drone control service unit (130).

In addition, the communication interface unit (600) performs various operations under the control of the control unit (620), and for example, when the drone (100) is operated for a purpose such as forest fire surveillance, it can perform a flight by remote control of the integrated control unit (140) of FIG. 1. In addition, when the drone (100) is flying for a purpose such as forest fire surveillance, if it deviates from the designated path due to unintended wind or gusts, and furthermore, if a control command from the integrated control unit (140) is not received, the communication interface unit (600) can automatically control the drone by the internally installed AI FC under the control of the control unit (620), and it is entirely possible to provide data such as location and altitude information generated through this.

The surveillance unit (610) may include a camera or various types of sensors. For example, the camera may be used to capture images of a specific area and provide photographs while flying along a designated route according to a control command from the ground control unit (110) or integrated control unit (140) of FIG. 1. For example, it may also be possible to capture photographs of wet plants in a specific area. In addition, the surveillance unit (610) may generate location information, such as latitude and longitude information related to the flight location, of the drone (100) using various types of sensors. For example, in addition to location information using GPS, inertial sensors such as a gyro sensor or an acceleration sensor may be used to generate location information of the current location, and altitude may be measured using an altitude sensor. Various types of altitude sensors may be used, such as an atmospheric pressure detection-type altitude sensor, an ultrasonic altitude sensor, or a vision sensor. Above all, the sensors constituting the monitoring unit (610) according to the embodiment of the present invention may be able to operate under the control of the control unit (620) when the control commands related to the flight or operation of the drone (100) are disconnected from the outside, i.e., are not received. For example, the drone (100) may encounter unintended wind or gusts during flight, and in this case, the drone (100) may maintain flight safety by making its own judgment and operating on the next route without relying on external control. The sensors constituting the monitoring unit (610) may be involved in such operations.

The control unit (620) includes a processor such as a CPU, an MPU, a GPU, and may further include a memory such as RAM. The processor and the memory may be configured as a single chip in the form of an IC chip, etc. The control unit (620) may perform the overall control operation of the communication interface unit (600), the monitoring unit (610), the AI FC unit (630), and the storage unit (640). When a control signal (or command) related to flight is received from the outside through the communication interface unit (600), the control unit (620) may fly the unmanned aerial vehicle (100) according to the control signal. To this end, the unmanned aerial vehicle (100) may find a path and fly based on sensor data received through various types of sensors constituting the monitoring unit (610). Of course, it is also possible to analyze the image data of the captured images constituting the monitoring unit (610), identify an object based on the analysis result, and fly based on the identified object. However, it may be desirable to measure direction using a gyro sensor, measure latitude and longitude through communication with GPS, and further measure altitude using an altitude sensor and perform flight based on the measurement results.

The control unit (620) may attempt to communicate with the ground control unit (110) or the integrated control unit (140) when an external control command is not received through the communication interface unit (600), for example, when wind or gusts are detected by the sensors constituting the monitoring unit (610), when a reference value is exceeded, or when a preset condition is satisfied. During this process, if a control signal is not received within a specified time, an AI program installed in the AI FC unit (630) may be executed. The AI program may be a type of simulation program, and may be map data or metaverse data related to the area of the flight path. Since the control unit (620) may pre-register data related to the flight path, it may perform autonomous flight using the flight path data stored within it. In this process, the location or altitude may be found. In other words, since information such as the location or altitude of the flight path may be set in the map data or metaverse data, the location and altitude may be found by analyzing the sensor data collected by the monitoring unit (610). Of course, metaverse data, which represents a virtual space such as map data or the metaverse where flights are simulated, may also be preset with data related to geographical features along the flight path. Accordingly, the control unit (620) may readily determine the current location and altitude based on geographical features. In other words, if a designated object is detected in the video captured at point C during a flight along the A-B-C-D route, the flight is deemed appropriate. The control unit (620) may operate in conjunction with the AI FC unit (630) to perform these operations.

The AI FC unit (630) can operate as a twin with the drone control service unit (130) of FIG. 1. Operating as a twin here means providing data to the drone control service unit (130) in real time to implement the real world as a virtual world. This can imply two-way communication. For example, the AI FC unit (630) can collect various information about the current flight path through communication with the drone control service unit (130). It can collect and transmit terrain, weather, surveillance information, etc. Furthermore, it is entirely possible to receive data related to plans such as routes, collision avoidance strategies, and airspace use from the drone control service unit (130) and operate accordingly.

The AI FC unit (630) according to an embodiment of the present invention can internally load map data or metaverse data related to the flight path, and through this, airspace information such as no-fly zones can be checked in normal times, and information such as temperature, humidity, wind direction, and wind speed can be collected and provided to the drone control service device (130) so that the corresponding information can be displayed in the virtual world of the metaverse. Of course, in addition, information such as latitude and longitude, altitude, speed, direction, and gimbal pitch can also be provided so that it can be displayed in the virtual space. Above all, the AI FC unit (630) can perform operations for autonomous flight when an unexpected situation occurs during flight along the scheduled flight path. Through this, operations such as takeoff and landing and flight, as well as multiple flight path inspection and optimized path generation can be performed. For example, when a control command is not received from an external source, autonomous flight can be performed using the map data or metaverse data stored internally, and in the process, the flight path can be confirmed and takeoff and landing or flight can be performed. In addition, by analyzing the captured images received through the sensors or cameras that constitute the detection unit (610), the current location (e.g., latitude and longitude), altitude, direction, etc. can be determined, and autonomous flight can be performed based on information in map data or metaverse data. Of course, in this process, if the objects in the captured images captured by the camera are determined, and a designated object is detected, it can be determined that normal flight is being performed along the designated flight path.

The storage unit (640) can temporarily store various types of information or data processed under the control of the control unit (620). Here, since the terms information and data are used interchangeably in practice, the concept of the terms will not be specifically limited. However, the drone (100) can temporarily store identification information such as a device ID in the storage unit (640) together with image data of a captured image or sensor data of a sensor, and then retrieve it and generate it in a designated data format through the AI FC unit (630) and transmit it to the drone control service device (130) of FIG. 1.

In addition to the above, the communication interface unit (600), sensor unit (610), control unit (620), AI FC unit (630), and control unit (640) of FIG. 6 can perform various operations, and other detailed information has been sufficiently explained above, so it will be replaced with those contents.

According to an embodiment of the present invention, the communication interface unit (600), sensor unit (610), control unit (620), AI FC unit (630), and control unit (640) of FIG. 6 are configured as physically separate hardware modules, but each module may store and execute software for performing the above operations therein. However, the software is a collection of software modules, and each module may be formed of hardware, so there is no particular limitation on the configuration such as software or hardware. For example, the storage unit (640) may be hardware such as storage or memory. However, since it is also possible to store information (repository) in software, there is no particular limitation on the above.

Meanwhile, as another embodiment of the present invention, the control unit (620) may include a CPU and a memory, and may be formed as a single chip. The CPU may include a control circuit, an operation unit (ALU), a command interpretation unit, and a registry, and the memory may include a RAM. The control circuit may perform a control operation, the operation unit may perform an operation of binary bit information, and the command interpretation unit may perform an operation of converting a high-level language into machine language and vice versa, including an interpreter or a compiler, and the registry may be involved in software data storage. According to the above configuration, for example, at the initial operation of the drone control service device (130), a program stored in the AI FC unit (630) may be copied and loaded into memory, i.e., RAM, and then executed, thereby rapidly increasing the data operation processing speed. In the case of a deep learning model, it may be loaded into the GPU memory instead of the RAM and executed by accelerating the execution speed using the GPU.

FIG. 7 is a drawing for explaining a drone control process according to an embodiment of the present invention.

As illustrated in FIG. 7, the drone (100) according to an embodiment of the present invention can perform a flight along a designated path under the control of a ground control unit (110) or a comprehensive control unit (140) (S700, S710). When a flight is performed under such control, the drone (100) can transmit image data of a video shot related to the flight or sensor data acquired from various types of sensors to the ground control unit (110) or the comprehensive control unit (140) so that the data can be processed. For example, the drone control service unit (130) can display the drone's flight path and flight environment (e.g., location, altitude, weather, etc.) on the metaverse screen (S705, S715).

If an emergency situation, such as an unintended path deviation or communication interruption due to wind or gusts, occurs while the drone (100) is communicating with the ground control unit (110) or the integrated control unit (140), the drone (100) can activate the AI FC installed inside (S720). The AI FC is equipped with an artificial intelligence program for autonomous flight in an emergency situation, and the drone (100) can perform autonomous flight using map data related to the flight path or 3D virtually implemented metaverse data. For example, various information about each location related to the flight path can be matched on the map data or metaverse data. For example, if the flight path of A-B-C-D is designated, the drone (100) may include information about the latitude and longitude or altitude related to point C, and information about terrain features photographed at the corresponding location.

In this way, the drone (100) can perform operations such as takeoff and landing, flight, multiple flight path inspection, and optimized path generation by utilizing the AI FC (S730). For example, in the case of generating an optimized path, when flight is determined to be difficult as a result of analyzing sensor data such as wind speed or wind direction at the current location, a new path is generated based on the analysis results of the captured video, and for example, a new path connecting point C and point D can be generated and flight can be performed. For example, in the embodiment of the present invention, since the flight environment of the drone (100) is exemplified as an environment for forest fire surveillance and an environment such as UAM, various autonomous flight operations may be possible depending on the operating environment of the drone (100).

When data processing by the ground control unit (110) or the integrated control unit (140) is impossible, the drone (100) can directly transmit image data of images captured by a camera or sensor data acquired from various types of sensors to the drone control service unit (130) (S740). In other words, the data transmission path can be changed.

Accordingly, the drone control service device (130) can process the directly transmitted data and implement it on the metaverse screen, and can also transmit the metaverse data to the ground control device (110) or the integrated control device (140) to enable control (S750).

In addition to the above, the drone (100), ground control device (110), drone control service device (130), and integrated control device (140) can perform various operations, and other detailed information has been sufficiently explained above, so it will be replaced with that information.

Figure 8 is a flowchart showing the driving process of the drone of Figure 1.

For convenience of explanation, referring to FIG. 8 together with FIG. 1, an unmanned aerial vehicle (100) according to an embodiment of the present invention, such as an artificial intelligence (AI) drone used for forest fire forecasting or an unmanned aerial vehicle used for UAM, operates the pre-installed AI FC in an emergency situation including a communication cutoff in which a control signal is not received from an external device (e.g., a ground control device or an integrated control device of FIG. 1) (S800). Here, the AI FC may be equipped with an artificial intelligence model or module, and may be used to find a location (e.g., latitude and longitude, etc.) or altitude through autonomous flight based on map data or metaverse data that the unmanned aerial vehicle (100) constructs itself when it unintentionally deviates from a designated path due to an emergency situation such as a gust of wind. Here, the map data can be obtained by securing a video of the area where the drone (100) will fly in advance, dividing it into cells in a grid shape, and storing the physical location, i.e., latitude and longitude information, altitude information, and even object information such as buildings located in the cell, by matching them to each cell. The metaverse data can refer to a simulation environment that implements (or simulates) the current flying drone (100) to prevent collisions with other drones (100) when flying by resetting the flight path, etc. Of course, in the case of map data, it can be applied to operations for collision avoidance like metaverse data, and the map data or metaverse data can be called simulation data according to an embodiment of the present invention. Of course, the AI model can learn various types of data related to flights or flight areas in advance for such operations, and it can be seen that it operates based on the learning results of the learning data.

In this way, the drone (100) can perform operations such as takeoff and landing, flight, multiple flight path check, and optimal path generation depending on the operation of the AI FC, that is, the execution of the AI model (S810). For example, if the drone (100) deviates from the designated path due to a gust of wind while flying on the designated path, the drone (100) can detect the direction and altitude by analyzing the sensor data collected from the various types of sensors it is equipped with and depending on the operation of the AI FC, and can determine how much it has deviated from the existing preset designated path and where the next path to fly is. To this end, it is entirely possible to detect the current location using the map data or metaverse data described above and to generate a new optimal path. For example, if the GPS module is used to communicate with the GPS, it is not difficult to detect the current location, and it is also entirely possible to measure the altitude using an altitude sensor, etc. To set the optimal path, the optimal path can be generated based on pre-learned learning data. The operating environment of the drone (100) according to the embodiment of the present invention is an environment in which multiple drones (100) fly, so that an optimal flight path can be generated or multiple flight paths can be checked while considering issues such as collision avoidance. Of course, in addition, the drone (100) can also determine objects in the captured image captured by the camera, i.e., geographical features (e.g., specific buildings) in an actual physical space, through video or image analysis, and determine the current location or the flight path based on this. Of course, since the information on these geographical features is also matched and stored on the above map data or metaverse data, it is also possible to determine the location by comparison with the corresponding data.

Furthermore, the drone (100) according to an embodiment of the present invention directly transmits flight data according to autonomous flight to a drone control service device (130), such as a cloud server, so that an emergency situation of the drone (100) is implemented on the metaverse screen and data according to autonomous flight is displayed on the screen (S820). Of course, the term "direct" here means that in non-emergency situations, flight is performed by the control of a separate control device, such as the ground control device (110) or the integrated control device (140) of FIG. 1, and data captured by the drone (100) or acquired by sensors is processed accordingly. However, in emergency situations, it may mean that data is transmitted directly through LoRa communication or the like without going through the external device. LoRa communication is known to be capable of communication up to 15 km.

Accordingly, the drone control service device (130) will enable managers of the ground control device (110) or the integrated control device (140) to easily check the flight status of the drone (100) on the metaverse screen even in an emergency situation, thereby ensuring the safety of the flight and the stability of the system, i.e., the service.

Furthermore, the drone (100) according to an embodiment of the present invention can operate in conjunction with an AI drone station (device), for example, in the case of forest fire surveillance, as described above. It flies along a route it should fly according to a designated program, and can transmit captured images or sensor data during the process. In this case, automatic takeoff and landing can be performed to the corresponding AI drone station when an emergency occurs. Furthermore, in the event of a problem with the AI drone station, such as a breakdown or battery replacement, the drone (100) can directly transmit data to notify the user, allowing for repairs to the AI drone station.

In addition to the above, the drone (100) according to the embodiment of the present invention can perform various operations, and other detailed information has been sufficiently explained above, so it will be replaced with that information.

Even though all components constituting the embodiments of the present invention have been described as being combined or operating in combination, the present invention is not necessarily limited to such embodiments. That is, within the scope of the present invention, all of the components may be selectively combined and operated one or more times. In addition, although all of the components may be implemented as individual hardware, some or all of the components may be selectively combined and implemented as a computer program having program modules that perform some or all of the functions of the combined hardware in one or more pieces. The codes and code segments constituting the computer program will be readily inferred by those skilled in the art. Such a computer program may be stored in a non-transitory computer-readable storage medium and read and executed by a computer, thereby implementing the embodiments of the present invention.

Here, the non-transitory readable storage medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the above-described programs may be stored and provided on a non-transitory readable storage medium, such as a CD, DVD, hard disk, Blu-ray disc, USB, memory card, or ROM.

Although embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

100: Unmanned aerial vehicle 110: Ground control device
120: Communication network 130: Drone control service device
140: Integrated control unit 600: Communication interface unit
610: Surveillance Unit 620: Control Unit
630: AI FC section 640: Storage section

Claims (10)

An unmanned aerial vehicle that stores simulation data of map data or metaverse data that enables judgment of the possibility of collision with other unmanned aerial vehicles in relation to a flight path, and when a control signal received from an external device is not detected, measures the flight position and altitude from the stored simulation data using sensor data of sensors including an inertial sensor and an altitude sensor that it has, and performs autonomous flight by finding the position and altitude based on the measurement results;
A drone control service device that implements the real world where the drone flies into a virtual world of the metaverse to control the drone's movements on the screen of the metaverse, and displays the analysis results of the video, photos or sensor data provided by the drone on the screen of the metaverse to enable control; and
A comprehensive control device that implements metaverse data provided by the drone control service device on a screen and controls the flight location and flight environment of the drone on the screen of the metaverse; including,
The above drone control service device operates to display the paths of multiple drones equipped with artificial intelligence (AI) flight controllers (FCs) performing autonomous flight on the screen of the metaverse and to be controlled by the integrated control device.
The above drone operates in an autonomous mode for autonomous flight, executes an AI program mounted on the AI FC, checks takeoff and landing, flight, and multiple flight paths, and generates an optimized path that takes collision avoidance into account to perform flight.
The drone judges that it is flying a normal route by flying at a normal location or altitude when the objects in the currently filmed video and the objects in the map data or metaverse environment are the same.
The above drone is a drone control system that processes captured images or sensor data through the external device when an unexpected situation in which the control signal is not detected does not occur, so that the analysis results are reflected on the screen of the metaverse, and directly transmits data according to the operation of the AI FC when the unexpected situation occurs to the drone control service device so that the analysis results of the data are reflected on the screen of the metaverse. In the first paragraph,
A drone control system in which the drone receives a control signal from a ground control device that controls the drone on the ground as an external device when the above-mentioned emergency situation does not occur, or from a comprehensive control device controlled by a control agent that remotely controls the drone from a long distance, and flies a designated route. delete In the second paragraph,
A drone control system further comprising a station device that allows the drone to take off and land and rapidly charge the drone to a speed higher than a reference speed to enable a task transition when the drone is used for the purpose of forest fire surveillance for forest fire prevention. delete A step in which an unmanned aerial vehicle stores simulation data of map data or metaverse data that enables the determination of the possibility of collision with another unmanned aerial vehicle in relation to a flight path, and when a control signal received from an external device is not detected, measures the flight position and altitude from the stored simulation data using sensor data of a sensor including an inertial sensor and an altitude sensor that the unmanned aerial vehicle has, and performs autonomous flight by finding the position and altitude based on the measurement results;
A step in which a drone control service device implements the real world in which the drone flies into a virtual world of the metaverse to control the movement of the drone on the screen of the metaverse, and displays the analysis results of captured images, photos or sensor data provided by the drone on the screen of the metaverse to enable control;
A step in which the integrated control device implements the metaverse data provided by the drone control service device on the screen and controls the flight location and flight environment of the drone on the screen of the metaverse; and
A step in which the above drone control service device operates to display the paths of a plurality of drones equipped with an artificial intelligence (AI) flight controller (FC) performing the autonomous flight on the screen of the metaverse so that the paths are controlled by the integrated control device; including,
A step in which the drone operates in an autonomous mode for autonomous flight, executes an AI program mounted on the AI FC, checks takeoff and landing, flight, and multiple flight paths, and generates an optimized path that takes collision avoidance into account to perform flight;
A step of determining that the drone is flying a normal path by flying at a normal location or altitude when the objects in the currently filmed video and the objects in the map data or metaverse environment are the same;
A step for processing the captured video or sensor data through the external device when an unexpected situation in which the control signal is not detected by the drone does not occur, so that the analysis result is reflected on the screen of the metaverse; and
A drone control method further comprising a step of directly transmitting data according to the operation of the AI FC when the above-mentioned emergency situation occurs to the drone control service device so that the analysis results of the data are reflected on the screen of the metaverse. In paragraph 6,
A method for controlling a drone, further comprising: a step of receiving a control signal from a ground control device that controls the drone on the ground as an external device when the unexpected situation does not occur, or a comprehensive control device controlled by a control agent that remotely controls the drone from a long distance, and flying the drone along a designated path. delete In paragraph 7,
A method for controlling a drone, further comprising: a step of causing the station device to take off and land when the drone is used for the purpose of forest fire surveillance for forest fire prevention, and rapidly charging the drone to a speed higher than a reference speed to enable a task transition; delete