JP7643678B1 - AUTONOMOUS DRIVING SYSTEM, AUTONOMOUS DRIVING METHOD, MOBILE BODY, AND AUTONOMOUS DRIVING PROGRAM - Google Patents

Abstract

[Problem] To achieve safer automated driving.
[Solution] A target moving object (120) refers to its own position data and map data to determine whether monitoring is required, and transmits a monitoring request along with driving condition data. A surrounding condition monitoring device (130) receives the monitoring request, monitors the surrounding conditions of the target moving object, and transmits monitoring data. An automatic driving control device (400) receives the driving condition data and the monitoring data, determines a driving obstacle for the target moving object based on the driving condition data and the monitoring data, and transmits control data to the target moving object.
[Selected Figure] Figure 1

Description

This disclosure relates to a system for achieving safe autonomous driving.

There is a shortage of drivers for delivery trucks and taxis, etc., which is a problem that can be resolved with the realization of autonomous driving.
To realize autonomous driving, it is necessary to be able to check for obstacles in the blind spots of autonomous vehicles.

Patent Document 1 discloses an information provision system for compensating for blind spots of an autonomous vehicle to prevent collision with a moving object such as an oncoming vehicle.
In this information providing system, roadside devices installed near roads detect moving objects.

JP 2023-50627 A

Patent Document 1 does not mention the timing at which the roadside device starts detecting a moving object, and it is assumed that the roadside device is constantly detecting a moving object.
As a result, moving objects are detected even during times when there is no risk of collision with the moving object, which requires communication and processing of a huge amount of information, including unnecessary detection information, and increases the load on the information provision system.

The present disclosure aims to achieve safer autonomous driving by monitoring the surroundings of an autonomous vehicle using sensors installed independently of the autonomous vehicle when the autonomous vehicle passes through an area where there is a blind spot from the autonomous vehicle.

The autonomous driving system of the present disclosure includes a target moving body that is an autonomous moving body, a surrounding situation monitoring device, and an autonomous driving control device.
The target moving body is
A group of mobile sensors;
an on-board device that determines whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the target mobile body, and map data indicating a monitoring area set as an area in which the surrounding situation of the target mobile body should be monitored by the surrounding situation monitoring device when the target mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, transmits a monitoring request to the surrounding situation monitoring device and transmits driving situation data obtained by the mobile body sensor group and indicating the driving situation of the target mobile body to the automatic driving control device;
Equipped with
The surrounding situation monitoring device is
A monitoring sensor group;
a monitoring control device that receives the monitoring request, causes the monitoring sensor group to monitor the surrounding situation of the target moving object, and transmits monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device;
Equipped with
The autonomous driving control device receives the driving status data and the monitoring data, determines a driving obstacle of the target moving body based on the driving status data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle,
The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding condition data obtained by the group of moving object sensors and indicating the surrounding condition of the target moving object.

According to the present disclosure, when an autonomous vehicle passes through an area where there is a blind spot from the autonomous vehicle, the surroundings of the autonomous vehicle can be monitored using a sensor installed independently of the autonomous vehicle, thereby making it possible to achieve safer autonomous driving.

FIG. 1 is a configuration diagram of an autonomous driving system 100 according to a first embodiment. FIG. 2 is a configuration diagram of an in-vehicle device 200 according to the first embodiment. FIG. 2 is a configuration diagram of a monitoring control device 300 according to the first embodiment. FIG. 2 is a configuration diagram of an automatic driving control device 400 according to the first embodiment. 4 is a flowchart of the vehicle-mounted device 200 according to the first embodiment. 4 is a flowchart of the monitoring control device 300 in the first embodiment. 4 is a flowchart of the autonomous driving control device 400 in the first embodiment. FIG. 11 is a configuration diagram of an autonomous driving system 100 according to a second embodiment. 13 is a flowchart of the vehicle-mounted device 200 according to the second embodiment. 13 is a flowchart of the monitoring control device 300 in the second embodiment. FIG. 11 is a configuration diagram of an autonomous driving system 100 according to a third embodiment. 13 is a flowchart of a monitoring control device 300 according to the third embodiment. 13 is a flowchart of the autonomous driving control device 400 in embodiment 4. 13 is a flowchart of the vehicle-mounted device 200 according to the fourth embodiment. FIG. 13 is a configuration diagram of an autonomous driving system 100 according to a fifth embodiment. FIG. 13 is a configuration diagram of an automatic driving control device 400 in embodiment 5. 13 is a flowchart of the autonomous driving control device 400 in embodiment 5. FIG. 23 is a diagram showing a part of the functional configuration of an autonomous driving control device 400 in a sixth embodiment. A figure showing an example of a video stream AI processing pipeline 439 in embodiment 6. A figure showing an example of a video stream AI processing pipeline 439 in embodiment 6. A figure showing an example of a video stream AI processing pipeline 439 in embodiment 6.

In the embodiments and drawings, the same or corresponding elements are given the same reference numerals. Explanations of elements given the same reference numerals as elements described above will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate data flow or processing flow.

Embodiment 1.
The autonomous driving system 100 will be described with reference to FIGS. 1 to 7. FIG.

***Configuration Description***
The configuration of an autonomous driving system 100 will be described with reference to FIG.
The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110.

The target moving body 120 is a moving body that travels on the ground by autonomous driving.
For example, the target moving object 120 is a car, a bus, or a taxi that operates autonomously.

The target moving object 120 includes an in-vehicle device 200 and a moving object sensor group 290 .
The moving body sensor group 290 is one or more types of sensors mounted on the target moving body 120 .
For example, the mobile sensor group 290 includes a depth camera 291 , a Lidar 292 , and a positioning sensor 293 .
The positioning sensor 293 is a sensor that measures the position of the target moving object 120. For example, the positioning sensor 293 performs positioning using a satellite positioning system. An example of the satellite positioning system is the Global Positioning System (GPS).
The mobile object sensor group 290 is used to grasp the traveling conditions of the target mobile object 120 and the conditions around the target mobile object 120 .
The driving conditions include position, speed, and acceleration.
The surroundings include the positions of objects present in the surroundings.

The configuration of the vehicle-mounted device 200 will be described with reference to FIG.
The vehicle-mounted device 200 is a computer including hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input/output interface 205. These pieces of hardware are connected to each other via signal lines.

The processor 201 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 201 is a CPU.
IC is an abbreviation for Integrated Circuit.
CPU is an abbreviation for Central Processing Unit.

The memory 202 is a volatile or non-volatile storage device. The memory 202 is also called a primary storage device or a main memory. For example, the memory 202 is a RAM. Data stored in the memory 202 is saved in the secondary storage device 203 as necessary.
RAM is an abbreviation for Random Access Memory.

The auxiliary storage device 203 is a non-volatile storage device. For example, the auxiliary storage device 203 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed.
ROM is an abbreviation for Read Only Memory.
HDD is an abbreviation for Hard Disk Drive.

The auxiliary storage device 203 is a receiver and a transmitter. For example, the auxiliary storage device 203 is a communication chip or a NIC. The communication of the vehicle-mounted device 200 is performed using a communication device 204.
NIC is an abbreviation for Network Interface Card.

The input/output interface 205 is a port to which an input device and an output device are connected. For example, a mobile sensor group 290 is connected to the input/output interface 205.

The vehicle-mounted device 200 includes elements such as a monitoring request unit 211, a control data acquisition unit 212, and a driving control unit 213. These elements are realized by software.

The auxiliary storage device 203 stores an automatic driving program for causing the computer to function as a monitoring request unit 211, a control data acquisition unit 212, and a driving control unit 213. The automatic driving program is loaded into the memory 202 and executed by the processor 201.
The auxiliary storage device 203 further stores an OS. At least a part of the OS is loaded into the memory 202 and executed by the processor 201.
The processor 201 executes an automatic driving program while executing the OS.
OS is an abbreviation for Operating System.

Data for the automatic driving program (input data, output data, etc.) is stored in memory unit 220.
The memory 202 functions as the storage unit 220. However, a storage device such as the auxiliary storage device 203, a register in the processor 201, or a cache memory in the processor 201 may function as the storage unit 220 instead of the memory 202 or together with the memory 202.

The autonomous driving program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

Returning to FIG.
The surrounding situation monitoring device 130 is a device that monitors the surrounding situation of the target moving object 120 .
For example, the surrounding situation monitoring device 130 is a drone or a roadside unit. A drone is an aircraft that flies autonomously in the sky. A roadside unit is a device that is installed on a road in a monitoring area. The monitoring area will be described later.

The surrounding situation monitoring device 130 includes a monitoring control device 300 and a group of monitoring sensors 390 .
The monitoring sensor group 390 is one or more types of sensors mounted on the surrounding situation monitoring device 130 .
For example, the monitoring sensor group 390 includes a depth camera 391 and an RGB camera 392 .
The monitoring sensors 390 are used to monitor the surroundings of the target moving object 120 .

The configuration of the monitoring and control device 300 will be described with reference to FIG.
The monitoring and control device 300 is a computer including hardware such as a processor 301, a memory 302, an auxiliary storage device 303, a communication device 304, and an input/output interface 305. These pieces of hardware are connected to each other via signal lines.

The processor 301 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 301 is a CPU.
The memory 302 is a volatile or non-volatile storage device. The memory 302 is also called a primary storage device or a main memory. For example, the memory 302 is a RAM. Data stored in the memory 302 is saved in the secondary storage device 303 as necessary.
The auxiliary storage device 303 is a non-volatile storage device. The auxiliary storage device 303 is also called storage. For example, the auxiliary storage device 303 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 303 is loaded into the memory 302 as needed.
The communication device 304 is a receiver and a transmitter. For example, the communication device 304 is a communication chip or a NIC. The communication of the monitor and control device 300 is performed using the communication device 304.
The input/output interface 305 is a port to which an input device and an output device are connected.

The monitoring control device 300 includes elements such as a request receiving unit 311, a monitoring control unit 312, a monitoring data generating unit 313, and a monitoring data providing unit 314. These elements are realized by software.

The auxiliary storage device 303 stores a monitoring program for causing the computer to function as a request receiving unit 311, a monitoring control unit 312, a monitoring data generating unit 313, and a monitoring data providing unit 314. The monitoring program is loaded into the memory 302 and executed by the processor 301.
The auxiliary storage device 303 further stores an OS. At least a part of the OS is loaded into the memory 302 and executed by the processor 301.
The processor 301 executes a monitoring program while executing the OS.

Input and output data of the monitoring program is stored in the storage unit 320 .
The memory 302 functions as the storage unit 320. However, a storage device such as the auxiliary storage device 303, a register in the processor 301, or a cache memory in the processor 301 may function as the storage unit 320 instead of the memory 302 or together with the memory 302.

The monitoring program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

Returning to FIG. 1, the explanation will be continued.
The autonomous driving control device 400 is a device that manages and controls the autonomous driving of the target moving body 120.

The configuration of the automatic driving control device 400 will be described with reference to FIG.
The autonomous driving control device 400 is a computer including hardware such as a processor 401, a memory 402, an auxiliary storage device 403, a communication device 404, and an input/output interface 405. These pieces of hardware are connected to each other via signal lines.

The processor 401 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 401 is a CPU.
The memory 402 is a volatile or non-volatile storage device. The memory 402 is also called a primary storage device or a main memory. For example, the memory 402 is a RAM. Data stored in the memory 402 is saved in the secondary storage device 403 as necessary.
The auxiliary storage device 403 is a non-volatile storage device. The auxiliary storage device 403 is also called storage. For example, the auxiliary storage device 403 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 403 is loaded into the memory 402 as needed.
The communication device 404 is a receiver and a transmitter. For example, the communication device 404 is a communication chip or a NIC. Communication of the autonomous driving control device 400 is performed using the communication device 404.
The input/output interface 405 is a port to which an input device and an output device are connected.

The autonomous driving control device 400 includes elements such as a data reception unit 411, a driving obstacle determination unit 412, a control data generation unit 413, and a control data provision unit 414. These elements are realized by software.

The auxiliary storage device 403 stores a control program for causing the computer to function as a data receiving unit 411, a driving obstacle determining unit 412, and a control data generating unit 413. The control program is loaded into the memory 402 and executed by the processor 401.
The auxiliary storage device 403 further stores an OS. At least a part of the OS is loaded into the memory 402 and executed by the processor 401.
The processor 401 executes a control program while executing the OS.

Input and output data for the control program are stored in memory unit 420.
The memory 402 functions as the storage unit 420. However, a storage device such as the auxiliary storage device 403, a register in the processor 401, or a cache memory in the processor 401 may function as the storage unit 420 instead of the memory 402 or together with the memory 402.

The control program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

Returning to FIG. 1, the explanation will be continued.
The map database 110 is a device that manages the map data 111 .
The map data 111 will be described later.

*** Operation Description ***
The operation procedure of the automatic driving system 100 corresponds to an automatic driving method. The operation procedure of the automatic driving system 100 corresponds to a processing procedure by an automatic driving system program. The automatic driving system program includes an automatic driving program, a monitoring program, and a control program.

The following describes the operation of the vehicle-mounted device 200. First, the premise of the operation of the vehicle-mounted device 200 will be described.
The mobile body sensor group 290 periodically observes the traveling conditions of the target mobile body 120 and the surrounding conditions of the target mobile body 120. For example, a depth camera 291 measures the surroundings of the target mobile body 120, a lidar 292 measures the distance to each point around the target mobile body 120, and a positioning sensor 293 performs satellite positioning.
The mobile sensor group 290 provides driving condition data and surrounding condition data.
The traveling condition data is data that indicates the traveling condition of the target moving object 120. The traveling condition data indicates the position, speed, acceleration, etc. of the target moving object 120. The position of the target moving object 120 is referred to as the self-position. Data that indicates the self-position is referred to as self-position data.
The surrounding situation data is data that indicates the surrounding situation of the target moving object 120. The surrounding situation data indicates various objects and the like observed by the moving object sensor group 290 mounted on the target moving object 120.
The driving control unit 213 controls the driving of the target moving object 120 based on the driving plan data and the surrounding condition data.
The driving plan data indicates the planned driving route of the target moving object 120 .

The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG.
Steps S121 to S125 are repeatedly executed.

In step S121, the monitoring request unit 211 refers to the self-location data and the map data 111 and determines whether or not monitoring by the surrounding situation monitoring device 130 is required.
The map data 111 is obtained by communicating with the map database 110 .

The map data 111 indicates one or more surveillance areas.
The monitoring area is set as an area in which the surrounding conditions of the target moving object 120 should be monitored by the surrounding conditions monitoring device 130 when the target moving object 120 passes through.
Specifically, the monitoring area is a place where there is a blind spot from the target moving object 120, such as the area before and after a curve.

Whether or not monitoring by the surrounding situation monitoring device 130 is required is determined as follows.
The monitoring request unit 211 determines whether the target moving object 120 has entered the monitoring area.
For example, the monitoring request unit 211 maps the position of the target moving object 120 on the map shown in the map data 111. Then, the monitoring request unit 211 determines whether the position of the target moving object 120 has been mapped within the monitoring area.
When the target moving object 120 enters a monitoring area, the monitoring request unit 211 determines that monitoring by the surrounding situation monitoring device 130 is necessary.

If it is determined that monitoring by the surrounding conditions monitoring device 130 is necessary, processing proceeds to step S122.

In step S<b>122 , the monitoring request unit 211 transmits a monitoring request to the surrounding situation monitoring device 130 .
The monitoring request is data for requesting the surrounding situation monitoring device 130 to monitor the surrounding situation of the target moving object 120 .
The monitoring request indicates the identifier of the target mobile object 120, the position of the target mobile object 120, the range of the monitoring area, and the like.

In step S123, the control data acquisition unit 212 transmits the driving plan data and driving condition data to the autonomous driving control device 400.

Steps S122 and S123 may be performed in reverse order.

In step S124, the control data acquisition unit 212 receives the control data transmitted from the autonomous driving control device 400.
The control data is data for notifying a driving obstacle of the target moving body 120 .
For example, the control data indicates the positions of objects (obstacles) that impede the travel of the target moving body 120 .

In step S125, the driving control unit 213 controls the travel of the target moving body 120 based on the control data and the surrounding condition data.

The travel of the target moving object 120 is controlled as follows.
First, the driving control unit 213 changes the driving plan. For example, the driving control unit 213 changes the planned driving route to a route that can avoid the obstacles indicated in the control data and the objects indicated in the surrounding situation data.
Next, the driving control unit 213 determines control (driving control) for the driving device of the target moving body 120 so that the target moving body 120 can travel according to the changed travel plan. Examples of the driving device are a steering device (handle) and a motor (engine).
Then, the driving control unit 213 controls the driving device of the target moving object 120 according to the determined driving control.

The procedure of the operation of the monitoring and control device 300 will be described with reference to FIG.
Steps S131 to S134 are executed every time a monitoring request is transmitted from the target moving object 120.

In step S131, the request receiving unit 311 receives a monitoring request sent from the target mobile object 120.

In step S132, the monitoring control unit 312 causes the monitoring sensor group 390 to monitor the surrounding conditions of the target moving object 120.

The surroundings of the target moving object 120 are monitored as follows.
The monitoring control unit 312 outputs an operation command to the monitoring sensor group 390 .
The monitoring sensor group 390 starts operating upon receiving an operation command. For example, the depth camera 391 and the RGB camera 392 each capture images of the surroundings of the target moving object 120.

Sensor data is obtained by the monitoring sensor group 390 monitoring the surrounding conditions of the target moving object 120.

In step S<b>133 , the monitoring data generation unit 313 generates monitoring data using the sensor data obtained from the monitoring sensor group 390 .
The monitoring data is data that indicates the surrounding conditions of the target moving object 120. The monitoring data indicates the positions of various objects detected by the monitoring sensor group 390, etc.

The monitoring data is generated as follows:
First, the monitoring data generating unit 313 executes object detection using the sensor data at each time, and detects each object present around the target moving object 120 .
Next, the monitoring data generation unit 313 calculates the speed of each object based on the position of each object at each time, and predicts the movement path of each object.
The monitoring data generator 313 then generates data indicating the position, speed, and predicted path of each detected object. The generated data is monitoring data.

In step S134, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

The procedure of the operation of the automatic driving control device 400 will be described with reference to FIG.
Steps S141 to S145 are executed each time the target moving object 120 transmits driving plan data and driving status data and the surrounding status monitoring device 130 transmits monitoring data.

In step S141, the data reception unit 411 receives driving plan data and driving status data transmitted from the target moving body 120.

In step S142, the data reception unit 411 receives the monitoring data transmitted from the surrounding situation monitoring device 130.

Steps S141 and S142 may be performed in reverse order.

In step S143, the driving obstacle determination unit 412 determines whether or not there is a driving obstacle for the target moving body 120 based on the driving condition data and the monitoring data.

The presence or absence of a travel obstacle for the target moving object 120 is determined as follows.
The traveling obstacle determination unit 412 determines whether an obstacle facing the target moving body 120 exists within a proximity range from the position of the target moving body 120. The proximity range is a range that has the position of the target moving body 120 as its base point, and the size of the proximity range is determined in advance.
The position of the target moving body 120 and the position of each object are managed using a position database. The position database is managed by the autonomous driving control device 400. The driving obstacle determination unit 412 maps the position of the target moving body 120 and the position of each object on a map shown in the position database. Then, the driving obstacle determination unit 412 makes a determination by referring to the map on which the position of the target moving body 120 and the position of each object are mapped.
If there is an obstacle facing the target moving body 120 within a proximity range from the position of the target moving body 120, the traveling obstacle determination unit 412 determines that there is an obstacle to the traveling of the target moving body 120.

If it is determined that there is an obstacle to the running of the target moving object 120, the process proceeds to step S144.
If it is determined that there is no obstacle to the running of the target moving body 120, the processing ends.

In step S144, the control data generation unit 413 generates control data to notify of a driving disorder of the target moving body 120.

In step S145, the control data providing unit 414 transmits the control data to the target moving body 120.

***Advantages of First Embodiment***
According to the first embodiment, when the target moving object 120 passes through a monitoring area where a blind spot occurs from the target moving object 120, the surroundings of the target moving object 120 can be monitored using the monitoring sensor group 390 provided independently of the target moving object 120. This makes it possible to realize safer automated driving.

***Additional Notes to the First Embodiment***
The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110.
The target moving object 120 is a moving object capable of autonomous driving. The target moving object 120 has sensors for detecting the surrounding environment, such as an optical sensor and a LIDAR, and a communication function.
The surrounding situation monitoring device 130 has sensors such as a camera that exists independently of the target moving object 120, and a communication function.
The autonomous driving control device 400 has communication and calculation functions, and notifies the target moving body 120 of the control data necessary for planning a driving plan.
The map database 110 has a storage function and a communication function. The map database 110 stores map data 111 used to determine the necessity of checking the surrounding situation. The map database 110 can communicate with the target moving object 120, the surrounding situation monitoring device 130, and the automatic driving control device 400.
The target moving object 120 transmits information such as its own position, speed, and acceleration to the autonomous driving control device 400. The target moving object 120 also maps its own position on the map shown in the map data 111 and determines whether or not it is necessary to monitor the surrounding conditions. If it is necessary to monitor the surrounding conditions, the target moving object 120 transmits a monitoring request to the surrounding conditions monitoring device 130.
The surrounding situation monitoring device 130 may be fixed to the roadside, or may be an aircraft launched from the target moving object 120. The surrounding situation monitoring device 130 acquires information such as the position, speed, and acceleration of an obstacle that affects the travel plan of the target moving object 120 from the monitoring sensor group 390. The surrounding situation monitoring device 130 then transmits the information such as the position, speed, and acceleration of the obstacle to the automatic driving control device 400. The surrounding situation monitoring device 130 also maps the position of the obstacle on the map shown in the map data 111.
The autonomous driving control device 400 determines a driving obstacle for the target moving body 120 based on information such as the position (including future position) of the target moving body 120 and information such as the position (including future position) of an obstacle. Then, the autonomous driving control device 400 generates control data based on the determination result of the driving obstacle and transmits the control data to the target moving body 120.

Embodiment 2.
Regarding an embodiment in which the surrounding situation monitoring device 130 is a drone 131, the differences from the first embodiment will be mainly described with reference to Figs. 8 to 10 .

***Configuration Description***
The configuration of the autonomous driving system 100 will be described with reference to FIG.
The autonomous driving system 100 is equipped with a drone 131 as a surrounding situation monitoring device 130.
Drone 131 is an autonomous flying object that flies in the sky.
In addition to the elements described in embodiment 1, the target moving body 120 is equipped with a drone port 121.
The drone port 121 is a facility where the processor 301 is stored and where the processor 301 takes off and lands. The drone port 121 has a charging port for charging the drone 131.

*** Operation Description ***
The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG.
Steps S221 to S225 correspond to steps S121 to S125 in the first embodiment.

In step S221, the monitoring request unit 211 refers to the self-location data and the map data 111 and determines whether or not monitoring by the drone 131 is required.

If it is determined that monitoring by drone 131 is necessary, processing proceeds to step S222.

In step S222, the monitoring request unit 211 sends a monitoring request to the drone 131.

The monitoring request includes self-location data and monitoring area data.
The monitoring area data indicates the monitoring area through which the target mobile object 120 passes.

In step S223, the control data acquisition unit 212 transmits the driving plan data and driving condition data to the autonomous driving control device 400.

In step S224, the control data acquisition unit 212 receives the control data transmitted from the autonomous driving control device 400.

In step S225, the driving control unit 213 controls the travel of the target moving body 120 based on the control data and the surrounding condition data.

The procedure of the operation of the monitoring and control device 300 will be described with reference to FIG.
Steps S231 to S234 correspond to steps S131 to S134 in the first embodiment.

In step S231, the request receiving unit 311 receives a monitoring request sent from the target mobile object 120.

The monitoring request includes self-location data and monitoring area data.

In step S232, the monitoring control unit 312 causes the monitoring sensor group 390 to monitor the surrounding conditions of the target moving body 120.

The monitoring control unit 312 causes the drone 131 to fly above the monitoring area in order to have the monitoring sensor group 390 monitor the surrounding conditions of the target moving object 120.

Drone 131 flies over the surveillance area as follows.
First, the monitoring control unit 312 plans a flight route for the drone 131 based on the self-location data and the monitored area data.
The planned flight route is a route from taking off from the drone port 121 of the target moving body 120, flying over the monitored area, and landing at the drone port 121 of the target moving body 120.
Then, the monitoring control unit 312 causes the drone 131 to fly along the planned flight path. To this end, the monitoring control unit 312 controls the driving device of the drone 131.

In step S233, the monitoring data generation unit 313 generates monitoring data using sensor data obtained from the monitoring sensor group 390.

In step S234, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

***Advantages of the Second Embodiment***
According to the second embodiment, it becomes possible to monitor from the sky a monitoring area that is a blind spot from the target moving object 120 traveling on the ground.

***Additional Information about the Second Embodiment***
The ambient condition monitoring device 130 is an autonomous flying vehicle.
The autonomous flying object receives a monitoring request, calculates a flight plan based on the information on the monitoring area that triggered the monitoring request, and starts flying. The autonomous flying object obtains the position, speed, and acceleration of an obstacle from data obtained from an on-board sensor, and transmits the position, speed, and acceleration of the obstacle to the autonomous driving control device 400.
The target moving body 120 has a port for the autonomous flying object to take off and land. The target moving body 120 acquires position information of an area where a blind spot occurs from the map data 111, and launches the autonomous flying object when it enters the area where a blind spot occurs. The autonomous flying object then monitors the situation in the blind spot area to determine the position, speed, and acceleration of an obstacle, and transmits the position, speed, and acceleration of the obstacle to the autonomous driving control device 400.

Embodiment 3.
The embodiment in which the position of an obstacle is detected using a color sensor will be described below with reference to Figs. 11 and 12, focusing mainly on the differences from the first embodiment.

***Configuration Description***
The configuration of the autonomous driving system 100 will be described with reference to FIG.
In the surrounding situation monitoring device 130 , the monitoring sensor group 390 includes a color sensor 393 .
The color sensor 393 is an RGB camera or an RGB-D camera.
R stands for red, G stands for green, B stands for blue, and D stands for depth.

*** Operation Description ***
The procedure of the operation of the monitoring and control device 300 will be described with reference to FIG.
Steps S331 to S334 correspond to steps S131 to S134 in the first embodiment.

In step S331, the request receiving unit 311 receives a monitoring request sent from the target mobile object 120.

In step S332, the monitoring control unit 312 causes the monitoring sensor group 390 to monitor the surrounding conditions of the target moving body 120.

In step S333, the monitoring data generation unit 313 generates monitoring data using sensor data obtained from the monitoring sensor group 390.

The monitoring data indicates the positions of each object present around the target moving body 120.

The position of each object is estimated as follows: The sensor data obtained by the color sensor 393 is referred to as color sensor data.
First, the monitoring data generation unit 313 executes object detection using color sensor data to detect objects present around the target moving body 120 and the plane on which the objects are located.
The monitoring data generating unit 313 then estimates the position of the object on the detected plane.

In step S334, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

***Advantages of Third Embodiment***
According to the third embodiment, it becomes possible to more accurately detect the position of each object that may become an obstacle.

***Additional Notes on the Third Embodiment***
If the connected sensor is an RGB sensor, the surroundings monitoring device 130 detects a horizontal plane and estimates the coordinates of the detected obstacle relative to the detected horizontal plane. However, the estimation may also be performed by the autonomous driving control device 400.
When the sensor that exists independently of the target moving object 120 is a camera that obtains RGB information, the surrounding situation monitoring device 130 performs object detection, detects a plane based on the RGB information, and estimates the position of the object relative to the detected plane.
When the sensor that exists independently of the target moving body 120 is a camera that obtains RGB-D information, the surrounding situation monitoring device 130 performs object detection, detects a plane based on the RGB-D information, and estimates the position of the object relative to the detected plane.

The third embodiment may be implemented in combination with the second embodiment. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving object 120 may be equipped with a drone port 121.

Embodiment 4.
The manner in which the target moving body 120 deals with a travel obstacle depending on the type of the obstacle will be described below with reference to Figs. 13 and 14, mainly with respect to the points that differ from the first embodiment.

***Configuration Description***
The configuration of the autonomous driving system 100 is similar to that in the first embodiment.
However, the map data 111 indicates one or more escape zones.

*** Operation Description ***
The procedure of the operation of the automatic driving control device 400 will be described with reference to FIG.
Steps S441 to S445 correspond to steps S141 to S145 in the first embodiment.

In step S441, the data reception unit 411 receives the driving plan data and driving condition data transmitted from the target moving body 120.

In step S442, the data reception unit 411 receives monitoring data transmitted from the surrounding situation monitoring device 130.

In step S443, the driving obstacle determination unit 412 determines whether or not there is a driving obstacle for the target moving body 120 based on the driving condition data and the monitoring data.

When it is determined that there is a traveling obstacle for the target moving body 120, the traveling obstacle determination unit 412 determines the type of obstacle based on the monitoring data. Specifically, the traveling obstacle determination unit 412 determines whether the obstacle is a moving object or a non-moving object. Examples of moving objects are vehicles and people. An example of a non-moving object is falling rocks. The traveling obstacle determination unit 412 also determines whether there are multiple obstacles. An example of multiple obstacles is multiple pedestrians.
In addition, the driving obstacle determination unit 412 determines the type of road (target road) on which the target moving object 120 and the obstacle exist, based on the position database. Specifically, the driving obstacle determination unit 412 determines whether the target road is a narrow road having a width of only one lane.

If it is determined that there is an obstacle to the running of the target moving body 120, the process proceeds to step S444.
If it is determined that there is no obstacle to the running of the target moving body 120, the processing ends.

In step S444, the control data generation unit 413 generates control data to notify of a driving disorder of the target moving body 120.

The control data indicates the type of driving obstruction, etc.

The type of driving impairment is determined as follows:
If the target road is a narrow road and the obstacle is a moving object, the control data generating unit 413 determines that the type of driving obstacle is "impossible to pass each other.""Impossible to pass each other" means that the target moving body 120 and the obstacle cannot pass each other when traveling.
If the target road is a narrow road and the obstacle is a stationary object, the control data generating unit 413 determines that the type of driving obstacle is "impassable.""Impassable" means that the target moving body 120 cannot pass through the obstacle by avoiding it.
When multiple obstacles are present, the control data generating unit 413 transmits control data indicating "slow down required" as the control information. "Slow down required" means that slow down is required to avoid the obstacles.

In step S445, the control data providing unit 414 transmits the control data to the target moving body 120.

The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG.
Steps S421 to S425 correspond to steps S121 to S125 in the first embodiment.

In step S421, the monitoring request unit 211 refers to the self-location data and the map data 111 and determines whether monitoring by the surrounding situation monitoring device 130 is required.

If it is determined that monitoring by the surrounding conditions monitoring device 130 is necessary, processing proceeds to step S422.

In step S422, the monitoring request unit 211 sends a monitoring request to the surrounding situation monitoring device 130.

In step S423, the control data acquisition unit 212 transmits the driving plan data and driving condition data to the autonomous driving control device 400.

In step S424, the control data acquisition unit 212 receives the control data transmitted from the autonomous driving control device 400.

The control data indicates the type of driving obstacle for the target moving body 120, etc.

In step S425, the driving control unit 213 controls the travel of the target moving body 120 based on the control data and the surrounding condition data.

The travel of the target moving object 120 is controlled according to the type of travel obstacle as follows.
If the type of driving obstacle is “no passing”, the driving control unit 213 causes the target moving body 120 to evacuate to the nearest escape zone shown in the map data 111.
If the type of driving obstacle is “impassable”, the driving control unit 213 makes the target moving body 120 make a U-turn using the nearest escape zone shown in the map data 111 .
If the type of driving obstacle is "requires slowing down," the driving control unit 213 causes the target moving body 120 to drive slowly.

***Advantages of the Fourth Embodiment***
According to the fourth embodiment, it becomes possible to realize safe automatic driving according to the type of driving obstacle.

***Additional Notes on the Fourth Embodiment***
The autonomous driving control device 400 obtains information on the position, speed, and acceleration from the target moving body 120, and obtains information on the position, speed, and acceleration of an obstacle from the surroundings monitoring device 130. The autonomous driving control device 400 records future position information calculated based on the respective position information of the target moving body 120 and the obstacle, together with the predicted time, in a database having version information and space/time axis information.
Each time the position information and future predicted positions of the target moving body 120 and the obstacle are changed, the autonomous driving control device 400 updates the version and updates the position information of the target moving body 120 and the obstacle. If the position of the target moving body 120 is in a close relationship with the position of the obstacle at the time of updating, the autonomous driving control device 400 determines that there is a driving obstacle, and the target moving body 120 modifies the driving plan.
When the type of driving obstacle is determined to be "no passing", the target moving body 120 acquires the position information of the escape zone from the map data 111 and changes the driving plan to retreat to the escape zone.
If the type of driving obstacle is determined to be "impassable", the target moving body 120 acquires the position information of the escape zone from the map data 111 and changes the driving plan to make a U-turn using the escape zone.
If the type of driving obstacle is determined to be "required to drive slowly," the target moving body 120 changes its driving plan to drive slowly.

The fourth embodiment may be implemented in combination with at least one of the second and third embodiments. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving object 120 may include a drone port 121. The monitoring control device 300 may also detect the position of an obstacle using a color sensor 393.

Embodiment 5.
The embodiment of notifying an obstacle of a travel obstacle of the target moving body 120 will be described with reference to Figs. 15 to 17, mainly focusing on the points different from the first embodiment.

***Configuration Description***
The configuration of the autonomous driving system 100 will be described with reference to FIG. 15 .
The autonomous driving system 100 includes one or more mobile objects 101 and one or more mobile terminals 102 .
The moving body 101 is a moving body similar to the target moving body 120 .
The mobile terminal 102 is carried by a pedestrian. For example, the mobile terminal 102 is a smartphone. An application program for the autonomous driving system 100 is executed on the smartphone.
Information on each moving body 101 and information on each mobile terminal 102 are registered in advance in the automatic driving control device 400.

The configuration of the automatic driving control device 400 will be described with reference to FIG.
The autonomous driving control device 400 further includes an element called a driving obstacle notification unit 415.
The autonomous driving control program further causes the computer to function as a driving obstacle notification unit 415.

*** Operation Description ***
The following describes the operation of the autonomous driving control device 400. First, the premise of the operation of the autonomous driving control device 400 will be described.
Each moving object 101 periodically transmits its own position data to the autonomous driving control device 400. The own position data indicates the position of the moving object 101.
Each mobile terminal 102 periodically transmits its own position data to the autonomous driving control device 400. The own position data indicates the positions of the pedestrian and the mobile terminal 102.

The procedure of the operation of the autonomous driving control device 400 will be described with reference to FIG.
Steps S542 to S546 correspond to steps S141 to S145 in the first embodiment.

In step S541, the data receiving unit 411 receives self-location data from each moving object 101.
Furthermore, the data receiving unit 411 receives self-location data from each mobile terminal 102 .

In step S542, the data reception unit 411 receives the driving plan data and driving status data transmitted from the target moving body 120.

In step S543, the data reception unit 411 receives the monitoring data transmitted from the surrounding situation monitoring device 130.

In step S544, the driving obstacle determination unit 412 determines whether or not there is a driving obstacle for the target moving body 120 based on the driving condition data and the monitoring data.

In step S545, the control data generation unit 413 generates control data to notify of a driving disorder of the target moving body 120.

In step S546, the control data providing unit 414 transmits the control data to the target moving body 120.

In step S547, the traveling obstacle notification unit 415 determines whether the obstacle is any one of the moving bodies 101 based on the self-position data of each of the one or more moving bodies 101.
If it is determined that the obstacle is one of the moving bodies 101, the driving obstacle notification unit 415 transmits notification data to the moving body 101 that is the obstacle.
The notification data is data for notifying a traveling obstacle of the target moving body 120. For example, the traveling obstacle notification unit 415 generates control data for the moving body 101, which is an obstacle, as the notification data.

In addition, the driving obstacle notification unit 415 determines whether the obstacle is a pedestrian carrying one of the mobile terminals 102 based on the self-position data of each of the one or more mobile terminals 102 .
When it is determined that the obstacle is a pedestrian carrying one of the mobile terminals 102, the driving obstacle notification unit 415 transmits notification data to the mobile terminal 102 of the pedestrian who is the obstacle.

***Advantages of Fifth Embodiment***
According to the fifth embodiment, moving objects and pedestrians that are obstacles to the target moving object 120 are notified of a road obstacle of the target moving object 120. This makes it possible to realize safer automated driving.

***Additional Notes on the Fifth Embodiment***
When it is determined that there is a driving obstacle and that the obstacle is a moving object driven by a human or a pedestrian, the autonomous driving control device 400 sends a notification to the smartphone carried by the human. Also, when the obstacle is a moving object capable of being driven automatically, the autonomous driving control device 400 transmits information for instructing a change to the driving plan.
Obstacles may be differentiated into objects that are expected to be movable and objects that are not expected to be movable.
For example, potential obstacles include vehicles driven by humans, autonomous vehicles (moving objects capable of autonomous driving), non-moving objects (parked vehicles, falling rocks, etc.), and other objects (pedestrians, bicycles, vulnerable road users, etc.).

The fifth embodiment may be implemented in combination with at least one of the second to fourth embodiments. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving body 120 may include a drone port 121. The monitoring control device 300 may also detect the position of an obstacle using a color sensor 393. The target moving body 120 may also deal with a driving obstacle depending on the type of the driving obstacle.

Embodiment 6.
The embodiment in which the moving image stream AI processing pipeline system is applied will be described with reference to Fig. 18 to Fig. 21, mainly focusing on the differences from the first embodiment. AI is an abbreviation for artificial intelligence.

***Configuration Description***
The configuration of the autonomous driving system 100 will be described with reference to FIG. 18.
The autonomous driving system 100 includes a plurality of surrounding situation monitoring devices 130. In Fig. 18, the autonomous driving system 100 includes four surrounding situation monitoring devices 130, namely, one drone and three roadside devices. The number of surrounding situation monitoring devices 130 is not limited to four.
The monitoring sensor group 390 of each surroundings monitoring device 130 includes a video image sensor.

The autonomous driving control device 400 further includes elements such as a switcher unit 431, an object detection unit 432, a position estimation unit 433, an annotation unit 434, and a learning unit 435. These elements are realized by software. The autonomous driving control device 400 further includes a database 438. The database 438 has space and time axis information.
The autonomous driving control device 400 has a plurality of types of object detection models. The object detection models are learned models generated by machine learning or the like in order to detect objects shown in an image. In FIG. 18, the autonomous driving control device 400 has a general-purpose object detection model, a general-purpose road detection model, and a general-purpose human detection model. The road detection model is an object detection model for detecting roads. The human detection model is an object detection model for detecting people. The object detection models used are not limited to these models.

*** Operation Description ***
The operation of the multiple surrounding condition monitoring devices 130 and the automatic driving control device 400 will be described with reference to FIG. 18 .

The monitoring control device 300 of each surrounding condition monitoring device 130 transmits monitoring data, including a video stream obtained by a video image sensor, to the autonomous driving control device 400 (S133 and S134 in FIG. 6).

In step S142 (see FIG. 7), the autonomous driving control device 400 receives monitoring data transmitted from each surrounding situation monitoring device 130. The monitoring data includes a video stream.
In step S143 (see FIG. 7), the autonomous driving control device 400 uses an object detection model to detect obstacles to the travel of the target moving body 120, and determines whether there is a travel obstacle to the target moving body 120 using the obstacle detection result.

Obstacles are detected as follows:
The input destination of the video stream from each surrounding condition monitoring device 130 is determined in advance in association with the surrounding condition monitoring device 130 that is the transmission source.
The switcher unit 431 sends each video stream to an object detection model of an input destination associated with the surrounding condition monitoring device 130 of the transmission source. For example, the switcher unit 431 sends video stream A from roadside unit A to object detection and annotation, sends video stream B from roadside unit B to object detection, road detection, and annotation, and sends video stream C from roadside unit C to human detection and annotation.
The object detection model used at the input destination associated with the surrounding situation monitoring device 130 that is the transmission source is called a corresponding model.
The object detection unit 432 performs object detection for each video stream by using a corresponding model and inputting the video stream.
The position estimation unit 433 estimates the position (coordinate values) of the detected object based on the result of the object detection, and records the detection information of the detected object in the database 438. The detection information indicates the type, position, etc. of the detected object.
The user refers to the detection information of each object recorded in database 438 using autonomous driving control device 400, selects the object detection model to be updated, and specifies the selected object detection model to autonomous driving control device 400. The selected object detection model is referred to as a selected model.
The switcher unit 431 sends the video stream input to the selection model to the annotation unit 434 .
The annotation unit 434 uses a selection model to input a video stream and corrects the detection information of a detected object through annotations. The detection information is corrected as follows. The annotation unit 434 displays the video stream on a display. The user refers to the video stream, annotates the objects shown in the video stream, and inputs the annotation information to the autonomous driving control device 400. The annotation information indicates the position and label of the object. The label identifies the type of object. The annotation unit 434 receives the annotation information and corrects the detection information based on the annotation information.
The learning unit 435 updates the selection model by learning using the video stream and the corrected detection information as learning data. For example, the parameters of the selection model are updated. As a result, a new version of the object detection model is generated.
When a video stream is input to any type of object detection model, the object detection unit 432 performs object detection using the object detection model with the video stream input for each version of the same type of object detection model. Then, detection information for each version of the object detection model is recorded in the database 438.
The user refers to each piece of detection information for each object recorded in database 438 using autonomous driving control device 400. The user determines an object detection model to be used for determining a driving obstacle from a plurality of types of object detection models with different versions. Then, the user specifies the determined object detection model to autonomous driving control device 400. The determined object detection model is referred to as a determined model.
Thereafter, the object detection unit 432 performs object detection using the decision model with the video stream as input, and detects obstacles to the travel of the target moving object 120. The detection result includes prediction information of the obstacle's future behavior. The obstacle's future behavior is predicted based on the time series of the obstacle's position.
The determination model is changed as needed. Two or more determination models may be used. In this case, an obstacle is identified based on two or more detection results corresponding to two or more determination models. In this case, a method such as a weighted average is used.

The input destination of the above video stream is determined by the settings of the video stream AI processing pipeline 439.
The video stream AI processing pipeline 439 is set up for each surrounding situation monitoring device 130 and indicates the flow of the video stream.
19 to 21 show an example of a video stream AI processing pipeline 439. The "x" in "Vx" indicates a version number.
In Fig. 19, a video stream A from a roadside unit A is sent to a general-purpose object detection model (V0), sent to an annotation unit 434, annotation information is added, and sent to a learning unit 435 as a part of learning data. Then, an object detection model (V1) is generated by updating the general-purpose object detection model (V0). In addition, the general-purpose object detection model (V0) and detection information obtained by position estimation are recorded in a database 438. Thereafter, the video stream A is also sent to the object detection model (V1), sent to the annotation unit 434, annotation information is added, and the annotation information of the video stream A is recorded in a database 438.
20 , a video stream B from a roadside unit B is sent to an object detection model (V2) generated by updating a general-purpose object detection model (V0), and then sent to an annotation unit 434 where annotation information is added. Then, the movement of an obstacle is estimated based on the annotation information of the video stream B, and the obstacle detection information including the estimated movement information is recorded in a database 438.
In FIG. 21, a video stream D from a drone D is sent to a general-purpose road detection model (V0) and a road detection model (V1) generated by updating the general-purpose road detection model (V0).

***Effects of the Sixth Embodiment***
According to the sixth embodiment, it becomes possible to more accurately judge a driving obstacle using a video stream.

***Additional Information about the Sixth Embodiment***
The autonomous driving control device 400 processes information from the video image sensors of the multiple monitoring control devices 300. To this end, the autonomous driving control device 400 has a video switcher. The video switcher has a function of switching the flow of information from the video image sensors between image processing functions such as object detection, road detection, and lane detection, and annotation functions for learning.

The video switcher acquires moving images obtained from a plurality of monitor control devices 300 through a real-time communication stream, and distributes each stream to a required processing function.
The autonomous driving control device 400 has the function of setting up a video switcher and an AI processing pipeline.

The video switcher settings include the following functions. For example, the function to perform various detections and classifications using an AI model, the function to convert the detection results by the AI model into video, the function to perform annotation for learning, the function to save video images, and the function to adjust the image of the video images. The video switcher settings are defined by describing a pipeline in the processing order of these functions.

The video stream AI processing pipeline system using a video switcher can be used for a variety of purposes in addition to autonomous driving.
For example, the video stream AI processing pipeline system can be used for the following purposes:
(1) Detecting the number of vehicles entering and leaving a parking lot. (2) Detecting the number of parked vehicles in a parking lot. (3) Detecting people entering and leaving a facility. (4) Detecting the number of pedestrians on a road (going up or down). (5) Detecting the age and gender of pedestrians. (6) Estimating the speed of pedestrians. (7) Detecting stopping locations at a facility. (8) Detecting and classifying work content at a facility. (9) Detecting human behavior.

Embodiment 6 may be implemented in combination with at least one of embodiments 2 to 5.

***Additional Information on the Implementation ***
The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110.
The target moving object 120 is a moving object capable of autonomous driving. The target moving object 120 has sensors for detecting the surrounding environment, such as an optical sensor and a LIDAR, and a communication function.
The surrounding situation monitoring device 130 has sensors such as a camera that exists independently of the target moving object 120, and a communication function.
The autonomous driving control device 400 has a computer capable of communicating with the target moving object 120 and the surrounding situation monitoring device 130 .
The map database 110 stores map data 111 indicating surveillance areas and escape zones.
The target moving body 120 estimates its own position using a computer, detects surrounding objects that may be obstacles using sensors, and estimates the position information of the obstacles through calculation.
The surrounding situation monitoring device 130 detects the target moving body 120 and surrounding objects that may be obstacles using sensors, and estimates the position information of the obstacles through calculation.
The surroundings monitoring device 130 complements the missing information about blind spots of the sensors of the target moving body 120. The automatic driving control device 400 generates information required to change the driving plan. The target moving body 120 determines the intersection of the predicted path of the object and the planned path of the target moving body 120, and recalculates and updates the driving plan, such as the speed and path, so as not to hit the object. The updated driving plan is converted into control information and used for driving control of the target moving body 120.

The autonomous driving system 100 includes a target moving object 120, a drone 131, an autonomous driving control device 400, and a map database 110.
The target moving object 120 is a moving object capable of autonomous driving, and has sensors such as an optical sensor and LIDAR for detecting the surrounding environment, and a communication function.
The drone 131 is an aircraft having a flying function, and has sensors such as a camera that exist independently of the target moving object 120, and a communication function.
The autonomous driving control device 400 has a communication function and a calculation function. The autonomous driving control device 400 notifies the target moving object 120 of control data required for planning a driving plan.
The map database 110 has a storage function and a communication function. The map database 110 stores map data 111 used to determine the necessity of checking the surrounding situation. The map data 111 indicates a monitoring area and an escape zone. The map database 110 is capable of communicating with the target moving object 120, the surrounding situation monitoring device 130, and the automatic driving control device 400.
The drone 131 has a function of creating a flight plan for a round trip to a position where the blind spot can be viewed when the approach of the target moving body 120 to an area with a large blind spot is detected, and a function of controlling the drone 131 based on the flight plan.

Each embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. Each embodiment may be implemented partially or in combination with other embodiments. Procedures described using flow charts, etc. may be modified as appropriate.

Each element of the in-vehicle device 200, the monitoring control device 300, and the autonomous driving control device 400 may be realized by software, hardware, firmware, or a combination of these.
The "part" of each element of the in-vehicle device 200, the monitoring control device 300, and the autonomous driving control device 400 may be read as "processing,""step,""circuit," or "processing circuit."

Various aspects of the present disclosure are described below as appendices.
(Appendix 1)
An autonomous driving system including a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device,
The target moving body is
A group of mobile sensors;
an on-board device that determines whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the target mobile body, and map data indicating a monitoring area set as an area in which the surrounding situation of the target mobile body should be monitored by the surrounding situation monitoring device when the target mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, transmits a monitoring request to the surrounding situation monitoring device and transmits driving situation data obtained by the mobile body sensor group and indicating the driving situation of the target mobile body to the automatic driving control device;
Equipped with
The surrounding situation monitoring device is
A monitoring sensor group;
a monitoring control device that receives the monitoring request, causes the monitoring sensor group to monitor the surrounding situation of the target moving object, and transmits monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device;
Equipped with
The autonomous driving control device receives the driving status data and the monitoring data, determines a driving obstacle of the target moving body based on the driving status data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle,
The vehicle-mounted device receives the control data and controls the driving of the target moving object based on the control data and surrounding condition data obtained by the group of mobile object sensors and indicating the surrounding conditions of the target moving object.

(Appendix 2)
The surrounding situation monitoring device is an autonomous flying vehicle according to claim 1.

(Appendix 3)
The autonomous driving system of claim 2, wherein the target moving body is provided with a drone port from which the aircraft takes off and lands.

(Appendix 4)
the vehicle-mounted device transmits the monitoring request including the self-location data and monitoring area data indicating the monitoring area;
The monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitored area data from taking off from the drone port to flying over the monitored area and landing at the drone port, and causes the aircraft to fly along the flight path.An autonomous driving system as described in Appendix 3.

(Appendix 5)
The autonomous driving system according to claim 1, wherein the surrounding situation monitoring device is a roadside unit installed in the monitoring area.

(Appendix 6)
the monitoring sensor group includes a color sensor;
The monitoring and control device includes:
Detecting objects present around the target moving object and a plane on which the objects are located based on color sensor data obtained by the color sensor;
6. The autonomous driving system of claim 1, wherein the position of the object in a detected plane is estimated.

(Appendix 7)
The autonomous driving system according to claim 6, wherein the color sensor is an RGB camera or an RGB-D camera.

(Appendix 8)
The autonomous driving system of any one of appendix 1 to appendix 7, wherein the autonomous driving control device determines that a driving obstacle exists when an obstacle facing the target moving body is present within a proximity range from the position of the target moving body.

(Appendix 9)
When the obstacle is a moving object, the autonomous driving control device transmits the control data indicating that passing is impossible as the type of the driving obstacle to the target moving body,
The autonomous driving system according to claim 8, wherein the vehicle-mounted device causes the target moving object to retreat to an escape zone indicated in the map data when the type of the driving obstacle is such that passing is not possible.

(Appendix 10)
When the obstacle is a stationary object, the autonomous driving control device transmits the control data indicating impassability as the type of the driving obstacle to the target moving body,
The autonomous driving system according to claim 8 or 9, wherein the vehicle-mounted device makes the target moving body make a U-turn by utilizing an escape zone indicated in the map data when the type of the driving obstacle is the impassability.

(Appendix 11)
When a plurality of obstacles are present, the autonomous driving control device transmits to the target moving body the control data indicating a need to slow down as a type of the driving obstacle;
The autonomous driving system according to any one of claims 8 to 10, wherein the vehicle-mounted device causes the target moving body to move slowly when the type of the driving obstacle requires the target moving body to move slowly.

(Appendix 12)
An autonomous driving system as described in any one of Appendix 8 to Appendix 11, wherein the autonomous driving control device receives self-position data from each of one or more moving bodies, determines whether the obstacle is any of the one or more moving bodies based on the self-position data of each of the one or more moving bodies, and if it determines that the obstacle is any of the one or more moving bodies, notifies the moving body that is the obstacle of the driving disturbance.

(Appendix 13)
The autonomous driving system according to any one of Appendix 8 to Appendix 12, wherein the autonomous driving control device receives self-location data from each of one or more mobile devices carried by one or more pedestrians, determines whether the obstacle is any of the one or more pedestrians based on the self-location data of each of the one or more mobile devices, and if it determines that the obstacle is any of the one or more pedestrians, notifies the mobile device of the pedestrian who is the obstacle of the driving obstruction.

(Appendix 14)
The autonomous driving system includes a plurality of the surrounding situation monitoring devices,
The group of monitoring sensors of each of the surrounding situation monitoring devices includes a video image sensor,
The monitoring control device of each of the surrounding situation monitoring devices transmits the monitoring data including the video stream obtained by the video sensor to the autonomous driving control device,
The automatic driving control device includes:
For each of the video streams, an object detection model corresponding to the surrounding situation monitoring device that is a transmission source among a plurality of types of object detection models is used to input the video stream and perform object detection, and detection information of the detected object is recorded in a database;
Using a selection model, which is an object detection model selected from the plurality of types of object detection models, the video stream is input and detection information of the detected object is corrected by annotation, and the selection model is updated by learning the video stream and the corrected detection information to generate a new version of the object detection model;
14. The autonomous driving system of claim 1, further comprising: an object detection model determined from the plurality of versions of the object detection models; an input of the video stream; and an object detection method for detecting an obstacle to the travel of the target moving body.

(Appendix 15)
An autonomous driving method using an autonomous driving system including a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device,
The target moving body is
determining whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data indicating the position of the target moving body and map data indicating a monitoring area set as an area in which the surrounding situation of the target moving body should be monitored by the surrounding situation monitoring device when the target moving body passes, and when determining that monitoring by the surrounding situation monitoring device is necessary, transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating the driving situation of the target moving body to the automatic driving control device;
The surrounding situation monitoring device is
receiving the monitoring request, monitoring a surrounding situation of the target moving object, and transmitting monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device;
The automatic driving control device includes:
receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving object based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving object of the driving obstacle;
The target moving body is
An autonomous driving method for receiving the control data and driving the target moving body autonomously based on the control data and surrounding condition data indicating the surrounding conditions of the target moving body.

(Appendix 16)
It is a vehicle that runs autonomously,
A group of mobile sensors;
an on-board device that determines whether or not monitoring by the surrounding conditions monitoring device is necessary by referring to self-position data obtained by the mobile body sensor group and indicating the position of the mobile body, and map data indicating a monitoring area set as an area in which the surrounding conditions of the mobile body should be monitored by a surrounding conditions monitoring device when the mobile body passes, and when it determines that monitoring by the surrounding conditions monitoring device is necessary, transmits a monitoring request to the surrounding conditions monitoring device and transmits driving condition data obtained by the mobile body sensor group and indicating the driving condition of the mobile body to an automatic driving control device, receives control data from the automatic driving control device that notifies of a driving obstacle of the mobile body determined based on the driving condition data and the monitoring data obtained by the surrounding conditions monitoring device and indicating the surrounding conditions of the mobile body, and controls the driving of the mobile body based on the control data and the surrounding conditions data obtained by the mobile body sensor group and indicating the surrounding conditions of the mobile body;
A moving body comprising:

(Appendix 17)
The surrounding situation monitoring device is an autonomous flying object,
The moving body is the moving body described in Appendix 16, which is equipped with a drone port on which the air vehicle takes off and lands.

(Appendix 18)
A monitoring sensor group;
a monitoring control device that receives a monitoring request from a target moving body traveling autonomously when the target moving body passes through a monitoring area, causes the monitoring sensor group to monitor the surrounding conditions of the target moving body to obtain monitoring data indicating the surrounding conditions of the target moving body, and transmits the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving condition data indicating the driving condition of the target moving body;
An ambient situation monitoring device comprising:

(Appendix 19)
The target moving object includes a drone port;
The surrounding situation monitoring device is an autonomous flying vehicle that takes off and lands using the drone port.

(Appendix 20)
The monitoring control device receives the monitoring request including self-location data indicating the position of the target moving body and monitoring area data indicating the monitoring area, plans a flight path from taking off from the drone port based on the self-location data and the monitoring area data, flying over the monitoring area and landing at the drone port, and flies the flying body along the flight path.The surrounding condition monitoring device described in Appendix 19.

(Appendix 21)
19. The surrounding conditions monitoring device according to claim 18, which is a roadside unit provided in the monitoring area.

(Appendix 22)
the monitoring sensor group includes a color sensor;
The monitoring and control device includes:
Detecting objects present around the target moving object and a plane on which the objects are located based on color sensor data obtained by the color sensor;
22. The surrounding situation monitoring device according to claim 18, wherein the position of the object in a detected plane is estimated.

(Appendix 23)
23. The surrounding situation monitoring device according to claim 22, wherein the color sensor is an RGB camera or an RGB-D camera.

(Appendix 24)
Receiving driving status data indicating a driving status of a target moving object from the target moving object that is driving autonomously;
receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device which receives a monitoring request from the target moving object when the target moving object passes through a monitoring area and monitors a surrounding situation of the target moving object;
determining a traveling obstacle of the target moving object based on the traveling condition data and the monitoring data;
An automatic driving control device that transmits control data notifying the target moving body of the driving obstacle.

(Appendix 25)
The autonomous driving control device of claim 24, which determines that a driving obstacle exists when an obstacle facing the target moving body is present within a proximity range from the position of the target moving body.

(Appendix 26)
The autonomous driving control device of claim 25, wherein when the obstacle is a moving object, the control data indicating that passing is not possible as the type of driving obstacle is transmitted to the target moving body, thereby causing the target moving body to retreat to an escape zone.

(Appendix 27)
An autonomous driving control device as described in Appendix 25 or Appendix 26, which, when the obstacle is a stationary object, causes the target moving body to make a U-turn using an escape zone by transmitting the control data to the target moving body indicating that passage is prohibited as the type of driving obstacle.

(Appendix 28)
An automatic driving control device as described in any one of Appendix 25 to Appendix 27, which, when multiple obstacles are present, causes the target moving body to slow down by transmitting the control data to the target moving body indicating the need to slow down as the type of driving obstacle.

(Appendix 29)
An autonomous driving control device as described in any one of Appendix 25 to Appendix 28, which receives self-location data from each of one or more moving bodies, determines whether the obstacle is any of the one or more moving bodies based on the self-location data of each of the one or more moving bodies, and if it determines that the obstacle is any of the one or more moving bodies, notifies the moving body that is the obstacle of the driving disturbance.

(Appendix 30)
An autonomous driving control device as described in any one of Appendix 25 to Appendix 29, which receives self-location data from each of one or more mobile devices carried by one or more pedestrians, determines whether the obstacle is any of the one or more pedestrians based on the self-location data of each of the one or more mobile devices, and if it determines that the obstacle is any of the one or more pedestrians, notifies the mobile device of the pedestrian who is the obstacle of the driving obstacle.

(Appendix 31)
A process of determining whether or not monitoring by the surrounding situation monitoring device is required by referring to self-location data indicating the location of a moving body traveling by automatic driving and map data indicating a monitoring area set as an area in which the surrounding situation of the moving body should be monitored by the surrounding situation monitoring device when the moving body passes;
When it is determined that monitoring by the surrounding situation monitoring device is necessary, a process of transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating a driving situation of the moving body to an automatic driving control device;
A process of receiving control data notifying a driving obstacle of the moving body from the automatic driving control device, the control data being determined based on the driving situation data and monitoring data indicating the surrounding situation of the moving body obtained by the surrounding situation monitoring device;
A process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object;
An autonomous driving program that causes a computer to execute the above.

(Appendix 32)
A process of receiving a monitoring request from a target moving object when the target moving object travels through a monitoring area while the target moving object is autonomously driven;
A process of causing a group of monitoring sensors to monitor the surroundings of the target moving object and obtaining monitoring data indicating the surroundings of the target moving object;
A process of transmitting the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving situation data indicating the driving situation of the target moving body;
A monitoring program that causes a computer to execute the following.

(Appendix 33)
A process of receiving driving status data indicating a driving status of a target moving object from the target moving object that is driving autonomously;
receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device that receives a monitoring request from the target moving object when the target moving object passes through a monitoring area and monitors the surrounding situation of the target moving object;
A process of determining a traveling obstacle of the target moving object based on the traveling condition data and the monitoring data;
A process of transmitting control data notifying the target moving object of the obstacle to the target moving object;
A control program that causes a computer to execute the above.

(Appendix 34)
An autonomous driving system including a target moving object that is a moving object that runs on the ground autonomously, an aircraft that flies autonomously in the sky, and an autonomous driving control device,
The target moving body is
A group of mobile sensors;
an in-vehicle device that transmits driving status data obtained by the moving body sensor group and indicating a driving status of the target moving body to the automatic driving control device;
Equipped with
The flying object is
A monitoring sensor group;
a monitoring control device that causes the monitoring sensor group to monitor the surrounding situation of the target moving object from the sky and transmits monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device;
Equipped with
The autonomous driving control device receives the driving status data and the monitoring data, determines a driving obstacle of the target moving object based on the driving status data and the monitoring data, and transmits control data notifying the target moving object of the driving obstacle,
The vehicle-mounted device receives the control data and controls the driving of the target moving object based on the control data and surrounding condition data obtained by the group of mobile object sensors and indicating the surrounding conditions of the target moving object.

(Appendix 35)
The autonomous driving system of claim 34, wherein the target moving body is provided with a drone port from which the aircraft takes off and lands.

(Appendix 36)
the vehicle-mounted device transmits a monitoring request including monitoring area data indicating a monitoring area and self-location data obtained by the mobile sensor group and indicating a location of the target mobile object;
The monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitored area data from taking off from the drone port to flying over the monitored area and landing at the drone port, and flies the aircraft along the flight path.An autonomous driving system as described in Appendix 35.

(Appendix 37)
An autonomous driving method using an autonomous driving system including a target moving body that is a moving body that runs on the ground autonomously, an aircraft that flies autonomously in the sky, and an autonomous driving control device,
The target moving object is
Transmitting driving status data indicating a driving status of the target moving object to the automatic driving control device;
The flying object is
Monitoring the surroundings of the target moving object from the sky, and transmitting monitoring data indicating the surroundings of the target moving object to the automatic driving control device;
The automatic driving control device includes:
receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving object based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving object of the driving obstacle;
The target moving object is
An autonomous driving method for receiving the control data and driving the target moving body autonomously based on the control data and surrounding condition data indicating the surrounding conditions of the target moving body.

(Appendix 38)
It is a vehicle that travels on the ground autonomously.
A group of mobile sensors;
an on-board device that transmits driving status data indicating the driving status of the moving body obtained by the group of moving body sensors to an autonomous driving control device, receives control data from the autonomous driving control device notifying the moving body of a driving obstacle determined based on the driving status data and monitoring data indicating the surrounding status of the moving body obtained by an aircraft flying overhead by autonomous driving, and controls the driving of the moving body based on the control data and surrounding status data indicating the surrounding status of the moving body obtained by the group of moving body sensors;
A moving body comprising:

(Appendix 39)
A mobile body as described in Appendix 38, comprising a drone port for the air vehicle to take off and land.

(Appendix 40)
It is an autonomous flying vehicle that flies through the sky.
A monitoring sensor group;
a monitoring control device that causes the monitoring sensor group to monitor from the sky the surrounding conditions of a target moving object traveling on the ground in an autonomous driving manner, to obtain monitoring data indicating the surrounding conditions of the target moving object, and transmits the monitoring data to an autonomous driving control device that notifies the target moving object of a traveling obstacle of the target moving object determined based on the monitoring data and traveling condition data indicating the traveling condition of the target moving object;
An aircraft equipped with the above.

(Appendix 41)
The target moving object includes a drone port;
The air vehicle of claim 40, wherein the air vehicle takes off and lands using the drone port.

(Appendix 42)
The monitoring control device receives a monitoring request including monitoring area data indicating a monitoring area and self-location data indicating the position of the target moving body, plans a flight path from taking off from the drone port based on the self-location data and the monitoring area data, flying over the monitoring area and landing at the drone port, and flies the aircraft along the flight path.

(Appendix 43)
Receiving driving status data indicating a driving status of a target moving object from the target moving object traveling on the ground by autonomous driving;
receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device that flies above the target moving object in an autonomous driving manner and monitors a surrounding situation of the target moving object;
determining a traveling obstacle of the target moving object based on the traveling condition data and the monitoring data;
An automatic driving control device that transmits control data notifying the target moving body of the driving obstacle.

(Appendix 44)
A process of transmitting driving status data indicating a driving status of a moving object traveling on the ground by autonomous driving to an autonomous driving control device;
A process of receiving control data notifying a driving obstacle of the moving body from the automatic driving control device, the control data being determined based on the driving status data and monitoring data indicating the surrounding conditions of the moving body obtained by an aircraft flying in the sky by automatic driving;
A process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object;
An autonomous driving program that causes a computer to execute the above.

(Appendix 45)
A process of having a group of monitoring sensors mounted on an aircraft flying autonomously in the sky monitor the surroundings of a target moving object traveling autonomously on the ground, thereby obtaining monitoring data indicating the surroundings of the target moving object;
A process of transmitting the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving situation data indicating the driving situation of the target moving body;
A monitoring program that causes a computer to execute the following.

(Appendix 46)
A process of receiving driving status data indicating a driving status of a target moving object from the target moving object traveling on the ground by autonomous driving;
A process of receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device that flies above the target moving object in an autonomous driving manner and monitors the surrounding situation of the target moving object;
A process of determining a traveling obstacle of the target moving object based on the traveling condition data and the monitoring data;
A process of transmitting control data notifying the target moving object of the obstacle to the target moving object;
A control program that causes a computer to execute the above.

100 Autonomous driving system, 101 Mobile body, 102 Mobile terminal, 110 Map database, 111 Map data, 120 Target mobile body, 121 Drone port, 130 Surrounding situation monitoring device, 131 Drone, 200 Vehicle-mounted device, 201 Processor, 202 Memory, 203 Auxiliary storage device, 204 Communication device, 205 Input/output interface, 211 Monitoring request unit, 212 Control data acquisition unit, 213 Driving control unit, 220 Memory unit, 290 Mobile body sensor group, 291 Depth camera, 292 Lidar, 293 Positioning sensor, 300 Monitoring control device, 301 Processor, 302 Memory, 303 Auxiliary storage device, 304 Communication device, 305 Input/output interface, 311 Request reception unit, 312 Monitoring control unit, 313 Monitoring data generation unit, 314 Monitoring data providing unit, 320 memory unit, 390 monitoring sensor group, 391 depth camera, 392 RGB camera, 393 color sensor, 400 autonomous driving control device, 401 processor, 402 memory, 403 auxiliary storage device, 404 communication device, 405 input/output interface, 411 data reception unit, 412 driving obstacle determination unit, 413 control data generation unit, 414 control data providing unit, 415 driving obstacle notification unit, 420 memory unit, 431 switcher unit, 432 object detection unit, 433 position estimation unit, 434 annotation unit, 435 learning unit, 438 database, 439 video stream AI processing pipeline.

Claims (18)

An autonomous driving system including a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device,
The target moving object is
A group of mobile sensors;
an on-board device that determines whether the target moving object has entered the monitoring area by referring to self-location data obtained by the group of moving object sensors and indicating the position of the target moving object, and map data indicating a monitoring area set as an area in which the surrounding conditions of the target moving object should be monitored by the surrounding conditions monitoring device when the target moving object passes through, and that transmits a monitoring request to the surrounding conditions monitoring device when it is determined that the target moving object has entered the monitoring area, and transmits driving condition data obtained by the group of moving object sensors and indicating the driving conditions of the target moving object to the automatic driving control device;
Equipped with
The surrounding situation monitoring device is
A monitoring sensor group;
a monitoring control device that receives the monitoring request, causes the monitoring sensor group to monitor the surrounding situation of the target moving object, and transmits monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device;
Equipped with
The autonomous driving control device receives the driving status data and the monitoring data, determines a driving obstacle of the target moving body based on the driving status data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle,
The vehicle-mounted device receives the control data and controls the driving of the target moving object based on the control data and surrounding condition data obtained by the group of mobile object sensors and indicating the surrounding conditions of the target moving object. The autonomous driving system according to claim 1 , wherein the surrounding situation monitoring device is an autonomous flying vehicle. The autonomous driving system of claim 2 , wherein the target moving body is provided with a drone port from which the aircraft takes off and lands. the vehicle-mounted device transmits the monitoring request including the self-location data and monitoring area data indicating the monitoring area;
The autonomous driving system of claim 3, wherein the monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitored area data from taking off from the drone port to flying over the monitored area and landing at the drone port, and causes the aircraft to fly along the flight path. The autonomous driving system according to claim 1 , wherein the surrounding situation monitoring device is a roadside unit installed in the monitoring area. the monitoring sensor group includes a color sensor;
The monitoring and control device includes:
Detecting objects present around the target moving object and a plane on which the objects are located based on color sensor data obtained by the color sensor;
The autonomous driving system according to any one of claims 1 to 5, wherein the position of the object on the detected plane is estimated, and data indicating the position of each object existing around the target moving body is generated as the monitoring data. The autonomous driving system according to claim 6, wherein the color sensor is an RGB camera or an RGB-D camera. The autonomous driving system according to claim 1 , wherein the autonomous driving control device determines that a driving obstacle exists when an obstacle facing the target moving body is present within a proximity range from the position of the target moving body. When the obstacle is a moving object, the autonomous driving control device transmits the control data indicating that passing is impossible as the type of the driving obstacle to the target moving body,
The autonomous driving system according to claim 8 , wherein the vehicle-mounted device causes the target moving object to retreat to an escape zone indicated in the map data when the type of the driving obstacle is the non-passing obstacle. When the obstacle is a stationary object, the autonomous driving control device transmits the control data indicating impassability as the type of the driving obstacle to the target moving body,
The autonomous driving system according to claim 8 or 9, wherein the vehicle-mounted device makes the target moving object make a U-turn by utilizing an escape zone indicated in the map data when the type of the driving obstacle is the impassability. When a plurality of obstacles are present, the autonomous driving control device transmits to the target moving body the control data indicating a need to slow down as a type of the driving obstacle;
The autonomous driving system according to claim 8 or claim 9, wherein the vehicle-mounted device makes the target moving object move slowly when the type of the driving obstacle requires the target moving object to move slowly. The autonomous driving control device receives self-position data from each of one or more moving bodies,
An autonomous driving system as described in claim 8 or claim 9, which determines whether the obstacle is any of the one or more moving bodies based on the self-position data of each of the one or more moving bodies, and if it determines that the obstacle is any of the one or more moving bodies, notifies the moving body that is the obstacle of the driving obstruction. The autonomous driving system according to claim 8 or claim 9, wherein the autonomous driving control device receives self-location data from each of one or more mobile devices carried by one or more pedestrians, determines whether the obstacle is any of the one or more pedestrians based on the self-location data of each of the one or more mobile devices, and if it determines that the obstacle is any of the one or more pedestrians, notifies the mobile device of the pedestrian who is the obstacle of the driving obstruction. The autonomous driving system includes a plurality of the surrounding situation monitoring devices,
The group of monitoring sensors of each of the surrounding situation monitoring devices includes a video image sensor,
The monitoring control device of each of the surrounding situation monitoring devices transmits the monitoring data including the video stream obtained by the video sensor to the autonomous driving control device,
The automatic driving control device includes:
For each of the video streams, an object detection model corresponding to the surrounding situation monitoring device that is a transmission source among a plurality of types of object detection models is used to input the video stream and perform object detection, and detection information of the detected object is recorded in a database;
displaying the video stream on a display, receiving annotation information input to the autonomous driving control device, correcting detection information of an object detected by inputting the video stream using a selection model that is an object detection model selected from the plurality of types of object detection models based on the annotation information , learning the video stream and the corrected detection information to update the selection model and generate a new version of the object detection model;
The autonomous driving system according to claim 1, wherein the video stream is input and the object detection is performed using a decision model, which is an object detection model designated for the autonomous driving control device among the multiple types of version-based object detection models, to detect obstacles to the travel of the target moving body. An autonomous driving method using an autonomous driving system including a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device,
The target moving body is
a monitoring device that monitors the surrounding conditions of the target moving object when the target moving object passes through the target moving object and the surrounding conditions monitoring device; and a monitoring request is transmitted to the surrounding conditions monitoring device when the target moving object is determined to have entered the monitoring area, and driving condition data indicating the driving condition of the target moving object is transmitted to the automatic driving control device;
The surrounding situation monitoring device is
receiving the monitoring request, monitoring a surrounding situation of the target moving object, and transmitting monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device;
The automatic driving control device includes:
receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving object based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving object of the driving obstacle;
The target moving body is
An autonomous driving method for receiving the control data and driving the target moving body autonomously based on the control data and surrounding condition data indicating the surrounding conditions of the target moving body. It is a vehicle that runs autonomously,
A group of mobile sensors;
an on-board device that determines whether the moving body has entered the monitoring area by referring to self-location data obtained by the moving body sensor group and indicating the position of the moving body, and map data indicating a monitoring area set as an area in which the surrounding conditions of the moving body should be monitored by a surrounding conditions monitoring device when the moving body passes through, and when it is determined that the moving body has entered the monitoring area, transmits a monitoring request to the surrounding conditions monitoring device, and transmits driving condition data obtained by the moving body sensor group and indicating the driving conditions of the moving body to an automatic driving control device, receives control data from the automatic driving control device that notifies of a driving obstacle of the moving body determined based on the driving condition data and the monitoring data obtained by the surrounding conditions monitoring device and indicating the surrounding conditions of the moving body, and controls the driving of the moving body based on the control data and the surrounding conditions data obtained by the moving body sensor group and indicating the surrounding conditions of the moving body;
A moving body comprising: The surrounding situation monitoring device is an autonomous flying object,
The mobile body according to claim 16 , wherein the mobile body is provided with a drone port on which the air vehicle takes off and lands. A process of determining whether the moving body has entered a monitoring area by referring to self-location data indicating the location of the moving body traveling by autonomous driving and map data indicating a monitoring area set as an area in which the surrounding situation of the moving body should be monitored by a surrounding situation monitoring device when the moving body passes through the monitoring area;
a process of transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating a driving situation of the moving object to an automatic driving control device when it is determined that the moving object has entered the monitoring area;
A process of receiving control data notifying a driving obstacle of the moving body from the automatic driving control device, the control data being determined based on the driving situation data and monitoring data indicating the surrounding situation of the moving body obtained by the surrounding situation monitoring device;
A process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object;
An autonomous driving program that causes a computer to execute the above.

Images