WO2024204271A1 - Control device, and control method - Google Patents

Abstract

This control device identifies the positions of a first object and a second object in a real space on the basis of: a first reflected wave and a second reflected wave, the first and second reflected waves being obtained by reflection of electromagnetic waves, emitted in a first direction of the real space, by the first object and the second object, respectively; and a captured image including the first object and the second object, obtained by imaging the real space.

Description

Control device and control method CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to patent application No. 2023-056518, filed in Japan on March 30, 2023, the entire disclosure of which is incorporated herein by reference.

This disclosure relates to a control device and a control method.

Technology for detecting objects such as fallen objects on the road is known (for example, see Patent Document 1). The road monitoring device described in Patent Document 1 includes an imaging unit that captures images of at least the front or rear of the vehicle and sequentially outputs the image data, and a discrimination unit that determines whether or not a specified monitoring object is included in the image corresponding to the image data based on the image data.

JP 2016-151887 A

A control device according to an embodiment of the present disclosure includes:
The positions of the first object and the second object in the real space are identified based on a first reflected wave reflected by a first object and a second reflected wave reflected by a second object of an electromagnetic wave radiated in a first direction in the real space, and an image of the real space including the first object and the second object.

A control device according to an embodiment of the present disclosure includes:
The positions of the first object and the second object in the real space are identified based on a first reflected wave formed by reflection by a first object and a second reflected wave formed by reflection by a second object of an electromagnetic wave irradiated to a first irradiation range in the real space, a third reflected wave formed by reflection by the first object of an electromagnetic wave irradiated to a second irradiation range in the real space, and an image generated by imaging the real space including the first irradiation range and the second irradiation range.

A control method according to an embodiment of the present disclosure includes:
The method includes identifying the positions of the first object and the second object in the real space based on a first reflected wave reflected by a first object and a second reflected wave reflected by a second object of an electromagnetic wave radiated in a first direction in the real space, and an image of the real space including the first object and the second object.

A control method according to an embodiment of the present disclosure includes:
The method includes identifying positions of the first object and the second object in the real space based on a first reflected wave formed by reflection by a first object and a second reflected wave formed by reflection by a second object of an electromagnetic wave irradiated to a first irradiation range in the real space, a third reflected wave formed by reflection by the first object of an electromagnetic wave irradiated to a second irradiation range in the real space, and an image generated by imaging the real space including the first irradiation range and the second irradiation range.

FIG. 1 is a diagram showing a schematic configuration of a detection system according to an embodiment of the present disclosure. FIG. 2 is a block diagram of the detection system shown in FIG. 1 . FIG. 4 is a diagram showing an example of a captured image. FIG. 11 is a diagram showing an example of data on the intensity of a reflected wave. FIG. 11 is a diagram showing an example of data on the intensity of a reflected wave. FIG. 11 is a diagram showing an example of data on the intensity of a reflected wave. 4 is a diagram for explaining a detection direction corresponding to the irradiation range shown in FIG. 3 . FIG. FIG. 4 is a diagram for explaining the direction of light in the configuration shown in FIG. 3 . 10 is a flowchart illustrating an example procedure of an object detection process according to an embodiment of the present disclosure.

It is desirable to detect objects with high accuracy. According to one embodiment of the present disclosure, objects can be detected with high accuracy.

Embodiments of the present disclosure are described below with reference to the drawings.

(Configuration of the detection system)
1, a detection system 1 according to the present embodiment is mounted on a moving object 2. The moving object 2 is, for example, a vehicle, a motorcycle, a robot, or the like. However, the detection system 1 does not have to be mounted on the moving object 2. When the detection system 1 is not mounted on the moving object 2, it may be incorporated in an electronic device such as a smartphone.

As shown in Figures 1 and 2, the detection system 1 includes a distance measuring device 10, an imaging device 20, and a control device 30. The distance measuring device 10 and the imaging device 20 can communicate with the control device 30 via a communication line. The control device 30 does not have to be mounted on the moving body 2. If the control device 30 is not mounted on the moving body 2, it may be configured to be able to communicate with the distance measuring device 10 and the imaging device 20 mounted on the moving body 2 via a network or the like.

The distance measuring device 10 is configured to include, for example, LiDAR (Light Detection And Ranging). LiDAR is configured to include, for example, a laser light source that emits electromagnetic waves, an optical system, and a detection unit that detects the electromagnetic waves. The electromagnetic waves are, for example, infrared rays, visible light, ultraviolet rays, or radio waves.

The ranging device 10 emits electromagnetic waves into the real space around the ranging device 10. The electromagnetic waves emitted by the ranging device 10 are reflected by an object and return to the ranging device 10 as reflected waves. The ranging device 10 detects the electromagnetic waves that are reflected by an object and return, i.e., the reflected waves, from the emitted electromagnetic waves. The ranging device 10 measures the distance to the object by detecting the reflected waves. The ranging device 10 may measure the distance to the object by any method, such as the ToF (Time of Flight) method. In the case of ToF (Time of Flight), the ranging device 10 measures the distance to the object by directly measuring the time from when the electromagnetic waves are emitted to when the reflected waves of the emitted electromagnetic waves are detected.

The ranging device 10 measures the distance to an object and obtains data on the intensity of the reflected wave reflected back by the object. For example, if the ranging device 10 includes a LiDAR and the detection unit of the LiDAR further includes an element such as an APD (Avalanche Photo Diode), the intensity of the reflected wave may be a value based on the intensity of the optical signal input to the detection unit. In this embodiment, the ranging device 10 measures the distance to an object by scanning real space with electromagnetic waves and detecting the reflected waves that are reflected by objects, and obtains data on the intensity of the reflected wave reflected back by the object.

The range in real space that the ranging device 10 scans with electromagnetic waves is also referred to as the "scanning range." The scanning range includes multiple irradiation ranges. The irradiation range is the range that is irradiated with electromagnetic waves when the ranging device 10 irradiates electromagnetic waves in one direction in real space. For example, the irradiation range corresponds to the angle θ shown in Figures 7 and 8 described below. For example, the ranging device 10 detects the reflected wave of the electromagnetic wave irradiated to one irradiation range as a reflected wave arriving from one direction. In other words, one irradiation range corresponds to one detection direction in which the ranging device 10 detects the electromagnetic wave. In addition, the ranging device 10 scans the scanning range with electromagnetic waves and acquires data on the distance from the ranging device 10 to the object on which the reflected wave is reflected for each irradiation range. The acquired distance data is also referred to as the "distance measurement image." The ranging image includes multiple pixels. One pixel of the ranging image corresponds to one irradiation range of the scanning range. Additionally, each pixel in the distance measurement image is associated with distance data acquired in the corresponding illumination range and data on the intensity of the reflected waves.

The distance measuring device 10 acquires data on the distance to the object and data on the intensity of the reflected wave for each irradiation range, as shown in Figures 4 to 6 described below. The distance measuring device 10 transmits the acquired data on the distance to the object and data on the intensity of the reflected wave for each irradiation range to the control device 30.

The imaging device 20 is configured to include an imaging optical system and an imaging element. The imaging device 20 generates an image by capturing an image of the real space around the imaging device 20. The captured image includes a plurality of pixels. The range in which the imaging device 20 captures the real space is also referred to as the "imaging range." At least a portion of the imaging range of the imaging device 20 overlaps with at least a portion of the scanning range of the distance measuring device 10. The imaging device 20 may generate an image at any frame rate. The imaging device 20 transmits data of the generated captured image to the control device 30.

The captured image of the imaging device 20 and each irradiation range of the scanning range of the distance measuring device 10 are associated in advance. For example, as described above, one pixel of the distance measuring image corresponds to one irradiation range of the scanning range. Therefore, the coordinate system set for the captured image and the coordinate system set for the distance measuring image may be associated in advance. By associating these two coordinate systems in advance, it is possible to associate the captured image of the imaging device 20 with each irradiation range of the scanning range of the distance measuring device 10 in advance. Alternatively, each pixel of the captured image of the imaging device 20 and each pixel of the distance measuring image of the distance measuring device 10 may be associated in advance. By associating each pixel of the captured image with each pixel of the distance measuring image in advance, it is possible to associate the captured image of the imaging device 20 with each irradiation range of the distance measuring device 10 in advance.

The optical axis of the imaging device 20 and the optical axis of the distance measuring device 10 may coincide. For example, the optical axis of the imaging optical system of the imaging device 20 and the optical axis of the LiDAR optical system of the distance measuring device 10 may coincide. In this case, a part of the imaging optical system of the imaging device 20 and a part of the LiDAR optical system of the distance measuring device 10 may be common. When a part of the imaging optical system and a part of the LiDAR optical system are common, the imaging device 20 and the distance measuring device 10 may be configured as a single device. In this case, the imaging device 20 and the distance measuring device 10 may be configured as described in JP 2018-132384 A. However, as long as the captured image of the imaging device 20 and each irradiation range of the distance measuring device 10 are previously associated with each other, the optical axis of the imaging device 20 and the optical axis of the distance measuring device 10 do not have to coincide.

The control device 30 is, for example, a computer. The control device 30 may be any computer, such as a general-purpose computer, a workstation, or a dedicated computer.

The control device 30 includes a communication unit 31, an output unit 32, a memory unit 33, and a control unit 34.

The communication unit 31 includes at least one communication module capable of communicating with the distance measuring device 10 and the imaging device 20 via a communication line. The communication module is a communication module that complies with the standard of the communication line. The communication line includes at least one of a wired line and a wireless line.

The communication unit 31 may be configured to include at least one communication module capable of communicating with other systems of the mobile body 2. The communication module is a communication module compatible with a communication standard between the control device 30 and other systems. The communication standard is, for example, a standard such as CAN (Controller Area Network).

The output unit 32 is capable of outputting data. The output unit 32 includes at least one output interface capable of outputting data. The output interface is, for example, a display. If the mobile object 2 is a vehicle, the display may be disposed on the dashboard of the vehicle. The output unit 32 may include a device capable of outputting information to an external storage medium.

The memory unit 33 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The RAM is, for example, an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The ROM is, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory). The memory unit 33 may function as a main memory device, an auxiliary memory device, or a cache memory. The memory unit 33 stores data used in the operation of the control device 30 and data obtained by the operation of the control device 30.

The control unit 34 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), etc. The control unit 34 controls each part of the control unit 30 and executes processes related to the operation of the control unit 30.

The control unit 34 receives data on the distance to the object and data on the intensity of the reflected wave for each irradiation range of the distance measuring device 10 from the distance measuring device 10 via the communication unit 31. The control unit 34 also receives captured image data from the imaging device 20 via the communication unit 31.

For example, the control unit 34 receives data of a captured image 40 as shown in FIG. 3. The captured image 40 includes objects on the tire 41 and the road surface 42. In this disclosure, an object included in the captured image means an object depicted in the captured image. In the configuration shown in FIG. 3, the imaging range of the imaging device 20 is the same as the scanning range of the distance measuring device 10. In the captured image 40, the position of each irradiation range of the scanning range of the distance measuring device 10 is indicated by a dashed line. As described above, one irradiation range corresponds to one pixel of the distance measuring image. Therefore, as can be seen from the position of the dashed line, the size of the pixels of the captured image 40 is smaller than the size of the pixels of the distance measuring image. In other words, the resolution of the captured image 40 is higher than the resolution of the distance measuring image.

For example, the control unit 34 receives data on the distance to an object and data on the intensity of the reflected wave as shown in Figures 4 to 6. In Figures 4 to 6, the horizontal axis indicates the distance to the object, and the vertical axis indicates the intensity of the reflected wave.

The data shown in Figure 4 is data on the distance to an object in the irradiation range 43 shown in Figure 3 and data on the intensity of the reflected wave. As shown in Figure 3, the portion of the captured image 40 corresponding to the irradiation range 43 includes the tire 41 and the road surface 42. In other words, the data on the intensity of the reflected wave in the irradiation range 43 includes data based on the reflected wave from the tire 41 (first reflected wave) and the reflected wave from the road surface 42 (second reflected wave). Therefore, the data on the intensity of the reflected wave shown in Figure 4 includes an intensity peak P1 caused by the reflected wave of the electromagnetic wave reflected by the tire 41, and an intensity peak P2 caused by the reflected wave of the electromagnetic wave reflected by the road surface 42. Here, the intensity peaks P1 and P2 can be said to be multiple intensity peaks with different detection times detected from reflected waves received during a specified period. The specified period is, for example, the period from when irradiation of the electromagnetic wave starts on the irradiation range 43 to when irradiation of the electromagnetic wave ends in that frame during the period (one frame) during which the electromagnetic wave is irradiated over the entire scanning range, or a period shorter than the period from when irradiation of the electromagnetic wave starts on the irradiation range 43 to when irradiation of the electromagnetic wave ends in that frame.

The data shown in FIG. 5 is data on the distance to an object in the irradiation range 44 shown in FIG. 3 and data on the intensity of the reflected wave. As shown in FIG. 3, the portion of the captured image 40 that corresponds to the irradiation range 44 includes the tire 41 but does not include the road surface 42. In other words, the data on the intensity of the reflected wave in the irradiation range 44 includes data based on the reflected wave from the tire 41. Therefore, the data on the intensity of the reflected wave shown in FIG. 5 includes an intensity peak P3 that occurs due to the reflected wave when the electromagnetic wave is reflected by the tire 41.

The data shown in FIG. 6 is data on the distance to an object in the irradiation range 45 shown in FIG. 3 and data on the intensity of the reflected wave. As shown in FIG. 3, the portion of the captured image 40 that corresponds to the irradiation range 45 includes the road surface 42 but does not include the tires 41. In other words, the data on the intensity of the reflected wave in the irradiation range 45 includes data based on the wave reflected from the road surface 42. Therefore, the data on the intensity of the reflected wave shown in FIG. 6 includes an intensity peak P4 that occurs due to the electromagnetic wave being reflected by the road surface 42.

Here, as described above, the distance measuring device 10 detects the reflected wave of the electromagnetic wave irradiated to one irradiation range as a reflected wave arriving from one direction. In other words, one irradiation range corresponds to one detection direction in which the distance measuring device 10 detects the reflected wave of the electromagnetic wave. For example, FIG. 7 shows detection direction d1 corresponding to the irradiation range 43 shown in FIG. 3. The configuration shown in FIG. 7 corresponds to a top view of the configuration shown in FIG. 1. The angle θ corresponds to one irradiation range. In the configuration shown in FIG. 7, the detection direction d1 corresponding to the irradiation range 43 is the direction from the distance measuring device 10 to the center of the irradiation range 43.

The detection direction in which the ranging device 10 detects the reflected wave is regarded as the direction of arrival of the reflected wave reflected by the object. In other words, this detection direction is treated as the direction from the ranging device 10 to the object included in the irradiation range corresponding to the detection direction. Therefore, when data of multiple objects is included in one irradiation range, the direction from the ranging device 10 to these multiple objects is treated as the same direction. For example, in the configuration shown in FIG. 7, the irradiation range 43 includes data of the tire 41 and the road surface 42. Therefore, in the irradiation range 43, the direction from the ranging device 10 to each of the tire 41 and the road surface 42 is treated as the same detection direction d1. However, the actual direction from the ranging device 10 to each of the tire 41 and the road surface 42 is different. Also, as shown in FIG. 3, the part of the captured image 40 corresponding to the irradiation range 43 includes the tire 41 and the road surface 42. However, in the data on the intensity of the reflected wave in the irradiation range 43 shown in FIG. 4, only one object (here, a tire 41) is treated as being present at the distance corresponding to the intensity peak P1 where the reflected wave has the greatest intensity.

In this embodiment, the control unit 34 executes the following process to determine the direction of arrival of each of the multiple reflected waves reflected by the multiple objects when it determines that data of multiple objects is included in one irradiation range. By determining the direction of arrival of each of the multiple reflected waves, the control unit 34 can determine the position of each of the multiple objects in real space.

First, the control unit 34 determines whether multiple intensity peaks are detected from the data on the intensity of the reflected wave in one irradiation range of the distance measuring device 10. The control unit 34 may detect intensity peaks that exceed an intensity threshold. The intensity threshold may be set assuming the amount of noise in the reflected wave. For example, in the case of data on the reflected wave in the irradiation range 43 shown in FIG. 4, the control unit 34 determines that two intensity peaks, i.e., intensity peaks P1 and P2, are detected from the data on the intensity of the reflected wave. If the control unit 34 determines that multiple intensity peaks are detected from the data on the intensity of the reflected wave in one irradiation range, it determines that one irradiation range includes data on multiple objects.

When the control unit 34 determines that one irradiation range contains data of multiple objects, it determines the direction of arrival of each of the multiple reflected waves corresponding to each of the multiple intensity peaks based on the captured image. As described above with reference to FIG. 3, the size of a pixel in the captured image is smaller than the size of a pixel in the ranging image. Therefore, the control unit 34 determines the direction of arrival of each of the multiple reflected waves in one irradiation range by using information about the pixels of the captured image. The process of determining the direction of arrival is described below. Hereinafter, an irradiation range determined to contain data of multiple objects is also referred to as a "first irradiation range." For example, in FIG. 3, irradiation range 43 is the first irradiation range.

<Processing for identifying direction of arrival>
The control unit 34 divides the captured image into a plurality of image regions by performing bilateral filter and superpixel processing on the captured image. The image regions can be regions in which pixels with similar characteristics are arranged consecutively. The characteristics are, for example, the luminance or color of the pixels. In other words, the image regions are composed of pixels with similar luminance information or pixels with similar color information.

Hereinafter, the portion of the captured image that corresponds to the first illumination range is also referred to as the "first partial image." For example, in FIG. 3, the first partial image is the portion that corresponds to illumination range 43 of captured image 40. When captured image 40 is divided into a plurality of image regions as shown in FIG. 3, the first partial image that corresponds to illumination range 43 includes at least a portion of the image region that corresponds to tire 41 and a portion of the image region that corresponds to road surface 42.

The control unit 34 uses the above-mentioned features of the captured image and the data on the multiple intensity peaks and distance of the reflected wave in the first irradiation range to associate the image area corresponding to the object with one of the multiple intensity peaks of the reflected wave. Hereinafter, the other irradiation range adjacent to the first irradiation range is also described as the "second irradiation range." The part of the captured image corresponding to the second irradiation range is also described as the "second partial image." For example, in FIG. 3, the second irradiation range is irradiation ranges 44, 45, 46, 47, 48, 49, 50, and 51. Irradiation ranges 44 to 51 are irradiation ranges that surround irradiation range 43. The second partial image is the part of the captured image 40 that corresponds to each of irradiation ranges 44 to 51.

The control unit 34 determines whether the second partial image includes the first image area. The first image area is an image area corresponding to the first object among the multiple image areas after division. The first object is any one of the multiple objects included in the first illumination range, i.e., the first partial image. For example, in FIG. 3, the first object is a tire 41. In this case, the control unit 34 determines whether the second partial image corresponding to each of the illumination ranges 44 to 51 includes the first image area corresponding to the tire 41. The control unit 34 may determine whether the second partial image includes the first image area by determining whether the characteristics of the pixels included in the second partial image are the same as the characteristics of the pixels corresponding to the first object. The control unit 34 may consider the characteristics of the pixels included in the second partial image to be the same as the characteristics of the pixels corresponding to the first object when the characteristics of the pixels included in the second partial image and the characteristics of the pixels corresponding to the first object are similar within a predetermined range. For example, when the control unit 34 uses pixel brightness as a feature, the control unit 34 may determine that the features of the plurality of pixels are similar when the difference between values indicating brightness information of the plurality of pixels is within a predetermined range of values. Also, when the control unit 34 uses pixel color information as a feature, the control unit 34 may determine that the features of the plurality of pixels are similar when the difference between values indicating color information of the plurality of pixels (e.g., RGB values) is within a predetermined range of values. When the control unit 34 determines that the feature of the pixel included in the second partial image is the same as the feature of the pixel corresponding to the first object, the control unit 34 determines that the second partial image includes the first image area. For example, in FIG. 3, the control unit 34 determines that the second partial image corresponding to each of the irradiation ranges 44, 46, and 49 includes the first image area corresponding to the tire 41.

When the control unit 34 determines that the second partial image includes the first image area, it acquires data on the reflected wave in the second irradiation range corresponding to the second partial image. The control unit 34 compares the intensity peak of the acquired reflected wave with each of the multiple intensity peaks of the reflected wave in the first irradiation range, thereby associating the first image area with any of the multiple intensity peaks of the reflected wave in the first irradiation range. For example, in FIG. 3, the control unit 34 determines that the second partial image corresponding to the irradiation range 44 includes the first image area corresponding to the tire 41. In this case, the control unit 34 acquires data on the reflected wave in the irradiation range 44 as shown in FIG. 5. The control unit 34 compares the intensity peak P3 of the reflected wave as shown in FIG. 5 with each of the multiple intensity peaks of the reflected wave in the irradiation range 43 as shown in FIG. 4, thereby associating the first image area corresponding to the tire 41 with the intensity peak P1.

The control unit 34 determines whether or not the second partial image includes a second image area that is the same as or similar to the first object. The second image area is an image area that corresponds to the second object among the multiple image areas after division. The second object is any one of the multiple objects included in the first illumination range, i.e., the first partial image. However, the second object is an object different from the first object. For example, in FIG. 3, the second object is the road surface 42.

　In the same or similar manner as the first object, when the control unit 34 determines that the second partial image includes the second image area, it acquires data (second data) of the reflected wave in the second irradiation range corresponding to the second partial image. In the same or similar manner as the first object, the control unit 34 compares the intensity peak of the acquired reflected wave with each of the multiple intensity peaks of the reflected wave in the first irradiation range, thereby associating the second image area with any of the multiple intensity peaks of the reflected wave. For example, in FIG. 3, the control unit 34 determines that the second partial image corresponding to the irradiation range 45 includes the second image area corresponding to the road surface 42. In this case, the control unit 34 acquires data of the reflected wave in the irradiation range 45 as shown in FIG. 6. The control unit 34 compares the intensity peak P4 of the reflected wave as shown in FIG. 6 with each of the multiple intensity peaks of the reflected wave in the irradiation range 43 as shown in FIG. 4, thereby associating the second image area corresponding to the road surface 42 with the intensity peak P2.

The control unit 34 repeatedly executes the above-mentioned process for each object included in the first irradiation range, thereby associating the image area corresponding to each object with one of the multiple intensity peaks of the reflected wave in the first irradiation range. When the control unit 34 associates them, it identifies the direction of light from a predetermined position included in the image area corresponding to the object toward the imaging device 20. The predetermined position included in this image area is, for example, a position corresponding to the center of the image area included in the first partial image. The control unit 34 identifies the identified light direction as the direction of arrival of the reflected wave whose intensity peak is associated with the image area. In other words, the control unit 34 identifies the identified light direction as the direction of arrival of the reflected wave reflected by an object included in the first irradiation range.

For example, in FIG. 8, the control unit 34 identifies a light direction d2 that travels from a predetermined position included in the first image area corresponding to the tire 41 toward the imaging device 20. The control unit 34 identifies the light direction d2 as the arrival direction of the reflected wave reflected by the tire 41. The control unit 34 also identifies a light direction d3 that travels from a predetermined position included in the second image area corresponding to the road surface 42 toward the imaging device 20. The control unit 34 identifies the light direction d3 as the arrival direction of the reflected wave reflected by the road surface 42.

<Object position identification process>
When the control unit 34 specifies the direction of arrival of the reflected wave reflected by each object included in the first irradiation range, the control unit 34 may specify the position of each object in the real space. The control unit 34 may specify the position of each object included in the first irradiation range in the three-dimensional real space by the data of the distance corresponding to each intensity peak in the first irradiation range and the direction of arrival of the reflected wave corresponding to each intensity peak. For example, in FIG. 3, the control unit 34 specifies the position of the tire 41 in the three-dimensional real space for the irradiation range 43 by the data of the distance D1 corresponding to the intensity peak P1 shown in FIG. 4 and the light direction d2 shown in FIG. 8 specified as the direction of arrival. The control unit 34 also specifies the position of the road surface 42 in the three-dimensional real space by the data of the distance D2 corresponding to the intensity peak P2 shown in FIG. 4 and the light direction d3 shown in FIG. 8 specified as the direction of arrival.

The control unit 34 may generate point cloud data based on the positions of the identified objects in three-dimensional real space. The point cloud data is data that indicates the positions in three-dimensional space of objects that exist around the distance measuring device 10. The control unit 34 may display the generated point cloud data on the display of the output unit 32, or may output the generated point cloud data to an external storage medium by the output unit 32. The control unit 34 may transmit the generated point cloud data to another system of the mobile body 2 by the communication unit 31.

The process of associating the image area corresponding to the object with any one of the multiple intensity peaks in the first irradiation range is not limited to the above example. Other examples of this association process are described below.

<Other Example 1>
The control unit 34 determines whether the first image area includes the second partial image. For example, in FIG. 3, the control unit 34 determines whether the first image area corresponding to the tire 41 includes the second partial images corresponding to each of the illumination ranges 44 to 51.

When the control unit 34 determines that the first image area includes the second partial image, it acquires distance data of the second irradiation range corresponding to the second partial image. The control unit 34 compares the acquired distance data of the second irradiation range with multiple distance data corresponding to each of the multiple intensity peaks of the reflected wave in the first irradiation range. By comparing these distance data, the control unit 34 associates the first image area with any of the multiple intensity peaks of the reflected wave in the first irradiation range. For example, in FIG. 3, the control unit 34 determines that the first image area corresponding to the tire 41 includes the second partial image corresponding to the irradiation range 44. In this case, the control unit 34 compares the distance data corresponding to the intensity peak P3 in the irradiation range 44 as shown in FIG. 5 with the distance data corresponding to each of the multiple intensity peaks as shown in FIG. 4. By comparing these distance data, the control unit 34 associates the first image area corresponding to the tire 41 with the intensity peak P1.

　In the same or similar manner as the first object, the control unit 34 determines whether the second image area includes the second partial image. In the same or similar manner as the first object, when the control unit 34 determines that the second image area includes the second partial image, it associates the second image area with any of the multiple intensity peaks of the reflected wave in the first irradiation range. For example, in FIG. 3, the control unit 34 determines that the second image area corresponding to the road surface 42 is included in the second partial image corresponding to the irradiation range 45. In this case, the control unit 34 compares the distance data corresponding to the intensity peak P4 in the irradiation range 45 as shown in FIG. 6 with the distance data corresponding to each of the multiple intensity peaks as shown in FIG. 4. By comparing these data, the control unit 34 associates the second image area corresponding to the road surface 42 with the intensity peak P2.

The control unit 34 repeatedly performs the above-mentioned process for each object included in the first irradiation range, thereby associating the image area corresponding to each object with one of the multiple intensity peaks of the reflected wave in the first irradiation range.

<Other Example 2>
The control unit 34 may execute an image recognition process on the captured image. The control unit 34 may execute the image recognition process to identify the front-to-back relationship of multiple objects included in the first partial image. In the front-to-back relationship of multiple objects, "front" means the one closer to the distance measuring device 10. For example, in FIG. 3, the control unit 34 executes the image recognition process to identify that the first partial image corresponding to the illumination range 43 includes the tire 41 and a part of the road surface 42. Furthermore, the control unit 34 specifies that the tire 41 is in front of the road surface 42 as viewed from the distance measuring device 10. The control unit 34 may execute the image recognition process to identify an image area corresponding to each object included in the first illumination range.

The control unit 34 associates the image area corresponding to each object with one of the multiple intensity peaks of the reflected wave based on the front-to-back relationship of the identified multiple objects and the magnitude relationship of the multiple distance data corresponding to each of the multiple intensity peaks of the reflected wave in the first irradiation range. For example, in FIG. 3, the control unit 34 associates intensity peak P1 with the first image area corresponding to the tire 41 based on the front-to-back relationship of the tire 41 and the road surface 42 and the magnitude relationship of the distance data corresponding to each intensity peak of the reflected wave as shown in FIG. 4. The control unit 34 also associates intensity peak P2 with the second image area corresponding to the road surface 42.

<Other Example 3>
When the control unit 34 is unable to identify the direction of arrival of each of the multiple reflected waves in the first irradiation range based on one of the multiple features described above (e.g., luminance), the control unit 34 may identify the direction of arrival of each of the multiple reflected waves based on other of the multiple features (e.g., color). Furthermore, when the control unit 34 is unable to identify the direction of arrival of each of the multiple reflected waves in any or all of the multiple features described above, the control unit 34 may execute the image recognition process described above. Furthermore, when the control unit 34 is unable to identify the direction of arrival of each of the multiple reflected waves by executing the image recognition process, the control unit 34 may identify the direction of arrival of each of the multiple reflected waves in the first irradiation range based on the above-mentioned features.

FIG. 9 is a flowchart showing an example procedure of an object detection process according to an embodiment of the present disclosure. The object detection process shown in FIG. 9 corresponds to an example of a control method according to this embodiment. For example, when the moving object 2 starts moving, the control unit 34 starts the object detection process. The control unit 34 may detect that the moving object 2 has started moving by communicating with another system of the moving object 2 via the communication unit 31.

The control unit 34 receives data on the distance to the object for each irradiation range and data on the intensity of the reflected wave from the distance measuring device 10 via the communication unit 31 (step S1). The control unit 34 also receives data on the captured image from the imaging device 20 via the communication unit 31 (step S2).

The control unit 34 determines whether or not multiple intensity peaks are detected from the data on the intensity of the reflected waves in one irradiation range (step S3). If the control unit 34 determines that multiple intensity peaks are detected from the data on the intensity of the reflected waves in one irradiation range (step S3: YES), it determines that data on multiple objects is included in one irradiation range, and proceeds to processing in step S4. On the other hand, if the control unit 34 does not determine that multiple intensity peaks are detected from the data on the intensity of the reflected waves in one irradiation range (step S3: NO), it does not determine that data on multiple objects is included in one irradiation range, and proceeds to processing in step S5.

In the process of step S4, the control unit 34 identifies the direction of arrival of each of the multiple reflected waves corresponding to each of the multiple intensity peaks based on the captured image received in the process of step S2.

In the process of step S5, the control unit 34 determines whether or not the process of step S3 has been performed for all irradiation ranges received in the process of step S1. If the control unit 34 determines that the process of step S3 has been performed for all irradiation ranges (step S5: YES), the control unit 34 proceeds to the process of step S6. On the other hand, if the control unit 34 does not determine that the process of step S3 has been performed for all irradiation ranges (step S5: NO), the control unit 34 returns to the process of step S3.

In step S6, the control unit 34 generates point cloud data based on the distance image data received in step S1 and the direction of arrival identified in step S4.

After processing step S6, the control unit 34 returns to processing step S1. When repeatedly executing the processing of steps S1 to S6, the control unit 34 ends the object detection processing, for example, when the moving object 2 stops. The control unit 34 may detect that the moving object 2 has stopped moving by communicating with other systems of the moving object 2 via the communication unit 31.

In this manner, in the control device 30 according to this embodiment, the control unit 34 identifies the positions of the first object and the second object in real space based on the first reflected wave and the second reflected wave and the captured image including the first object and the second object. The first reflected wave and the second reflected wave are respectively electromagnetic waves radiated in a first direction in real space, for example, to one irradiation range, and reflected by the first object and the second object. For example, the first reflected wave is electromagnetic waves irradiated to the irradiation range 43 as shown in FIG. 3 and reflected by the tire 41. For example, the second reflected wave is electromagnetic waves irradiated to the irradiation range 43 as shown in FIG. 3 and reflected by the road surface 42. By identifying the positions of the first object and the second object in real space based on the first reflected wave and the second reflected wave and the captured image, the positions of multiple objects in real space can be identified even when data of multiple objects is included in one irradiation range.

Here, the distance measuring device 10 can measure the distance to an object with high accuracy compared to, for example, a stereo camera. However, the resolution of the distance measuring image generated by the distance measuring device 10 is often lower than the resolution of the distance measuring image generated by a stereo camera or the like. Even if the resolution of the distance measuring image is low and data for multiple objects is included in one illumination range, i.e., in one pixel, in this embodiment, the control unit 34 can identify the positions of each of the multiple objects in real space as described above. As a result, in this embodiment, objects can be detected with high accuracy.

Therefore, according to this embodiment, it is possible to detect objects with high accuracy.

Furthermore, in the control device 30 according to this embodiment, the control unit 34 may identify the positions of the first object and the second object in real space based on the correspondence between the first and second regions and the first and second reflected waves. The first region is a region including an image of the first object in the captured image. For example, the first region is a region including an image of the tire 41 in the captured image 40 as shown in FIG. 3. The second region is a region including an image of the second object in the captured image. For example, the second region is a region including an image of the road surface 42 in the captured image 40 as shown in FIG. 3. The control unit 34 may perform the above-mentioned processing to identify the correspondence between the first and second regions and the first and second reflected waves based on the intensity peaks of the first and second reflected waves detected from the data on the intensity of the reflected waves. The control unit 34 may identify the correspondence between the first and second regions and the first and second reflected waves by associating each of the first and second regions with either the intensity peaks of the first and second reflected waves detected from the data on the intensity of the reflected waves. By using the above-described correspondence, the position of the object can be identified with high accuracy.

Furthermore, in the control device 30 according to this embodiment, the control unit 34 may identify a real space direction corresponding to a first region for the distance measuring device 10 and a real space direction corresponding to a second region for the distance measuring device 10. The real space direction corresponding to the first region for the distance measuring device 10 is, for example, a light direction d2 as shown in FIG. 8. The real space direction corresponding to the second region for the distance measuring device 10 is, for example, a light direction d3 as shown in FIG. 8. The control unit 34 may identify the positions of the first object and the second object in real space based on the identified real space directions corresponding to the first region and the second region, and data on the distances to the first object and the second object. With this configuration, the positions of the objects can be identified with high accuracy.

Furthermore, in the control device 30 according to this embodiment, the control unit 34 may identify the positional relationship of the first object and the second object relative to the distance measuring device 10 by performing image recognition processing on the captured image. The control unit 34 may identify the front-to-back relationship of the multiple objects as the positional relationship of the multiple objects relative to the distance measuring device 10, as described above. The control unit 34 may identify the correspondence between the first and second regions and the first and second reflected waves based on the identified positional relationship of the first and second objects. With this configuration, the position of the object can be identified efficiently.

In addition, in the control device 30 according to this embodiment, the control unit 34 may identify the positions of the first object and the second object in real space based on the first reflected wave, the second reflected wave, the third reflected wave, and the captured image. The third reflected wave is the electromagnetic wave irradiated to the second irradiation range in real space and reflected by the first object. For example, when the second irradiation range is the irradiation range 44 shown in FIG. 3, the third reflected wave is the electromagnetic wave irradiated to the irradiation range 44 and reflected by the tire 41. In this case, the third reflected wave is the reflection corresponding to the intensity peak P3 among the data of the reflected wave in the irradiation range 44 shown in FIG. 5. By using the third reflected wave in this way, the position of the object can be identified with high accuracy. The control unit 34 may also identify the positions of the first object and the second object in real space based on the correspondence between the first and second regions and the first, second, and third reflected waves. The control unit 34 may associate the first and second reflected waves with the first and second regions based on the third reflected wave. For example, as described above, the control unit 34 compares each of the multiple intensity peaks of the reflected wave as shown in FIG. 4 with the intensity peak of the reflected wave as shown in FIG. 5. Through such a comparison, the control unit 34 identifies a correspondence between the first reflected wave corresponding to the intensity peak P1 shown in FIG. 4 and the third reflected wave corresponding to the intensity peak P3 shown in FIG. 5. The control unit 34 may also obtain first data on the intensity of the reflected wave including the first reflected wave and the second reflected wave, and second data on the intensity of the reflected wave including the third reflection. The first data is, for example, data on the intensity of the reflected wave in the irradiation range 43 as shown in FIG. 4. The second data is, for example, data on the reflected wave in the irradiation range 44 as shown in FIG. 5.

In addition, the control device 30 according to this embodiment may identify the positions of the first object and the second object in real space based on the directions of arrival of the first reflected wave, the second reflected wave, and the third reflected wave, the first partial image, the second partial image, and data on the distances to the first object and the second object. The data on the distances to the first object and the second object may be based on distance data measured based on the first reflected wave, the second reflected wave, and the third reflected wave. The control unit 34 may identify the direction of arrival of the third reflected wave based on the position of the scanning range in real space relative to the second irradiation range.

Furthermore, in this embodiment, the optical axis of the imaging device 20 and the optical axis of the distance measuring device 10 may coincide. By making the optical axis of the imaging device 20 and the optical axis of the distance measuring device 10 coincide, it is possible to reduce the occlusion area between the imaging device 20 and the distance measuring device 10. An occlusion area is an area of the captured image and the distance measuring image that is captured or detected in one, but not captured or detected in the other. By reducing the occlusion area, the control unit 34 can accurately identify the direction of arrival of each of the multiple reflected waves corresponding to each of the multiple intensity peaks, based on the captured image generated by the imaging device 20.

Although the present disclosure has been described based on the drawings and examples, it should be noted that a person skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions, etc. included in each functional unit can be rearranged so as not to cause logical inconsistencies. Multiple functional units, etc. may be combined into one or divided. Each embodiment of the present disclosure described above is not limited to being implemented faithfully to each of the embodiments described, and may be implemented by combining each feature as appropriate or omitting some of them. In other words, the contents of the present disclosure can be modified and corrected in various ways by a person skilled in the art based on the present disclosure. Therefore, these modifications and corrections are included in the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. can be added to other embodiments so as not to cause logical inconsistencies, or replaced with each functional unit, each means, each step, etc. of other embodiments. In addition, in each embodiment, multiple functional units, each means, each step, etc. can be combined into one or divided. Furthermore, each of the above-described embodiments of the present disclosure is not limited to being implemented faithfully according to each of the described embodiments, but may be implemented by combining each feature or omitting some features as appropriate.

In the above embodiment, the multiple objects are described as being tires 41 and road surface 42 as shown in FIG. 3. However, the multiple objects are not limited to this. The multiple objects may be any objects as long as they overlap on the captured image seen by the detection system 1. For example, the multiple objects may be different types of objects, such as a vehicle and a pedestrian, or a vehicle and an obstacle on the road surface. The multiple objects may also be the same type of object, such as multiple vehicles or multiple obstacles on the road surface. Furthermore, the detection system 1 may be mounted not only on a moving body, but also on an observation device such as a roadside unit installed on the road surface or a surveillance camera that monitors the surroundings.

In one embodiment, (1) the control device comprises:
The positions of the first object and the second object in the real space are identified based on a first reflected wave reflected by a first object and a second reflected wave reflected by a second object of an electromagnetic wave radiated in a first direction in the real space, and an image of the real space including the first object and the second object.

(2) The control device of (1) above,
The positions of the first object and the second object in the real space may be identified based on a correspondence between a first region of the captured image that includes an image of the one object and a second region of the captured image that includes an image of the second object, and the first reflected wave and the second reflected wave.

(3) The control device according to (1) or (2) above,
Acquire intensity data including an intensity of the first reflected wave and an intensity of the second reflected wave;
The correspondence relationship may be identified based on intensity peaks of the first reflected wave and the second reflected wave detected from the intensity data.

(4) Any one of the control devices according to (1) to (3) above,
The control device includes:
performing an image recognition process on the captured image to identify a positional relationship of the first object and the second object with respect to the distance measuring device;
The correspondence relationship may be identified based on the positional relationship.

(5) Any one of the control devices according to (1) to (4) above,
acquiring data on a distance to an object and the intensity data from a distance measuring device that measures a distance to the object by emitting an electromagnetic wave into a real space and detecting a wave reflected by the object;
The positions of the first object and the second object in real space may be identified based on the distances to the first object and the second object obtained by using a direction in real space corresponding to the first area and a direction in real space corresponding to the second area relative to the distance measuring device, and the distance data measured based on the first reflected wave and the second reflected wave.

In one embodiment, (6) the control device is
The positions of the first object and the second object in the real space may be identified based on a first reflected wave that is formed by reflecting an electromagnetic wave from a first object and a second reflected wave that is formed by reflecting an electromagnetic wave from a second irradiation range in the real space, a third reflected wave that is formed by reflecting an electromagnetic wave from the first object and a second irradiation range in the real space, and an image generated by imaging the real space including the first irradiation range and the second irradiation range.

(7) The control device according to (6) above,
The positions of the first object and the second object in the real space may be identified based on a correspondence between a first region of the captured image that includes an image of the one object and a second region of the captured image that includes an image of the second object, and the first reflected wave, the second reflected wave, and the third reflected wave.

(8) The control device according to (6) or (7) above,
A correspondence relationship between the first reflected wave and the second reflected wave and the first region and the second region may be identified based on the third reflected wave.

(9) Any one of the control devices according to (6) to (8) above,
Among the first reflected wave, the second reflected wave, and the third reflected wave, the first reflected wave and the third reflected wave having a difference in intensity peak within a predetermined range may be associated as a reflected wave from the first object.

(10) Any one of the control devices according to (6) to (9) above,
Arrival directions of the first reflected wave, the second reflected wave, and the third reflected wave;
A first partial image corresponding to the first illumination range and a second partial image corresponding to the second illumination range of the captured image;
and distances to the first object and the second object acquired based on distance data measured based on the first reflected wave, the second reflected wave, and the third reflected wave.
The positions of the first object and the second object in the real space may be identified.

In one embodiment, (11) a control method includes:
The method includes identifying the positions of the first object and the second object in the real space based on a first reflected wave reflected by a first object and a second reflected wave reflected by a second object of an electromagnetic wave radiated in a first direction in the real space, and an image of the real space including the first object and the second object.

In one embodiment, the control method includes:
The method includes identifying positions of the first object and the second object in the real space based on a first reflected wave formed by reflection by a first object and a second reflected wave formed by reflection by a second object of an electromagnetic wave irradiated to a first irradiation range in the real space, a third reflected wave formed by reflection by the first object of an electromagnetic wave irradiated to a second irradiation range in the real space, and an image generated by imaging the real space including the first irradiation range and the second irradiation range.

REFERENCE SIGNS LIST 1 Detection system 2 Mobile object 10 Distance measuring device 20 Imaging device 30 Control device 31 Communication unit 32 Output unit 33 Storage unit 34 Control unit 40 Captured image 41 Tire 42 Road surface 43, 44, 45, 46, 47, 48, 49, 50, 51 Irradiation range

Claims (12)

A control device that identifies the positions of the first object and the second object in real space based on a first reflected wave reflected by a first object and a second reflected wave reflected by a second object of electromagnetic waves emitted in a first direction in the real space, and an image of the real space including the first object and the second object. The control device according to claim 1, wherein the control device determines the positions of the first object and the second object in the real space based on a correspondence between a first region of the captured image that includes an image of the first object and a second region of the captured image that includes an image of the second object, and the first reflected wave and the second reflected wave. The control device includes:
Acquire intensity data including an intensity of the first reflected wave and an intensity of the second reflected wave;
The control device according to claim 2 , wherein the corresponding relationship is determined based on intensity peaks of the first reflected wave and the second reflected wave detected from the intensity data. The control device includes:
performing an image recognition process on the captured image to identify a positional relationship of the first object and the second object with respect to the distance measuring device;
The control device according to claim 2 , wherein the corresponding relationship is determined based on the positional relationship. The control device includes:
acquiring data on a distance to an object and the intensity data from a distance measuring device that measures a distance to an object by emitting an electromagnetic wave into a real space and detecting a wave reflected by the object;
4. The control device according to claim 3, which identifies the positions of the first object and the second object in real space based on the distances to the first object and the second object obtained by using a direction in real space corresponding to the first area and a direction in real space corresponding to the second area relative to the distance measuring device, and the distance data measured based on the first reflected wave and the second reflected wave. A control device that identifies the positions of the first object and the second object in the real space based on a first reflected wave and a second reflected wave that are formed by reflecting electromagnetic waves irradiated onto a first irradiation range in the real space from a first object and a second reflected wave that are formed by reflecting electromagnetic waves irradiated onto a second irradiation range in the real space from the first object, and an image generated by imaging the real space including the first irradiation range and the second irradiation range. The control device according to claim 6, wherein the control device identifies the positions of the first object and the second object in the real space based on a correspondence between a first region of the captured image that includes an image of the first object and a second region of the captured image that includes an image of the second object, and the first reflected wave, the second reflected wave, and the third reflected wave. The control device according to claim 7, wherein the control device determines a correspondence between the first reflected wave and the second reflected wave and the first region and the second region based on the third reflected wave. The control device according to claim 8, wherein the control device associates the first reflected wave and the third reflected wave, among the first reflected wave, the second reflected wave, and the third reflected wave, whose difference in intensity peaks is within a predetermined range, as reflected waves from the first object. The control device includes:
Arrival directions of the first reflected wave, the second reflected wave, and the third reflected wave;
A first partial image corresponding to the first illumination range and a second partial image corresponding to the second illumination range of the captured image;
and distances to the first object and the second object acquired based on distance data measured based on the first reflected wave, the second reflected wave, and the third reflected wave.
The control device according to claim 8 or 9, further comprising a controller configured to identify positions of the first object and the second object in the real space. A control method including identifying the positions of the first object and the second object in real space based on a first reflected wave reflected by a first object and a second reflected wave reflected by a second object of an electromagnetic wave radiated in a first direction in the real space, and an image of the real space including the first object and the second object. a control method including: identifying positions of the first object and the second object in a real space based on a first reflected wave formed by reflection by a first object and a second reflected wave formed by reflection by a second object of an electromagnetic wave irradiated to a first irradiation range in a real space, a third reflected wave formed by reflection by the first object of an electromagnetic wave irradiated to a second irradiation range in the real space, and an image generated by imaging the real space including the first irradiation range and the second irradiation range.

Images