JP6682833B2 - Database construction system for machine learning of object recognition algorithm - Google Patents

Description

  The present invention relates to a technology for recognizing or detecting an event such as the presence or state of an object through an algorithm using machine learning from an output of a certain sensor, and more specifically, while the vehicle such as an automobile travels. , A system for constructing a database for machine learning in a recognition algorithm using machine learning that can be used when detecting a nearby vehicle, a pedestrian or other obstacles using an in-vehicle sensor Pertain to.

  For example, in a moving vehicle or other moving body, detection of an object such as a preceding vehicle or moving body or other obstacle by using an in-vehicle sensor such as a camera, a LIDAR (Laser Imaging Detection and Ranging), or a radar. Various techniques have been proposed. In the detection of an object by these in-vehicle sensors, there is a weak point in detectability depending on the state of the surroundings of the vehicle or the moving body, the state of the object to be detected, etc. In this case, the object can be detected with high accuracy in a bright environment such as during the day, but at night, the detection accuracy of the object decreases, and the existence of the object depends on the distance between the vehicle-mounted sensor and the object. The presence or absence and the accuracy of the distance to the object may change. On the other hand, in the case of object detection by LIDAR, the detection accuracy is not so affected by the surrounding brightness, and the distance and direction to the object can be detected with high accuracy. , And generally, the recognition accuracy of the type of the object is slightly lower than that of the object detection by the image pickup by the camera. Therefore, conventionally, in the case of recognizing or detecting the presence and / or the state of the surrounding objects by the sensor mounted on the vehicle or the moving body as described above, using a plurality of different types of sensors, Various configurations have been proposed for recognizing or detecting an object from the outputs obtained from these sensors, or for integrating the outputs obtained from these sensors to recognize or detect the object. .

  For example, in Patent Document 1, a LIDAR point sequence representing position information of another vehicle in front of the own vehicle obtained by LIDAR is compared with another vehicle in front of the vehicle identified by the image obtained by the camera. It is disclosed that the position of the end portion of the other vehicle is specified so as to be superimposed on the end portion. Further, in Patent Document 2, in the recognition device for an object in front of the vehicle while traveling, the stereo image processing of images captured by the two electronic cameras 1 and 2 and the laser range finder are used to detect the front of the vehicle. In the plane area, the position where the object exists is detected, and the detection frequency of the existence of the object in those detection results is weighted and summed up. There is proposed a configuration for determining that there exists. In this case, it is described that the weighting ratio is adjusted by learning. In Patent Document 3, in recognition of an object in front of a moving vehicle, a group of reflection points existing in a range of substantially the vehicle width at positions at substantially equal distances based on the position of each reflection point specified by a laser radar. Is a vehicle candidate point group, the vehicle candidate point group is subjected to coordinate conversion into the coordinate system of the camera and collated with a rectangular area extracted by the camera, and if the vehicle candidate point group after coordinate conversion substantially matches the rectangular area, A configuration for determining the vehicle candidate point group as a front vehicle is described. In Patent Document 4, in a device that detects the degree of risk by referring to risk information from a state quantity obtained by converting a feature quantity vector extracted from image data in front of a vehicle into a one-dimensional state, The risk information that determines the degree is extracted from the driver's operation information as the teacher information, and the risk information is learned by using the correlation between the state quantity converted into the one-dimensional feature vector and the teacher information. Is proposed. In Patent Document 5, in a device that determines a road surface state from the reflection intensity (reflected light data) of a laser beam applied to a road surface while a vehicle is traveling, the reflected light data used for the determination is known. A configuration generated by learning using reflected light data obtained while a vehicle is traveling on a road surface is disclosed. Then, in Patent Document 6, in recognition of an object in front of a moving vehicle, an edge histogram in a camera image and a light-receiving intensity in a laser radar for a gaze area in a captured image during actual traveling An X, Y direction vector and a laser vector for a vehicle candidate region are created and fused from the histogram to generate a fusion vector, and the distance between the fusion vector and the dictionary fusion vector prepared in advance is generated in the vector space. A configuration is disclosed in which when the vehicle size is small, the vehicle candidate area is predicted to be a vehicle.

JP, 2009-98025, A Japanese Patent Laid-Open No. 2000-329852 Japanese Patent Laid-Open No. 2003-84064 Japanese Patent Laid-Open No. 2008-238831 JP-A-2014-228300 JP-A-2003-99762

  By the way, as in the case of detecting an object such as a preceding vehicle or an obstacle in front of the running vehicle by using a camera, LIDAR, radar, etc., a certain detection or recognition target object is detected by a plurality of types of sensors. In the detection or recognition of an object of the type that is detected or recognized using, as already mentioned, there are differences in the objects or environments that can be detected accurately depending on the type of sensor. Therefore, with respect to a certain detection target or in a certain detection environment, one sensor can accurately recognize the existence and / or type of an object, but the other sensor alone Then, it may be difficult to perform accurate recognition, or proper signal processing, for example, processing for recognizing the type of object may not be established in the first place. In addition, depending on the type of sensor, the processing conditions for executing accurate recognition may change during use. In such a case, that is, an appropriate signal processing method and condition for recognizing the existence, type and / or motion state of an object (hereinafter referred to as “object recognition”) from the output of a certain sensor. Alternatively, when the procedure is unknown, uncertain, or variable, the output of the sensor is separated from the output of the sensor by a machine learning method as a method of adjusting an appropriate signal processing method, condition or procedure for the output of the sensor. In order to achieve the most accurate object recognition from the output of the former sensor, the signal processing method is collated with the detection result using the output of the sensor in the state where the object can be recognized with high accuracy. , Configuring or adjusting conditions or procedures. That is, according to the configuration in which the detection result of the first sensor is used as teacher data, the signal processing method, condition or procedure for obtaining the detection result from the output of the second sensor is configured or adjusted by machine learning. It is expected that the accuracy of the detection result by the sensor will be improved. At that time, in general, the higher the accuracy of the teacher data and the larger and more diverse the database, that is, the more the data of various distances, directions, types of vehicles, etc., are included, the more machine learning will be performed. It is known that the accuracy of the detection result obtained by the second sensor obtained as a result of is improved.

  In this regard, in the construction or adjustment of a signal processing method, condition or procedure of a certain sensor using so-called “machine learning”, that is, construction of a recognition algorithm using machine learning, a large amount of supervised learning is performed. Data, that is, data obtained by collating the detection result of the first sensor, which is the teacher data in machine learning, with the output and / or the detection result of the second sensor, which is the input data, is required. Regarding such collation data, conventionally, a person manually creates teacher data or learning data by processing such as tagging the detection result of the first sensor and the output and / or detection result of the second sensor. Was there. However, it is difficult to manually create a large amount of supervised learning data at low cost, and it may be difficult to manually create the teacher data depending on the type of data. For example, in the image of the front of the vehicle, if you give a bounding box (frame to be set in the image) to the image of the preceding vehicle, you can easily add the bounding box manually if the image is good. However, it is difficult to accurately give a bounding box to the image of the preceding vehicle in the backlight condition or the image taken at night, and the intensity map of the point cloud acquired by LIDAR or the millimeter wave radar is used. It is not easy to manually create teacher data for data with complicated shapes and ambiguous boundaries. Furthermore, it is difficult to manually give teacher data to information that cannot be directly observed from images or moving images, such as vehicle speed. Therefore, in order to prepare a large amount of supervised learning data for constructing a recognition algorithm using machine learning, it is possible to automatically collect such learning data without manual operation by a person. preferable.

  Thus, an object of the present invention is a technique for executing recognition of an object such as another vehicle around a vehicle, a pedestrian, an obstacle, etc. using a plurality of types of sensors, Provide a system that automatically collects supervised learning data and builds a database of supervised learning data for machine learning to perform object recognition from the output of another sensor used as teacher data It is to be.

The above problem is a system that builds a database that accumulates supervised learning data used for machine learning for configuration or adjustment of an algorithm that executes recognition of an object in a surrounding area of a vehicle based on an output of a sensor,
A first sensor that sequentially detects the state of the surrounding area of the vehicle,
A first object recognition means for sequentially recognizing an object in the surrounding area of the vehicle based on output data obtained sequentially from the first sensor,
A second sensor that sequentially detects the state of the surrounding area of the vehicle,
When the reliability of the recognition result data of the object by the first object recognition means is equal to or higher than a predetermined degree, the output data sequentially obtained from the second sensor is used as the input data in the machine learning. Using the recognition result data of the objects sequentially recognized by the object recognizing means as the teacher data in machine learning, the data associating means for associating the teacher data with the input data corresponding to the teacher data,
This is achieved by a system including a learning data storage unit that stores a pair of associated input data and teacher data as supervised learning data for machine learning.

  In the above-mentioned configuration, the "sensor" may be any sensor, but typically, the surrounding area of the vehicle such as a camera, a LIDAR, or a millimeter wave radar that detects the state of the surrounding area of the vehicle as an image. It may be one that is usually used for detecting the state of. As used herein, the term "configuration or adjustment of an algorithm for performing object recognition" means, as described above, a suitable signal processing method for performing object recognition based on the output of a certain sensor. To configure a condition or procedure, or to adjust various conditions in an already constructed algorithm. As is well known in the field of machine learning, when an object recognition algorithm is constructed or developed using machine learning, a recognition result (correction result (correction result) that is correct for a certain sensor output (input data) Structure of arithmetic processing and determination processing and / or used in them so as to give a correct recognition result when an arbitrary output of the sensor is given by referring to a data group corresponding to (teaching data) Various parameters to be performed are determined. “Algorithm configuration or adjustment” means the configuration of such arithmetic processing and determination processing and / or the determination of various parameters and the like. In the present invention, in the case of “recognizing an object”, as described above, the presence of the object is detected and / or the type of the object is identified (whether the object is stationary, the detected object is Identification of whether it is a vehicle, person, animal or other obstacle) and / or estimation of the motion state (position, posture (orientation), speed, etc.) of an object. Further, in the above, the first sensor and the second sensor are typically sensors of different types or different specifications (measurement position, measurement angle range, different sensitivity, etc.), For example, a combination of a camera and a LIDAR may be used, but in some embodiments, the same type of sensor may be used. The "first object recognizing means" gives the result of object recognition based on the output of the first sensor by any method known in the art according to the type of the "first sensor". It may be a means.

  In brief, in the above configuration, as described above, at least two sensors, that is, the first sensor and the second sensor, are prepared, and each of them is sequentially arranged in the vehicle according to the respective aspects. The detection of the state of the surrounding area can be executed. Here, as the first sensor, a sensor for which an algorithm for recognizing an object is determined based on the output is selected as described above, and as the second sensor, a machine learning is performed. Based on the output, the one for which the algorithm for performing the recognition of the object is to be constructed or adjusted is selected. Then, the recognition result data of the object obtained sequentially based on the output of the first sensor (whether the object exists or not) may be included in the machine learning. It is selected as teacher data, the output of the second sensor is selected as input data in machine learning, these data are associated, and stored as supervised learning data in the database.

  The correspondence between the recognition result data of the object of the first sensor and the output of the second sensor and the storage of the supervised learning data are automatically executed by the processing of the computer. In one aspect, these processes store, for example, the output data of the first sensor, the recognition result data of the object of the first sensor, and the output data of the second sensor in an arbitrary data storage device. Afterwards (offline processing), while referring to the travel log data of the vehicle, the correspondence between the recognition result data of the object of the first sensor and the output of the second sensor and the storage of the supervised learning data are stored. May be performed. The data storage device may be installed in the own vehicle, or data may be transmitted via a network communication to a data storage device provided in an external facility to be stored. Good. Further, as another aspect, it may be sequentially executed while the vehicle is traveling. As described above, the correspondence between the recognition result data of the object by the first sensor and the output of the second sensor means that the reliability of the recognition result data of the object by the first sensor has a predetermined degree. It is executed only when the above is the case, that is, when the recognition result data of the object by the first sensor is usable as the teacher data.

  The above configuration of the present invention may further include a configuration in which the object recognition result obtained based on the output data of the second sensor is used as teacher data. That is, before the object recognition algorithm by the second sensor is configured, the object recognition result based on the output data of the first sensor is used as teacher data, and after the object recognition algorithm by the second sensor is configured. , A combination of the object recognition data based on the output of the first sensor and the object recognition data based on the output of the second sensor as teacher data, and this and the output of the second sensor currently obtained Correspondence may be made. In this case, a loop of learning data is formed in the object recognition algorithm based on the output of the second sensor, and further improvement in accuracy is expected.

  Thus, the above-mentioned device of the present invention further includes a second object recognition means for sequentially recognizing an object in the surrounding area of the vehicle based on the output data obtained sequentially from the second sensor, The data associating unit may further be configured to use the recognition result data of the objects sequentially recognized by the second object recognizing unit as the teacher data in the machine learning. According to such a configuration, the object recognition accuracy can be improved by the loop iteration processing, and the effect of compensating for the lack of the data amount can be obtained. In the configuration in which the recognition result data of the object obtained based on the output data of the second sensor is used as the teacher data to form the loop of the learning data, the recognition result data of the object is It is preferable to have a higher degree of reliability than when it is used for controlling. Therefore, when the reliability of the object recognition result data obtained based on the output data of the second sensor is higher than or equal to a predetermined degree higher than the degree to be satisfied when used for various controls, the teacher data May be used as.

  Furthermore, in another aspect, when at least two sensors are used, the recognition result data of the object of the at least two sensors is changed depending on the reliability of the recognition result data of each object. The one with higher reliability may be selected as the first sensor (provided that the reliability is equal to or higher than a predetermined degree), and the other may be selected as the second sensor. Further, a plurality of sensors may be used as the first sensor, and the first object recognition means may be configured to integrate the outputs of the plurality of sensors (sensor fusion) to give a recognition result of the object. It should be understood that such cases also belong to the scope of the present invention.

  Thus, according to the present invention described above, in the technique of executing recognition of an object such as a vehicle, a pedestrian, and an obstacle around the own vehicle using a plurality of types of sensors, the detection result of a certain sensor is A system that automatically prepares and collects supervised learning data for machine learning to execute object recognition from the output of another sensor used as supervised data, and builds a database of supervised learning data. It becomes possible to provide. According to this configuration, the labor of the user and the cost required to build the system are significantly reduced, or even when it is extremely difficult to build the system manually, it is possible to build a database of supervised learning data. Is obtained. In addition, if a configuration for forming a loop of learning data storage and object recognition algorithm configuration by machine learning by using the object recognition algorithm by the second sensor for preparation of learning data, the object recognition algorithm is used. Is improved, and the accuracy, diversity, and quantity of the obtained database are improved, and further improvement of the accuracy of the object recognition algorithm is expected. That is, by repeating the loop processing of the object recognition algorithm configuration by learning data storage and machine learning, it is possible to construct a database of highly accurate and diverse supervised learning data in a larger amount or more efficiently. Become.

  Other objects and advantages of the present invention will be apparent from the following description of the preferred embodiments of the present invention.

FIG. 1A is a block diagram showing the configuration of one embodiment of a database construction system for collecting and storing learning data for machine learning in a sensor output recognition algorithm according to the present invention. It is a figure. FIG. 1B is a diagram showing the process in the database construction in the form of a flow chart. 2A to 2C are diagrams showing the configuration of another embodiment of the database construction system according to the present invention in the form of a block diagram. 3 (A) to 3 (C) are diagrams showing the configuration of still another embodiment of the database construction system according to the present invention in the form of a block diagram. 4 (A) and 4 (B) are still another embodiment of the database construction system according to the present invention, and in particular, are diagrams showing the configuration in the case where there are two types of teacher data in the form of a block diagram. . 5A and 5B are still another embodiment of the database construction system according to the present invention, and in particular, an object obtained by an object recognition algorithm established by using learning data accumulated in the database. It is the figure which represented the structure at the time of utilizing a recognition result as teacher data in the format of a block diagram.

SC ... Camera image output SL ... LIDAR point cloud output SR ... Millimeter wave radar output Rr, Rr1, Rr2 ... Recognition result

  The present invention will be described in detail with respect to some preferred embodiments with reference to the accompanying drawings. In the drawings, the same reference numerals indicate the same parts.

Basic Configuration and Operation of System In the database construction system for executing the learning data collection for machine learning of the object recognition algorithm according to the present invention, as described in the section "Outline of the Invention", , The recognition result of the object sequentially obtained based on the output of the first sensor is used as teacher data, and the sequential output of the second sensor is used as input data. The preparation, collection, and accumulation of "supervised learning data" utilized in machine learning that constructs or adjusts algorithms for recognizing objects based on outputs is performed sequentially. Here, the “supervised learning data” used for machine learning is a set of the output of the second sensor and the recognition result of the first sensor associated with this. Therefore, the database construction system has, as a basic configuration, a first sensor, a recognition unit that sequentially recognizes an object based on the output of the first sensor, a second sensor, and a recognition unit that sequentially recognizes the object. Of the recognition result obtained in step 1 and the associating means for associating the output of the second sensor sequentially obtained with the output of the associated second sensor and the recognition result of the first sensor And a data storage means for storing the set as "supervised learning data". The recognizing means for sequentially recognizing the object based on the first sensor, the associating means, and the data storing means are connected to each other by a computer system, that is, an ordinary type, bidirectional common bus. It is realized by a system including a microcomputer having a CPU, a ROM, a RAM, and an input / output port device and a drive circuit, and the operation of each means described above is automatically achieved by executing a computer program in such a system.

  One of the most basic embodiments of the database construction system according to the present invention, as described above, is for a system for recognizing an object in a peripheral area of a vehicle, as illustrated in FIG. It is used to build a database. Referring to the figure, in such a database construction system, first, an in-vehicle camera is adopted as the "first sensor", and an in-vehicle LIDAR is adopted as the "second sensor". The camera and the LIDAR may each be of any of the known or known types used in the art for recognition of objects in the vehicle's peripheral area (particularly in the front area). In the case of the illustrated embodiment, the image output SC of the vehicle-mounted camera, which is the first sensor, is sequentially given to the object recognition unit, where the object in the image captured by the camera is detected. The presence and / or type of images of, for example, identification of (preceding) vehicles, pedestrians, animals, roadside fixed objects, roadside obstacles, etc., is arbitrary or known in the art. The recognition result Rr is sequentially output. Then, the recognition result Rr obtained based on the camera image and the point cloud data SL which is the output data of the LIDAR about the same time period or the same area as the camera image are sequentially given to the associating unit, In the associating unit, a process of associating the recognition result Rr and the point cloud data SL corresponding thereto with the former as the teacher data and the latter as the input data is executed, and the learning data is sequentially acquired. The prepared learning data is sequentially stored in the database, that is, the data storage means. In the configuration illustrated in FIG. 1A, the camera and the LIDAR are mounted on the vehicle, but the object recognition unit, the association unit, and the database may be mounted on the vehicle, or externally. It may be installed in any facility. When the object recognition unit, the association unit, and the database are installed in an external facility, the output of the camera and the LIDAR and / or the recognition result of the object recognition unit can be transmitted to an external facility through a wireless communication unit of any format, a network, or the like. May be sent to.

Example of associating process The preparation of the learning data in the above associating unit, that is, the associating process between the output of the second sensor and the recognition result of the first sensor is performed by the first sensor recognition result and the first sensor recognition result. It can be achieved by various arbitrary methods, for example, the processing procedure described in the above-mentioned series of patent documents or any other procedure, depending on the respective expression forms or modes with the outputs of the two sensors. . For example, as shown in FIG. 1A, the case where the recognition result Rr based on the camera image is adopted as the recognition result of the first sensor and the point cloud data SL of LIDAR is adopted as the output of the second sensor The attaching process may be achieved by the process illustrated in FIG. Specifically, referring to the figure, in the associating unit, first, the recognition result Rr based on the camera image is acquired (step 10) and the LIDAR point cloud data is acquired (step 12). To be done. Here, the expression form of the recognition result Rr based on the camera image is defined, for example, in a state where the range of the image of the vehicle or the image of another object in the image specifies the type of the object of the image. The range of the image may be represented by the coordinates in the image or the coordinates in the space captured in the image. The LIDAR point cloud data may be represented by position coordinates in each space of the detection points (light reflection points). Thus, when the respective data are obtained, the LIDAR point cloud data is divided into partial point groups by grouping according to the position in space in a general manner as the processing of the LIDAR point cloud data. (Step 14). Here, the division pattern may be arbitrarily determined in advance, and for example, a division method by equal solid angle division, or for each point, the distance between that point and the nearest point may be less than a threshold value. A division method or the like may be adopted in which the same group is set and those having a threshold value or more are set as different groups.

  After that, the partial point group obtained above is projected on the camera image (step 16), and the number of points included in the image of the recognition target in the camera image, for example, the range of the image of the vehicle is projected for each partial point group. Is calculated (step 18). Here, the projection of the partial point cloud onto the camera image is performed by the coordinates and points of the space in which the object of the recognition target image exists according to the expression format of the range of the recognition target image and the expression format of the point cloud data. It may be executed by using a geometrical transformation of the two so that the coordinates of the space in which the group data is represented match each other. In the recognition result Rr based on the camera image, typically, when an image of a certain object is recognized, the degree of reliability or certainty of its existence is expressed as a percentage. (For example, the existence probability is 75%). In that case, in order to reduce collation errors in calculating the ratio of the number of points included in the range of the recognition target image in the camera image, the reliability of the existence of the object in the recognition result is reduced. The calculation of the ratio of the number of points included in the image range is performed only for the range of the image having a predetermined degree or more, or only for the range of the image of the object located closest to the camera. Good. The above-mentioned predetermined degree may be set experimentally or theoretically as appropriate. Thus, a partial point group in which the ratio of the number of points included in the range of the image is equal to or greater than a predetermined threshold value that is arbitrarily set is associated with the recognition target, for example, a point group belonging to the vehicle, The point cloud is associated as a point cloud belonging to an object that is not a recognition target, and is stored as supervised learning data in the database (steps 20 to 24).

  The process illustrated in FIG. 1B is automatically executed by the process of the computer. Typically, the output data of the camera (first sensor), the recognition result Rr (recognition result data of the object of the first sensor) and the LIDAR point cloud data (output data of the second sensor) are arbitrary data. Correspondence of a series of data and storage of supervised learning data, which are stored in a storage device and illustrated in FIG. 1B, are offline processes, and are, for example, vehicle running log data (sensor data and sensor data). Stored in a). Further, as another aspect, the association of data and the storage of supervised learning data may be sequentially executed along with the acquisition of sensor data, in which case the supervised learning data is prepared every moment. Will be accumulated and accumulated.

  The accumulated "supervised learning data" is used in any manner for machine learning to construct or adjust an object recognition algorithm based on the point cloud data detected by LIDAR, thus obtained. Based on the point cloud data detected by LIDAR using an algorithm, it may be used for recognition of an object in a peripheral area of the vehicle, and the recognition result may be used for various controls in the vehicle. It should be understood that the recognition result based on the camera image may also be used for various controls in the vehicle. Further, the illustrated example is one example of the associating process, and associates the recognition result (teacher data) based on the first sensor and the output format (input data) of the second sensor according to the expression format or mode. It should be appreciated that processing may be performed. What is important is that the computer automatically performs the process of sequentially acquiring the recognition result based on the first sensor and the output of the second sensor, associating them with each other, and preparing and storing the learning data. It is a point to achieve it.

Example of Other Embodiments of Database Construction System According to Present Invention The database construction system according to the present invention may be realized by the configurations shown in FIGS. 2 to 5 in addition to the configuration illustrated in FIG. In any case, the teacher data and the input data may be associated with each other according to their respective expression formats, and the learning data may be prepared and stored in the same manner as described above.

(1) When the recognition result based on the output of LIDAR is used as the teacher data (FIG. 2 (A))
In the situation where LIDAR can obtain more accurate recognition result than the camera (backlight, night, rainy weather, etc.), the recognition result based on the output of LIDAR is used as the teacher data and the learning data is prepared and stored using the camera image as the input data. May be done. In this case, using learning data, an algorithm for recognizing an object based on a camera image is configured or adjusted by machine learning. Further, one of the configuration shown in FIG. 1A and the configuration shown in FIG. 2A can be selected depending on which of the LIDAR and the camera can obtain a correct recognition result. May be.

(2) In the case of a configuration (sensor fusion) for recognizing an object using both a camera image and LIDAR point cloud data (FIGS. 2B and 2C)
In this case, the recognition result of the object based on the camera image and the LIDAR point cloud data is used as teacher data, and the LIDAR point cloud data (FIG. 2B) or the camera image (FIG. 2C) is used as input data. Training data preparation and storage may be performed using. The teacher data may be information such as the distance to the object extracted from the recognition result of the object, the speed of the object, and the like. The camera image used as the input data may be a moving image.

(3) In the case of a configuration in which an object is recognized by a LIDAR and a millimeter wave radar (RADAR) (FIGS. 3A and 3B)
In the configurations of FIGS. 1A and 2A to 2C, a millimeter wave radar may be used instead of the camera. Since the output SR of the millimeter wave radar becomes a radar reflection intensity map, when the millimeter wave radar is used as the first sensor, the object recognition unit uses an arbitrary method based on the radar reflection intensity map SR. As a means for recognizing the object, and the teacher data becomes the object recognition result Rr based on the radar reflection intensity map SR (FIG. 3A). Further, in the case of a configuration (sensor fusion) in which object recognition is executed using both the radar reflection intensity map and the LIDAR point cloud data, the teacher data is the radar reflection intensity map SR and the LIDAR point cloud data SL. The object recognition result Rr based on and is obtained, and the input data is the radar reflection intensity map SR (FIG. 3B) or the LIDAR point cloud data (not shown). In particular, since it is difficult for humans to perform processing such as tagging a radar reflection intensity map of a millimeter wave radar, it is very advantageous that the processing can be automatically performed by a computer as described above.

(4) When adding other information to the learning data (Fig. 3 (C))
In addition to camera images, LIDAR point cloud data or radar reflection intensity maps, vehicle motion information such as vehicle speed and environment information Dt such as weather acquired by any sensor or detection device are added to the learning data. May be. In this case, it is expected that machine learning suitable for vehicle motion information and environment information will be possible.

(5) When using a plurality of teacher data (FIGS. 4A and 4B)
Two or more types of data (Rr1, Rr2) may be used as teacher data (one type in the examples described above). When there are two or more teacher data, information appropriately extracted from each data may be used in the association process. For example, in the example of FIG. 4A, the information on the position and type of the image of the object in the image is adopted from the recognition result Rr1 based on the camera image as the information referred to as the teacher data, and the radar reflection intensity of RADAR is used. From the recognition result Rr2 based on the map SR, the information about the distance to the object and the speed may be adopted. Further, as shown in FIG. 4B, when the recognition algorithm for the point cloud data of LIDAR which is the target of machine learning is established with a certain accuracy, recognition based on the point cloud data SL of LIDAR is performed. The result Rr2 may be adopted as the second teacher data.

(6) When the recognition result by the recognition algorithm obtained by machine learning is used as teacher data (FIGS. 5A and 5B)
After the object recognition algorithm based on the output of the second sensor is constructed or adjusted by machine learning using the learning data stored in the database, the recognition result using the recognition algorithm is further adopted as teacher data. You may For example, in the case of the configuration illustrated in FIG. 5A, first, similarly to the configuration described in FIG. 1A, the recognition result Rr1 based on the camera image and the point cloud data SL of LIDAR are obtained. After the preparation and storage of the learning data are performed through the association process for a certain period of time, an object recognition algorithm is configured based on the LIDAR point cloud data SL by machine learning using the learning data. Or adjusted. Then, the object recognition algorithm obtained by the machine learning executes object recognition based on the LIDAR point cloud data SL, and the recognition result Rr2 is also associated with the LIDAR point cloud data SL as teacher data. As a result, preparation and storage of learning data are executed. Here, there are two sets of teacher data, Rr1 and Rr2. For example, one of the teacher data with higher accuracy is preferentially selected according to the measurement situation and is associated with the input data. It may be like this. More specifically, for example, one teacher data is adjusted by adjusting the ratio of the contributions of the teacher data Rr1 and Rr2 in a mode in which the weight of a highly reliable recognition result that may be determined by an arbitrary method is increased. It may be prepared and associated with the input data. In this regard, when the recognition result Rr2 of the object recognition algorithm based on the output of the second sensor configured or adjusted by machine learning is used as teacher data, the reliability of the result is sufficiently high. It is preferable. Therefore, the recognition result Rr2 may be used as teacher data only when the reliability thereof is equal to or higher than a predetermined degree higher than the degree to be satisfied when used for various controls. In the above process, the machine learning using the learning data stored in the database may be executed in any mode.

  As described above, when the recognition result obtained by the machine learning using the learning data stored in the database is further used as the teacher data, a so-called learning data loop in the machine learning is formed. As the loop is repeated, it is expected that the recognition accuracy of the recognition algorithm targeted for machine learning will be improved.

  Further, as illustrated in FIG. 5B, a recognition algorithm of each object is configured by machine learning using learning data accumulated in a database for both outputs of two different types of sensors or It may be adjustable. In the case of the example in FIG. 5B, when the recognition of the object based on the camera image and the recognition of the object based on the LIDAR point cloud data are executed, the recognition algorithms of the respective objects are stored in the database. It is configured or adjusted by machine learning using the learned data. In the associating process, the teaching data based on the outputs of the different types of sensors may be appropriately associated with the outputs of the different types of sensors (that is, the learning data is prepared for each sensor). Be done.). According to such a configuration, a loop of learning data in machine learning is formed in the algorithm for recognizing each object of the outputs of different types of sensors, and the recognition accuracy of the object recognition algorithm based on both sensors is improved. Expected to improve.

  Thus, according to the series of database construction systems described above, the supervised learning data for machine learning for executing the recognition of an object from the output of another sensor using the detection result of a certain sensor as the teacher data is generated. It is possible to provide a system that automatically adjusts and collects and builds a database of supervised learning data, which greatly reduces the labor of the user and the cost required to build the system, or is built manually. Even if it is very difficult, there is an advantage that a database of supervised learning data can be constructed.

  Although the above description is made in connection with the embodiments of the present invention, many modifications and changes can be easily made by those skilled in the art, and the present invention is limited to the embodiments illustrated above. It will be apparent that the present invention is not limited and applies to various devices without departing from the inventive concept.

  For example, the sensor used in the present invention may be a sensor fixed outside the vehicle, such as a fixed camera installed at an intersection.

Claims (1)

A system for constructing a database for accumulating supervised learning data used for machine learning for configuration or adjustment of an algorithm for recognizing an object in a surrounding area of a vehicle based on an output of a sensor,
A first sensor that sequentially detects the state of the surrounding area of the vehicle,
A first object recognition means for sequentially recognizing an object in a surrounding area of the vehicle based on output data obtained sequentially from the first sensor;
A second sensor that sequentially detects the state of the surrounding area of the vehicle,
When the reliability of the recognition result data of the object by the first object recognition means is a predetermined degree or more, the output data sequentially obtained by the second sensor is used as the input data in the machine learning. Using the recognition result data of the objects sequentially recognized by the first object recognition means as the teacher data in the machine learning, the teacher data and the input data corresponding to the teacher data are associated with each other. Means for associating data of the object,
A teacher for machine learning for configuring or adjusting an algorithm that executes recognition of an object in the surrounding area of the vehicle based on the output of the second sensor, based on the set of the associated input data and teacher data. A system including learning data storage means for storing as learning data.

Images