JP2022182094A - Mobile body control device, mobile body control method, and program - Google Patents

Abstract

To further improve the recognition accuracy of zone lines that partition a region where a mobile body traffics.SOLUTION: A mobile body control device includes: a recognition part for recognizing a peripheral situation of a mobile body on the basis of output of an external sensor; and a zone line recognition part for recognizing zone lines that partition a region where the mobile body travels on the basis of the peripheral situation recognized by the recognition part. When recognition accuracy of the zone lines is determined to be in a degraded state, the zone line recognition part extracts a predetermined region from the peripheral situation, extracts edges in the predetermined region extracted, in order to recognize the zone lines according to the extracted results.SELECTED DRAWING: Figure 2

Description

The present invention relates to a mobile body control device, a mobile body control method, and a program.

In recent years, research on automatic driving that automatically controls the running of a vehicle has been progressing. In relation to this, when lane markings for the lane in which the vehicle is traveling go from being detected to being undetected, the virtual lane markings are changed based on the positions of previously detected lane markings. A technique is known for estimating and controlling the travel of the own vehicle so that the own vehicle is at a predetermined position with respect to the estimated virtual lane marking (see, for example, Patent Document 1).

WO2018/012179

However, since the positions of previously detected lane markings do not always continue as they are, there are cases where lane markings that are different from the actual lane markings are recognized.

Aspects of the present invention have been made in consideration of such circumstances, and provide a moving body control apparatus and a moving body control method capable of further improving the recognition accuracy of lane markings that demarcate an area through which a moving body passes. , and to provide programs.

A moving body control device, a moving body control method, and a program according to the present invention employ the following configurations.
(1): A mobile body control device according to an aspect of the present invention includes a recognition unit that recognizes a surrounding situation of a mobile body based on an output of an external sensor, and a moving body based on the surrounding situation recognized by the recognition unit. a lane marking recognition unit that recognizes a lane marking that divides an area through which a body passes, wherein the lane marking recognition unit recognizes the surrounding area when it is determined that the recognition accuracy of the lane marking is degraded. The moving body control device extracts a predetermined area from a situation, extracts edges in the extracted predetermined area, and recognizes the marking line based on the extracted result.

(2): In the aspect (1) above, a storage control unit that stores, in a storage unit, information regarding the lane marking before the lane marking recognition unit determines that the recognition accuracy of the lane marking has decreased. wherein the marking line recognizing unit extracts a marking line candidate based on the edge extracted from the predetermined area, and stores information about the extracted marking line candidate and information about the marking line stored in the storage unit , the demarcation line that demarcates the area through which the moving object passes is recognized.

(3): In the aspect of (2) above, the information about the marking line includes at least one of the position, direction, and type of the marking line.

(4): In the aspect (2) or (3) above, the storage unit further stores map information, and the lane marking recognition unit determines that the recognition accuracy of the lane marking has decreased. before it is determined that the information relating to the lane markings acquired from the map information based on the position information of the moving body or the recognition accuracy of the lane markings stored in the storage unit is degraded. The predetermined area is extracted based on the information about the marking line.

(5): In any one of the above (1) to (4), the predetermined areas are set on the left and right sides of the traveling direction of the moving body.

(6): In any one of the above aspects (1) to (5), the lane marking recognition unit extracts edges in the surrounding situation, and determines the length, reliability, and quality of the extracted edges. When at least one of them is less than the threshold value, it is determined that the lane marking recognition accuracy is lowered.

(7): A moving body control method according to an aspect of the present invention is such that a computer recognizes a surrounding situation of a moving body based on an output of an external sensor, and the moving body passes based on the recognized surrounding situation. When it is determined that the recognition accuracy of the marking line is degraded, the predetermined area is extracted from the peripheral situation, and the edges in the extracted predetermined area are extracted. and a moving body control method for recognizing the lane markings based on the extracted result.

(8): A program according to an aspect of the present invention causes a computer to recognize a surrounding situation of a moving body based on an output of an external sensor, and determines an area through which the moving body passes based on the recognized surrounding situation. Recognizing a demarcation line, extracting a predetermined area from the surrounding situation when it is determined that the recognition accuracy of the demarcation line is degraded, and extracting an edge in the extracted predetermined area; A program for recognizing the partition line based on the extracted result.

According to the aspects (1) to (8) above, it is possible to further improve the recognition accuracy of the demarcation line that demarcates the area through which the moving object passes.

1 is a configuration diagram of a vehicle system 1 using a vehicle control device according to an embodiment; FIG. 3 is a functional configuration diagram of a first controller 120 and a second controller 160. FIG. It is a figure for demonstrating an example of lane marking recognition in embodiment. FIG. 4 is a diagram for explaining the content of recognized lane line information 192; FIG. It is a figure for demonstrating extracting a predetermined area|region. FIG. 7 is a diagram for explaining an example of extraction of marking line candidates; FIG. 4 is a diagram for explaining recognition of lane markings of a traveling lane from lane marking candidates; It is a flow chart which shows an example of a flow of processing performed by automatic operation control device 100 of an embodiment.

Hereinafter, embodiments of a mobile body control device, a mobile body control method, and a program according to the present invention will be described with reference to the drawings. In addition, below, a vehicle is used as an example of a mobile body. The vehicle is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its driving source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electric power generated by a generator connected to the internal combustion engine, or electric power discharged from a secondary battery or a fuel cell. Moreover, below, as an example, an embodiment in which the mobile body control device is applied to an automatically driving vehicle will be described. Automatic driving means, for example, automatically controlling one or both of steering and acceleration/deceleration of a vehicle to execute driving control. The vehicle driving control may include various driving assistance such as ACC (Adaptive Cruise Control), ALC (Auto Lane Changing), LKAS (Lane Keeping Assistance System), and TJP (Traffic Jam Pilot). The automatic driving vehicle may be partially or wholly controlled by the manual operation of the passenger (driver). Further, mobile objects may include, in addition to (or instead of) vehicles, ships, flying objects (including, for example, drones, aircraft, etc.), and the like.

[overall structure]
FIG. 1 is a configuration diagram of a vehicle system 1 using a vehicle control device according to an embodiment. The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, and a vehicle sensor 40. , a navigation device 50 , an MPU (Map Positioning Unit) 60 , a driving operator 80 , an automatic driving control device 100 , a driving force output device 200 , a braking device 210 , and a steering device 220 . These apparatuses and devices are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, and the like. Note that the configuration shown in FIG. 1 is merely an example, and a part of the configuration may be omitted, or another configuration may be added. A combination of the camera 10, the radar device 12, the LIDAR 14, and the object recognition device 16 is an example of the "external sensor". Note that the external sensor ES may include, for example, a sonar (not shown). The HMI 30 is an example of an "output section". The automatic operation control device 100 is an example of a "moving body control device".

The camera 10 is, for example, a digital camera using a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to an arbitrary location of a vehicle (hereinafter referred to as vehicle M) on which the vehicle system 1 is mounted. When imaging the front, the camera 10 is attached to the upper part of the front windshield, the rear surface of the rearview mirror, or the like. For example, the camera 10 repeatedly images the surroundings of the vehicle M periodically. Camera 10 may be a stereo camera.

The radar device 12 radiates radio waves such as millimeter waves around the vehicle M and detects radio waves (reflected waves) reflected by an object to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location of the vehicle M. As shown in FIG. The radar device 12 may detect the position and velocity of an object by the FM-CW (Frequency Modulated Continuous Wave) method.

The LIDAR 14 irradiates light (or electromagnetic waves with wavelengths close to light) around the vehicle M and measures scattered light. The LIDAR 14 detects the distance to the object based on the time from light emission to light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 is attached to any location on the vehicle M. Further, when the vehicle M is provided with a sonar, the sonar is provided, for example, at the front end portion and the rear end portion of the vehicle M and installed on a bumper or the like. Sonar detects an object (for example, an obstacle) within a predetermined distance from its installation position.

The object recognition device 16 performs sensor fusion processing on detection results obtained by some or all of the components of the external sensor ES (camera 10, radar device 12, LIDAR 14, and sonar) to determine the position, type, and Recognize speed, etc. The object recognition device 16 outputs recognition results to the automatic driving control device 100 . The object recognition device 16 may output the detection result of the external sensor ES to the automatic driving control device 100 as it is. In this case, the object recognition device 16 may be omitted from the vehicle system 1 .

The communication device 20 uses, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like to communicate with other vehicles existing in the vicinity of the vehicle M, or communicate with a wireless base. It communicates with various server devices through the station.

The HMI 30 outputs various types of information to the occupants of the vehicle M and receives input operations by the occupants. The HMI 30 includes, for example, various display devices, speakers, buzzers, touch panels, switches, keys, microphones, and the like. Various display devices are, for example, LCD (Liquid Crystal Display) and organic EL (Electro Luminescence) display devices. The display device is provided, for example, near the front of the driver's seat (the seat closest to the steering wheel) on the instrument panel, and is installed at a position where the passenger can view it through the gap between the steering wheel or through the steering wheel. Also, the display device may be installed in the center of the instrument panel. Also, the display device may be a HUD (Head Up Display). The HUD projects an image onto a portion of the front windshield in front of the driver's seat, thereby allowing the eyes of the passenger seated in the driver's seat to visually recognize a virtual image. The display device displays an image generated by the HMI control unit 170, which will be described later. Further, the HMI 30 may include a driving changeover switch or the like for switching between automatic driving and manual driving by the passenger.

The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects angular velocity about a vertical axis, a direction sensor that detects the orientation of the vehicle M, and the like. Also, the vehicle sensor 40 may include a position sensor that acquires the position of the vehicle M. FIG. A position sensor is, for example, a sensor that acquires position information (longitude/latitude information) from a GPS (Global Positioning System) device. Also, the position sensor may be a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50 .

The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route determination section 53. The navigation device 50 holds first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory. The GNSS receiver 51 identifies the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be specified or supplemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 40 . The navigation HMI 52 includes a display device, speaker, touch panel, keys, and the like. The navigation HMI 52 may be partially or entirely shared with the HMI 30 described above. The route determining unit 53, for example, from the position of the vehicle M specified by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI52 route (hereinafter referred to as map upper route) is determined with reference to the first map information 54 . The first map information 54 is, for example, information in which road shapes are represented by links indicating roads and nodes connected by the links. The first map information 54 may include road curvature, POI (Point Of Interest) information, and the like. A route on the map is output to the MPU 60 . The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized, for example, by the function of a terminal device such as a smart phone or a tablet terminal owned by the passenger. The navigation device 50 may transmit the current position and the destination to the navigation server via the communication device 20 and acquire a route equivalent to the route on the map from the navigation server.

The MPU 60 includes, for example, a recommended lane determination unit 61, and holds second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determining unit 61 divides the route on the map provided from the navigation device 50 into a plurality of blocks (for example, by dividing each block by 100 [m] in the vehicle traveling direction), and refers to the second map information 62. Determine recommended lanes for each block. The recommended lane decision unit 61 decides which lane to drive from the left. For example, when the route on the map has a branch point, the recommended lane determination unit 61 determines the recommended lane so that the vehicle M can travel a rational route to the branch destination.

The second map information 62 is map information with higher precision than the first map information 54 . The second map information 62 includes, for example, lane marking information such as the position, direction, and type of lane markings that divide one or more lanes included in the road (hereinafter simply referred to as “margining lines”), and lane marking information. This includes lane center information or lane boundary information, etc. based on In addition, the second map information 62 includes information on protection fences such as guardrails and fences provided along the extension direction of the road, chatter bars (road studs), curbs, median strips, road shoulders, sidewalks, and the like. may The second map information 62 includes road information (road type), legal speed limit (speed limit, maximum speed, minimum speed), traffic regulation information, address information (address/zip code), facility information, telephone number information, etc. may be included. The second map information 62 may be updated at any time by the communication device 20 communicating with other devices.

The driving operator 80 includes, for example, a steering wheel, an accelerator pedal, a brake pedal, a shift lever, and other operators. A sensor that detects the amount of operation or the presence or absence of operation is attached to the driving operation element 80, and the detection result is applied to the automatic driving control device 100, or the driving force output device 200, the brake device 210, and the steering device. 220 to some or all of them. The manipulator does not necessarily have to be annular, and may be in the form of a deformed steering wheel, joystick, button, or the like.

The automatic operation control device 100 includes, for example, a first control unit 120, a second control unit 160, an HMI control unit 170, a memory control unit 180, and a storage unit 190. The first control unit 120, the second control unit 160, and the HMI control unit 170 are each implemented by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit) (including circuitry), or by cooperation of software and hardware. The program may be stored in advance in a storage device such as the HDD or flash memory of the automatic operation control device 100 (a storage device having a non-transitory storage medium), or may be detachable such as a DVD or CD-ROM. It is stored in a storage medium, and may be installed in the HDD or flash memory of the automatic operation control device 100 by attaching the storage medium (non-transitory storage medium) to the drive device. A combination of the action plan generation unit 140 and the second control unit 160 is an example of the “operation control unit”. The HMI control section 170 is an example of an "output control section."

The storage unit 190 may be implemented by the above-described various storage devices, SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), RAM (Random Access Memory), or the like. . The storage unit 190 stores, for example, recognition marking line information 192, programs, and other various types of information. Details of the recognized lane line information 192 will be described later. Further, the storage unit 190 may store the above-described map information (the first map information 54 and the second map information 62).

FIG. 2 is a functional configuration diagram of the first control unit 120 and the second control unit 160. As shown in FIG. The 1st control part 120 is provided with the recognition part 130 and the action plan production|generation part 140, for example. The first control unit 120, for example, realizes in parallel a function based on AI (Artificial Intelligence) and a function based on a model given in advance. For example, the "recognize intersections" function performs in parallel recognition of intersections by deep learning, etc., and recognition based on predetermined conditions (signals that can be pattern-matched, road markings, etc.). It may be realized by scoring and evaluating comprehensively. This ensures the reliability of automated driving.

The recognition unit 130 recognizes space information indicating the surrounding situation of the vehicle M based on information input from the external sensor ES, for example. For example, the recognition unit 130 recognizes the positions of objects (for example, other vehicles and other obstacles) in the vicinity of the vehicle M, and states such as speed and acceleration. The position of the object is recognized, for example, as a position on absolute coordinates with a representative point (the center of gravity, the center of the drive shaft, etc.) of the vehicle M as the origin, and used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by an area. If the object is a moving object such as another vehicle, the "state" of the object may be the acceleration or jerk of the other vehicle, or the "behavior state" (for example, whether the vehicle is changing lanes or is about to change lanes). may include

Further, the recognition unit 130 recognizes the lane in which the vehicle M is traveling (driving lane) from the circumstances around the vehicle M, for example. Note that lane recognition is performed by a marking line recognition unit 132 provided in the recognition unit 130 . The details of the function of the lane marking recognition unit 132 will be described later. The recognition unit 130 also recognizes adjacent lanes adjacent to the driving lane. An adjacent lane is, for example, a lane that can travel in the same direction as the driving lane. The recognition unit 130 also recognizes stop lines, obstacles, red lights, toll booths, road signs, and other road events.

Further, when recognizing the driving lane, the recognition unit 130 recognizes the position and posture of the vehicle M with respect to the driving lane. The recognition unit 130 uses, for example, the deviation of the reference point of the vehicle M from the lane center and the angle formed with the line connecting the lane center in the traveling direction of the vehicle M as the relative position and orientation of the vehicle M with respect to the traveling lane. may recognize. Alternatively, the recognition unit 130 may recognize the position of the reference point of the vehicle M with respect to one of the side edges (division lines or road boundaries) of the driving lane as the relative position of the vehicle M with respect to the driving lane. good. Here, the reference point of the vehicle M may be the center of the vehicle M or the center of gravity. Also, the reference point may be an end (front end or rear end) of the vehicle M, or may be a position where one of the wheels of the vehicle M is present.

In principle, the action plan generation unit 140 drives the recommended lane determined by the recommended lane determination unit 61, and further, the vehicle M automatically (operated by the driver) so as to respond to the surrounding conditions of the vehicle M. ) to generate a target trajectory to travel in the future. The target trajectory includes, for example, velocity elements. For example, the target trajectory is represented by arranging points (trajectory points) that the vehicle M should reach in order. A trajectory point is a point to be reached by the vehicle M for each predetermined travel distance (for example, about several [m]) along the road. A target velocity and acceleration for each is generated as part of the target trajectory. Also, the trajectory point may be a position that the vehicle M should reach at each predetermined sampling time. In this case, the information on the target velocity and target acceleration is represented by the intervals between the trajectory points.

The action plan generator 140 may set an automatic driving event (function) when generating the target trajectory. Autonomous driving events include constant speed driving events, low speed following driving events, lane change events, branching events, merging events, takeover events, and the like. The action plan generator 140 generates a target trajectory according to the activated event.

The second control unit 160 controls the driving force output device 200, the braking device 210, and the steering device 220 so that the vehicle M passes the target trajectory generated by the action plan generating unit 140 at the scheduled time. do.

The second control unit 160 includes an acquisition unit 162, a speed control unit 164, and a steering control unit 166, for example. The acquisition unit 162 acquires information on the target trajectory (trajectory point) generated by the action plan generation unit 140 and stores it in a memory (not shown). Speed control unit 164 controls running driving force output device 200 or brake device 210 based on the speed element associated with the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curve of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized by, for example, a combination of feedforward control and feedback control. As an example, the steering control unit 166 performs a combination of feedforward control according to the curvature of the road ahead of the vehicle M and feedback control based on deviation from the target trajectory.

HMI control unit 170 notifies the driver of vehicle M of predetermined information through HMI 30 . The predetermined information includes, for example, driving assistance information. The driving support information includes, for example, the speed of the vehicle M, the engine speed, the remaining amount of fuel, the radiator water temperature, the travel distance, the state of the shift lever, the lane markings recognized by the object recognition device 16, the automatic driving control device 100, and the like. It includes information such as lanes, other vehicles, lanes in which the vehicle M should travel, future target trajectories, and the like. Further, the driving assistance information may include information such as information indicating switching of the driving mode, which will be described later, and driving state by driving assistance (for example, the type of automatic operation being executed such as LKAS or ALC). For example, the HMI control unit 170 may generate an image including the predetermined information described above, display the generated image on the display device of the HMI 30, generate a sound indicating the predetermined information, and transmit the generated sound to the HMI 30. may be output from the speaker. Further, the HMI control section 170 may output information received by the HMI 30 to the communication device 20, the navigation device 50, the first control section 120, and the like.

The running driving force output device 200 outputs running driving force (torque) for the vehicle M to run to the driving wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU (Electronic Control Unit) for controlling these. The ECU controls the above configuration in accordance with information input from the second control unit 160 or information input from the operation operator 80 .

The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motors according to information input from the second control unit 160 or information input from the driving operator 80 so that brake torque corresponding to the braking operation is output to each wheel. The brake device 210 may include, as a backup, a mechanism that transmits hydraulic pressure generated by operating a brake pedal included in the operation operator 80 to the cylinders via a master cylinder. The brake device 210 is not limited to the configuration described above, and is an electronically controlled hydraulic brake device that controls the actuator according to information input from the second control unit 160 to transmit the hydraulic pressure of the master cylinder to the cylinder. good too.

The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the orientation of the steered wheels. The steering ECU drives the electric motor according to information input from the second control unit 160 or information input from the steering wheel 82 of the driving operator 80 to change the direction of the steered wheels.

[Regarding lane marking recognition]
The lane marking recognition in the embodiment will be described below. In addition, below, the scene where LKAS control by automatic driving|operating shall be performed shall be demonstrated. The LKAS control is, for example, a control that assists the vehicle M in maintaining its lane by controlling at least the steering of the vehicle M so that the vehicle M runs near the center of the lane while recognizing the lane markings. is.

FIG. 3 is a diagram for explaining an example of marking line recognition in the embodiment. In the example of FIG. 3, a road RD including two lanes L1 and L2 that can travel in the same direction (X-axis direction in the figure), and lane L1 along the extension direction (X-axis direction) of the road RD at a speed VM A moving vehicle M is shown. Lane L1 is assumed to be a traffic area for vehicles M defined by lane markings LL and CL. Lane L2 is an area defined by lane markings CL and RL, and is adjacent to lane L1. In the example of FIG. 3, it is assumed that objects (road structures) OB1 and OB2 such as guardrails are provided on both sides of the road RD (outside the division lines LL and RL when viewed from the center of the road RD). do. The object OB1 is installed along the extending direction of the lane marking LL, and the object OB2 is installed along the extending direction of the lane marking RL. Also, in the example of FIG. 3, a range (hereinafter referred to as a recognizable range) RA where an object can be recognized by the external sensor ES is shown. For convenience of explanation, the recognizable range RA shown in FIG. 3 indicates a region in the forward direction of the vehicle M, but may include both sides and rear of the vehicle M. As shown in FIG. Also, the recognizable range RA differs depending on the performance of the external sensor ES, for example.

The lane marking recognition unit 132 recognizes the lane marking of the lane L1 on which the vehicle M travels, based on, for example, spatial information indicating the surrounding conditions within the recognizable range RA by the external sensor ES. Note that the lane marking recognition is repeatedly executed at a predetermined timing. The predetermined timing may be, for example, a predetermined cycle, or may be timing based on the speed of the vehicle M or the traveling distance.

For example, the lane marking recognizing unit 132 extracts edges from the captured image of the recognizable range RA captured by the camera 10, and recognizes the position of the lane marking based on the extraction result. An edge is, for example, a pixel (or a group of pixels) whose pixel value difference from its surrounding pixels is larger than a reference, that is, a characteristic pixel. The lane marking recognition unit 132 uses, for example, a predetermined differentiation filter, a Prewitt filter, a Sobel filter, or another edge extraction filter for the luminance value of each pixel in the image, Extract edges. Note that the edge extraction filter described above is merely an example, and the lane marking recognition unit 132 may extract edges based on other filters or algorithms.

Further, based on the edge extraction result, if there is an edge line segment (for example, a straight line or a curved line) whose length is equal to or greater than the first threshold, the lane marking recognition unit 132 recognizes the line segment as a lane marking. do. Also, the marking line recognition unit 132 may connect line segments of edges whose positions and directions are similar to each other. “Approximate” means that the difference (difference) in edge position and direction is within a predetermined range. Also, "approximate" may mean that the degree of similarity is equal to or greater than a predetermined value.
Note that even if the line segment is equal to or greater than the first threshold, if the line segment is curved and has a curvature equal to or greater than a predetermined value (curvature radius is equal to or less than a predetermined value), the line segment recognition unit 132 may be recognized as not being a lane marking. As a result, line segments that are clearly not marking lines can be excluded, and the recognition accuracy of marking lines can be improved.

In addition to (or instead of) the length of the edge, the lane marking recognition unit 132 may derive one or both of the reliability and quality of the edge. For example, the marking line recognition unit 132 derives the reliability of edges as marking lines according to the degree of continuity and the degree of scattering of the extracted edges. For example, the marking line recognition unit 132 increases the reliability as the line segments of the edges are more continuous or as the edges are less scattered in the extending direction. In addition, the lane marking recognition unit 132 compares the edges extracted from the left and right sides of the vehicle M based on the position of the vehicle M. The greater the similarity of edge continuity and scattering, the higher the reliability of each edge as a lane marking. can be increased. Further, the marking line recognition unit 132 increases the quality of the marking line obtained from the edges as the number of extracted edges increases. Note that the quality may be replaced with an index value (quality value) that increases as the quality increases, for example.

Further, the lane marking recognition unit 132 determines whether or not the recognition accuracy of the lane marking has deteriorated. For example, the recognition accuracy of lane markings is affected when the actual lane markings are scratched or dirty, when road reflections occur due to outside light near the exit of a tunnel, when it rains, when outside lights or lights from oncoming lanes reflect the road surface, and when heavy rainfall occurs. It decreases due to deterioration in the performance of the external sensor ES, etc. due to the weather. For example, when the line segment length of the edge extracted by the edge extraction is less than the first threshold value, the lane marking recognition unit 132 determines that the lane marking recognition accuracy is lowered.

Further, when the reliability and quality value of the edge are derived, the lane marking recognition unit 132 recognizes the lane marking when the reliability is less than the second threshold or when the quality value is less than the third threshold. It may be determined that the recognition accuracy is lowered. That is, for example, the lane marking recognition unit 132 recognizes the lane marking when at least one of the edge length, reliability, and quality is less than each threshold value based on the result of the edge extraction process. It may be determined that the accuracy is lowered. As a result, it is possible to more accurately determine whether or not the recognition accuracy of the lane marking has decreased using a plurality of conditions. Note that the lane marking recognition unit 132 uses a criterion similar to or similar to the above-described determination criteria instead of determining whether or not the lane marking recognition accuracy is reduced, so that the lane marking is recognized normally. You may judge whether it recognizes.

When the marking line recognition unit 132 determines that the recognition accuracy of the marking line is not degraded, the storage control unit 180 stores the marking line recognition result (information about the marking line) by the marking line recognition unit 132 as a recognized marking. Stored in the line information 192 . The information about the lane marking includes, for example, information about the state of the lane marking.

FIG. 4 is a diagram for explaining the content of the recognized lane line information 192. As shown in FIG. The recognized lane marking information 192 is, for example, information in which recognized lane marking state information is associated with the vehicle position. The vehicle position is position information of the vehicle M acquired from the vehicle sensor 40 . The marking line state information includes, for example, the position, direction, and type of the recognized marking line. The position is, for example, the position of the lane marking with the recognized position of the vehicle M as a reference. The direction is, for example, the extending direction of the division line with the position of the vehicle M as a reference. Types of lane markings and the like are included. The type is, for example, the line type (solid line, dashed line), width, and color of the marking line. The types may also include, for example, the presence or absence of road studs, the presence or absence of median strips, and the like. When the left and right lane markings of the vehicle M are recognized, the memory control unit 180 stores the positions, directions, and types of both lane markings. Further, the storage control unit 180 may store the above-described edge length, reliability, and quality information in the recognized lane marking information 192 . Note that the storage control unit 180, for example, stores information about lane markings for a short period (for example, several seconds to several minutes) before it is determined that the recognition accuracy of the lane markings is degraded, in the recognized lane marking information 192. Store. This makes it possible to reduce the amount of data compared to storing long-term data.

Further, when the marking line recognition unit 132 determines that the recognition accuracy of the marking line is lowered, the lane marking recognition unit 132 extracts a predetermined area from the surrounding situation recognized by the recognition unit 130, and detects the edges of the extracted predetermined area. is extracted, and the marking line of the lane L1 on which the vehicle M travels is recognized based on the extraction result. In the scene of FIG. 3, the rubbing W1 on the marking line LL and the dirt D1 on the marking line CL are present within the recognizable range RA. Therefore, the lane marking recognizing unit 132 determines that the recognition accuracy of the lane markings LL and CL of the lane L1 on which the vehicle M is traveling is lowered. In this case, the lane marking recognition unit 132 first extracts a predetermined area from the surrounding situation.

FIG. 5 is a diagram for explaining extraction of a predetermined area. In the example of FIG. 5, for convenience of explanation, the area of the lane L1 on which the host vehicle M mainly travels is shown schematically on the road RD shown in FIG. The lane marking recognizing unit 132 recognizes a lane marking that is estimated to have a high likelihood of existence (existing probability is equal to or greater than a predetermined value) as an example of a predetermined area in the recognizable range RA of the vehicle M by the external sensor ES. Extract existing regions. For example, the marking line recognition unit 132 extracts the marking line existing area based on the position of the marking line before the recognition accuracy of the marking line is lowered. For example, the lane marking recognition unit 132 refers to the vehicle position of the recognized lane line information 192 stored in the storage unit 190 based on the position information of the vehicle M acquired from the vehicle sensor 40, and based on the position information of the vehicle M, The position and direction of the lane marking state information associated with the vehicle position included in the predetermined distance range are extracted, and the lane marking existing area is extracted based on the extracted position and direction. For example, the marking line recognizing unit 132 extracts the marking line existing area based on the degree of dispersion of the extracted positions and directions. Moreover, the lane marking recognition unit 132 may extract a lane marking existing area in which the likelihood of the future lane marking being present is predicted to be high from the extracted displacement of the position and direction.

Further, the lane marking recognition unit 132 refers to the second map information 62 based on, for example, the position information of the vehicle M instead of (or in addition to) the recognized lane line information 192, and determines whether the vehicle M is based on the position information of the vehicle M. A road on which the vehicle is to be traveled may be extracted, and a region (margining line existing region) in which a road marking that divides the driving lane exists may be extracted from the road marking information of the extracted road. Further, the lane marking recognition unit 132 extracts the final lane marking existing area based on the lane marking existing area extracted from the recognized lane marking information 192 and the lane marking existing area extracted from the second map information 62. may

Further, the lane marking recognition unit 132 sets lane marking existence areas on the left and right sides of the vehicle M with respect to the traveling direction of the vehicle M (forward direction), for example. Further, the lane marking recognition unit 132 may set the size and shape of the lane marking presence area based on the position of other vehicles existing in the vicinity and the presence or absence and shape of an object OB such as a guardrail. In the example of FIG. 5, two marking line existing areas LLA and CLA are extracted on the left and right sides of the vehicle M within the recognizable range RA of the external sensor ES. Note that the marking line recognizing unit 132 may make the marking line existing areas LLA and CLA the same size and shape, or may make them different sizes and shapes. For example, when the vehicle M has traveled on a curved road just before, the lane marking recognition unit 132 determines the shape and size according to the difference in the curvature (or radius of curvature) immediately before the right and left lane markings. make different. This makes it possible to set a more optimal marking line existing area.

The lane marking recognition unit 132 extracts edges for the lane marking existing areas LLA and CLA included in the image captured by the camera 10, for example. In this case, the lane marking recognition unit 132 extracts edges based on the various edge extraction filters described above, other filters or algorithms. Note that the marking line existing areas LLA and CLA have a higher likelihood of the marking line existing than other areas of the recognizable range RA. Therefore, the lane marking recognition unit 132 extracts edges using a filter or algorithm that facilitates edge extraction, rather than the edge extraction processing by the lane marking recognition unit 132 . This makes it possible to more reliably extract edges within the marking line existing area.

The marking line recognition unit 132 also extracts, as marking line candidates, line segments of edges whose lengths are greater than or equal to the fourth threshold and included in the marking line existing areas LLA and CLA. The fourth threshold may be the first threshold and may be smaller than the first threshold. By making it smaller than the first threshold, more marking line candidates can be extracted. Note that the marking line recognition unit 132 may connect line segments of edges whose positions and directions are similar to each other.

FIG. 6 is a diagram for explaining an example of extraction of marking line candidates. In the example of FIG. 6, the edge extraction by the marking line recognition unit 132 extracts three marking line candidates C1 to C3 in the marking line existing area LLA, and extracts one marking line candidate C4 in the marking line existing area CLA. . The marking line recognition unit 132 derives marking line candidate information for each of the extracted marking line candidates C1 to C4. The marking line candidate information includes information about the state of the marking line candidate as information about the marking line candidate. For example, the marking line candidate information includes the position information of each of the marking line candidates C1 to C4 based on the position of the vehicle M and the extending direction. The marking line candidate information may also include information about the type of marking line.

Next, the lane marking recognition unit 132 uses the position information of the vehicle M to refer to the vehicle position of the recognized lane line information 192, and the lane marking state information (in other words, , the lane marking state information that was last recognized in a state where the recognition accuracy has not decreased in lane marking recognition), compares the acquired lane marking state information with the lane marking candidate information, and runs from the lane candidate. Recognize lane markings.

FIG. 7 is a diagram for explaining recognition of lane markings of a traveling lane from lane marking candidates. In the example of FIG. 7, the marking line candidates C1 to C4 in the marking line existing areas LLA and CLA, and the marking lines LLp and CLp that were finally recognized while the recognition accuracy obtained from the recognition marking line information 192 was not degraded are showing. In the example of FIG. 7, the marking lines LLp and CLp are positioned within the marking line existing area based on the position of the vehicle M in order to facilitate comparison with the marking line candidates C1 to C4 in the marking line existing area. positioned.

In the example of FIG. 7, the marking line recognition unit 132 recognizes at least one of the position, direction, and type of the marking line candidate included in the marking line candidate information, and the corresponding data (position , direction, and type) are compared, and lane markings are recognized based on the comparison result. Specifically, the marking line recognition unit 132 compares at least one of the position, direction, and type of the marking line between the marking line candidates C1 to C3 and the marking line LLp, and determines the marking line candidates C1 to C3. The degree of approximation to the lane marking LLp is extracted. For example, the marking line recognizing unit 132 increases the degree of approximation as the difference in position is smaller, the difference in direction is smaller, and the line type is closer. The marking line recognition unit 132 also compares the marking line candidate C4 and the marking line CLP, and similarly extracts the degree of approximation of the marking line candidate C4 to the marking line LLp. Then, the lane marking recognition unit 132 extracts the lane line having the highest degree of approximation among the lane line candidates C1 to C4 as the lane line of the lane L1. In the example of FIG. 7, the marking line candidates C1 and C2 are different from the marking line LLp in position, and the marking line candidate C4 is different from the marking line CLp in extension direction and line type. Therefore, the lane marking recognition unit 132 recognizes the lane marking candidate C3 among the lane marking candidates C1 to C4 as the lane marking of the driving lane (lane L1). These recognition processes can prevent lane markings from being erroneously recognized.

Note that the marking line recognition unit 132 may not recognize a marking line when the degree of approximation is less than a predetermined value. Also, the marking line recognition unit 132 may recognize marking lines in each of the marking line existing areas LLA and CLA.

The operation control unit (action plan generation unit 140, second control unit 160) executes LKAS control based on the lane markings recognized by the lane marking recognition unit 132. FIG.

As described above, in the embodiment, even if the recognition accuracy of the lane marking is lowered, the edge of the region where the likelihood of existence of the lane marking is high is extracted, and the lane marking is recognized based on the extracted edge. , the lane marking recognition accuracy and reliability can be improved. In addition, processing resources can be reduced by recognizing marking lines in a limited area.

For example, when causing the HMI 30 to output the running state of the vehicle M by driving assistance or the like, the HMI control unit 170 causes the display device of the HMI 30 to display an image related to the lane marking recognized by the lane marking recognition unit 132. good too. In this case, the HMI control unit 170 displays the lane markings in different display modes (for example, by changing the color or by blinking) depending on whether the recognition accuracy of the lane markings is degraded or not. or change the pattern). Further, the HMI control unit 170 may cause the HMI 30 to output information indicating that the lane marking recognition accuracy is degraded. Thereby, the passenger can be notified of the state of the vehicle M more accurately.

[Processing flow]
Drawing 8 is a flow chart which shows an example of a flow of processing performed by automatic operation control device 100 of an embodiment. In the example of FIG. 8, among the processes executed by the automatic driving control device 100, the lane marking recognition process will be mainly described. Moreover, the process of FIG. 8 may be repeatedly performed, for example, while automatic operation control, such as LKAS, is performed.

In the example of FIG. 8, the recognition unit 130 recognizes the surrounding situation of the vehicle M based on the detection result of the external sensor ES (step S100). Next, the lane marking recognizing unit 132 recognizes the lane marking of the vehicle M from the spatial information indicating the surrounding situation of the vehicle M (step S102).

Next, the lane marking recognition unit 132 determines whether or not the recognition accuracy of the lane marking has deteriorated (step S104). When it is determined that the lane marking recognition accuracy is lowered, the lane marking recognizing unit 132 selects a lane having a high probability of existence of the lane marking as an example of a predetermined region in the recognizable range RA by the external sensor ES. A line existing region is extracted (step S106). In the process of step S106, the lane marking recognition unit 132 may extract the lane marking presence area, for example, based on the position of the lane marking before the recognition accuracy of the lane marking is degraded. (Second map information 62) may be referenced to extract the lane marking existing area. Alternatively, the final marking line existing area may be extracted based on the marking line existing areas extracted from each of them.

Next, the marking line recognition unit 132 captures an image of an area including the marking line existing area with the camera 10, and extracts edges within the marking line existing area from the captured image (step S108). Next, the marking line recognizing unit 132 extracts marking line candidates based on the edge extraction result (step S110). A marking line is recognized based on the recognition result of the line and the degree of approximation (step S112).

After the process of step S112, or when it is determined in the process of step S104 that the lane marking recognition accuracy has not decreased, the operation control unit (action plan generation unit 140, second control unit 160) Operation control such as LKAS is executed based on the line (step S114). Thus, the processing of this flowchart ends.

[Modification]
In the above-described embodiment, the lane marking recognition unit 132 may recognize the lane markings described above even when operation control other than LKAS is being executed. For example, when ALC control is performed, the lane marking recognition unit 132 may recognize not only the lane markings that separate the driving lane, but also the lane markings that separate the adjacent lane to which the vehicle is changing lanes. Further, when extracting the lane marking presence area, the lane marking recognition unit 132, instead of (or in addition to) the above-described method, for example, as shown in FIG. The marking line existing area for the driving lane may be extracted based on the position, direction, etc. of the marking line of the lane (lane L2). Furthermore, the lane marking recognizing unit 132 may extract the lane existing area based on the positions and directions of objects OB1 and OB2 such as guardrails installed on the road RD including the driving lane L1.

Further, in the above example, edges included in the image captured by the camera 10 are mainly extracted, and lane markings are recognized based on the extraction results. Edge extraction may be performed based on the detection results (LIDAR data) of the included LIDAR 14 . In addition, when there are uneven objects such as protective fences such as guardrails, chatter bars (road studs), curbs, median strips, etc., lane markings are recognized based on the detection results of the radar device 12 and sonar. Alternatively, the marking line existing area may be extracted. As a result, the marking lines can be recognized with higher accuracy.

In addition, the lane marking recognition unit 132 recognizes the lane marking pattern (for example, an array of solid lines and dashed lines) obtained from the second map information 62 and the lane marking around the vehicle M recognized from the image captured by the camera 10. The driving lane may be recognized by comparing with the pattern. In addition, the lane marking recognition unit 132 may recognize the driving lane by recognizing lane boundaries (road boundaries) including lane markings, road shoulders, curbs, medians, guardrails, etc., in addition to lane markings. . In this recognition, the position of the vehicle M acquired from the navigation device 50 and the processing result by the INS may be taken into account.

Further, when the lane marking recognition control is lowered and the lane marking recognition unit 132 cannot recognize the lane marking, the HMI control unit 170 causes the HMI 30 to output information indicating that the lane marking cannot be recognized. Alternatively, the HMI 30 may output information prompting the occupant of the vehicle M to manually drive the vehicle M by ending the LKAS control.

According to the embodiment described above, in the automatic driving control device 100 (an example of a mobile body control device), the recognition unit 130 that recognizes the surrounding situation of the vehicle M (an example of a mobile body) based on the output of the external sensor ES and a lane marking recognition unit 132 that recognizes the lane markings that divide the area where the vehicle M passes based on the surrounding situation recognized by the recognition unit 130, and the lane marking recognition unit 132 recognizes the lane marking with accuracy When it is determined that the vehicle M is in a lowered state, a predetermined area is extracted from the surrounding situation, edges are extracted in the extracted predetermined area, and the lane markings are recognized based on the extracted result. It is possible to further improve the recognition accuracy of the demarcation line that demarcates the passage area.

Specifically, according to the embodiment, for example, based on the vicinity of the previous marking line recognition result, the vicinity of the segmentation boundary, and the vicinity of the existing position of the marking line based on the high-precision map information, a predetermined By extracting only the area and performing edge extraction only on the extracted area, the marking line can be recognized more efficiently and accurately. Further, according to the embodiment, by comparing learning-based lane marking information recognized in a state where the recognition accuracy is not degraded and lane marking information recognized by edge extraction, You can improve your likeness. According to the embodiment, lane recognition accuracy and reliability can be improved by selecting lane markings that are similar to lane markings used in the previous control, regarding the state information of lane marking candidates obtained by edge extraction. In addition, erroneous detection of marking lines can be suppressed.

The embodiment described above can be expressed as follows.
a storage device storing a program;
a hardware processor;
By the hardware processor executing the program stored in the storage device,
Recognizing the surrounding situation of the moving object based on the output of the external sensor,
Recognizing a demarcation line that demarcates an area through which the moving body passes based on the recognized surrounding situation;
extracting a predetermined area from the surrounding situation when it is determined that the lane marking recognition accuracy is lowered;
extracting edges within the extracted predetermined region;
recognizing the partition line based on the extracted result;
A mobile body control device configured to:

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

1 Vehicle system 10 Camera 12 Radar device 14 LIDAR 16 Object recognition device 20 Communication device 30 HMI 40 Vehicle sensor 50 Navigation device 60 MPU 80 Driving Operation element 100 Automatic driving control device 120 First control unit 130 Recognition unit 132 Line recognition unit 140 Action plan generation unit 160 Second control unit 162 Acquisition unit 164 Speed control unit 166 Steering control unit 170 HMI control unit 180 Storage control unit 190 Storage unit 200 Driving force output device 210 Brake device 220 Steering device

Claims (8)

a recognition unit that recognizes the surrounding situation of the moving object based on the output of the external sensor;
a lane marking recognition unit that recognizes lane markings that divide an area through which the moving object passes based on the surrounding situation recognized by the recognition unit;
When it is determined that the recognition accuracy of the marking line is degraded, the marking line recognition unit extracts a predetermined area from the surrounding situation, extracts edges in the extracted predetermined area, and extracts the edges. recognizing the parcel line based on the result;
Mobile controller. further comprising a storage control unit configured to store, in a storage unit, information related to the lane marking before the lane marking recognition unit determines that the recognition accuracy of the lane marking has decreased,
The marking line recognition unit extracts marking line candidates based on the edges extracted from the predetermined area, and the degree of similarity between information regarding the extracted marking line candidates and information regarding the marking lines stored in the storage unit. Recognizing the division lines that divide the area through which the moving body passes, based on
The moving body control device according to claim 1 . the information about the lane marking includes at least one of the position, direction, and type of the lane marking;
The moving body control device according to claim 2 . The storage unit further stores map information,
When it is determined that the recognition accuracy of the lane marking is degraded, the lane marking recognizing unit receives information about the lane marking acquired from the map information based on the position information of the moving body, or the storage unit. extracting the predetermined area based on the information about the lane marking stored before it is determined that the recognition accuracy of the lane marking is lowered;
4. The moving body control device according to claim 2 or 3. The predetermined areas are set on the left and right sides of the traveling direction of the moving object,
The moving body control device according to any one of claims 1 to 4. The lane marking recognition unit extracts edges in the surrounding situation, and if at least one of the length, reliability, and quality of the extracted edges is less than a threshold, the recognition accuracy of the lane marking has decreased. determine that the state is
The moving body control device according to any one of claims 1 to 5. the computer
Recognizing the surrounding situation of the moving object based on the output of the external sensor,
Recognizing a demarcation line that demarcates an area through which the moving body passes based on the recognized surrounding situation;
extracting a predetermined area from the surrounding situation when it is determined that the lane marking recognition accuracy is lowered;
extracting edges within the extracted predetermined region;
recognizing the partition line based on the extracted result;
Mobile control method. to the computer,
Recognize the surrounding situation of the moving object based on the output of the external sensor,
Recognize a demarcation line that demarcates an area through which the moving body passes based on the recognized surrounding situation;
extracting a predetermined area from the surrounding situation when it is determined that the lane marking recognition accuracy is lowered;
extracting edges in the extracted predetermined region;
recognizing the partition line based on the extracted result;
program.

Images