JP6416654B2 - White line detector - Google Patents

Description

  The present invention relates to a white line detection device.

  As a technique for detecting a white line of a traveling lane in which a vehicle travels, there is a vehicle image processing apparatus described in Patent Document 1. The apparatus described in Patent Document 1 alternately obtains an image suitable for a three-dimensional object detection process and an image suitable for a white line detection process using a single camera. Then, the image recognition processing unit of the vehicle image processing apparatus detects a white line by executing a white line detection process using an image suitable for the white line detection process.

JP 2006-325094 A

  The image suitable for the white line detection process includes a three-dimensional object different from the white line. The three-dimensional object reflected in the image can become noise in the process of detecting a white line from the image. Therefore, it is conceivable to detect a three-dimensional object reflected in an image suitable for white line detection processing and perform noise processing using information of the three-dimensional object. However, an image suitable for white line detection processing is captured under conditions for clearly capturing the white line. On the other hand, since a three-dimensional object reflected in an image suitable for white line detection processing may not be clear, it is difficult to accurately detect the three-dimensional object and perform noise processing. Therefore, it is difficult to improve the accuracy of detecting white lines.

  Accordingly, an object of the present invention is to improve white line detection accuracy.

  One embodiment of the present invention is a white line detection device that detects a white line of a traveling lane in which a vehicle travels using an image captured by an in-vehicle camera mounted on the vehicle, and the in-vehicle camera captures an image at a first timing. A three-dimensional object image and a white line image captured by the in-vehicle camera at a second timing that is not simultaneously with the first timing, and a stereo process using the three-dimensional object image. A three-dimensional object extraction unit that obtains information on a three-dimensional object in the three-dimensional object image; a candidate point extraction unit that obtains information on candidate points that form edges in the white line image by edge processing using the white line image; A region specifying unit that obtains information of a region occupied by a three-dimensional object in a white line image by using the information of the three-dimensional object, the vehicle speed and direction of the vehicle, and the time difference between the first timing and the second timing; Point information And by utilizing the information of the region occupied by the three-dimensional object, comprising a detection unit for detecting a candidate point that is not included in the area where the solid object is occupied as points constituting the edge of the white line, the.

  According to the present invention, white line detection accuracy can be improved.

FIG. 1 is a functional block diagram of an image processing system including a white line detection device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating the imaging timing of the solid object image and the white line image. FIG. 3 is a flowchart showing the operation of the white line detection device. FIG. 4 is an example of a solid object image and a white line image for explaining the operation of the white line detection device. FIG. 5 is a functional block diagram of an image processing system including a white line detection device according to the second configuration. FIG. 6A is a graph showing the relationship between the positional difference between the curb edge and the white line edge and the likelihood, and FIG. 6B is a diagram for explaining the positional relationship between the curb edge and the white line edge. FIG. 7 is a functional block diagram of an image processing system including a white line detection device according to the third configuration.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a functional block diagram of an image processing system 100 including a white line detection device according to an embodiment of the present invention. The image processing system 100 includes a white line detection device 1, an in-vehicle camera 2, a vehicle sensor 3, a data recording unit 4, and a runway parameter estimation unit 5. The image processing system 100 is a system mounted on the host vehicle, and detects white lines indicating both sides of the traveling lane from an image captured by the in-vehicle camera 2 mounted on the host vehicle. The detected white line is used for parameter calculation of the traveling lane.

  The in-vehicle camera 2 includes two cameras 2a and 2b, which are imaging means, and a camera control unit 2c. The imaging condition of the in-vehicle camera 2 is controlled by a control signal output from the white line detection device 1 to the camera control unit 2c. The two cameras 2a and 2b are spaced apart from each other in the width direction of the host vehicle. The camera control unit 2c has a control parameter for acquiring a white line image and a control parameter for acquiring a three-dimensional object image, and changes the control parameter for each imaging frame. FIG. 2 is a diagram illustrating the imaging timing of the solid object image and the white line image. As shown in FIG. 2, for example, the camera control unit 2c repeatedly changes a frame F1 for acquiring a three-dimensional object image and a frame F2 for acquiring a white line image at intervals of 50 milliseconds. The vehicle sensor 3 acquires data related to traveling of the host vehicle and outputs the data to the white line detection device 1. The vehicle sensor 3 includes a sensor group that acquires numerical values indicating the motion state of the host vehicle, such as a vehicle speed sensor and a yaw rate sensor. The data recording unit 4 is configured to be referenced from the white line detection device 1. The data recording unit 4 records camera parameters such as the positions where the cameras 2a and 2b are arranged, the positional relationship between the cameras 2a and 2b, and the angles of view of the cameras 2a and 2b, and various setting values related to white line detection. Is done. The travel path parameter estimation unit 5 estimates various parameters indicating the shape of the road on which the host vehicle travels. The parameters include road curvature, curvature change rate, yaw angle (direction) of the vehicle with respect to the white line, and lateral position of the vehicle with respect to the white line. The runway parameter estimation unit 5 uses the white line information input from the white line detection device 1 as described later to estimate the parameters by Kalman filter processing.

  The white line detection device 1 is realized by executing a program on an ECU (Electric Control Unit). The ECU is a so-called computer, and includes a central processing unit (CPU), a ROM in which programs and various data are recorded, a RAM in which data is temporarily recorded, an input / output device, and the like as one unit. The white line detection device 1 includes an image acquisition unit 6, a three-dimensional object extraction unit 7, a candidate point extraction unit 8, a region specification unit 9, and a detection unit 11 as functional components. In the white line detection device 1, the image acquisition unit 6 acquires an image from the in-vehicle camera 2. The image acquisition unit 6 acquires a solid object image and a white line image. The three-dimensional object extraction unit 7 and the candidate point extraction unit 8 respectively extract information about the three-dimensional object and information about the candidate points constituting the white line edge from the image acquired by the image acquisition unit 6. The area specifying unit 9 specifies an area occupied by the three-dimensional object in the white line image. And the detection part 11 determines whether a candidate point is contained in the area | region which a solid object occupies, and detects the candidate point which is not contained in the area | region which a solid object occupies as information which shows a white line.

  The image acquisition unit 6 acquires image information from the in-vehicle camera 2 and outputs it to the three-dimensional object extraction unit 7 and the candidate point extraction unit 8. The image information includes two images captured by the two cameras 2a and 2b. The image acquisition unit 6 acquires an image for a three-dimensional object from the in-vehicle camera 2 and outputs it to the three-dimensional object extraction unit 7, and acquires an image for the white line from the in-vehicle camera 2 and outputs it to the candidate point extraction unit 8. repeat. The white line image is an image for detecting a white line, and the brightness (luminance value) of the white line area in the image is optimal. The three-dimensional object image is an image for detecting a three-dimensional object, and the brightness (luminance value) of an area that is a three-dimensional object in the image is optimal. The camera control unit 2c controls the exposure time of the camera so that the brightness of the white line or the three-dimensional object is optimized.

  The three-dimensional object extraction unit 7 uses the three-dimensional object image to obtain information about the three-dimensional object in the three-dimensional object image (hereinafter referred to as “three-dimensional object information”), and sends the acquired three-dimensional object information to the region specifying unit 9. Output. As a three-dimensional object, for example, another vehicle that travels in the same direction as the own vehicle in the traveling lane adjacent to the own vehicle, an oncoming vehicle that travels in the opposite lane, a curb provided on the outer edge of the traveling lane, There are road signs, traffic lights and utility poles. The three-dimensional object extraction unit 7 acquires three-dimensional information of the three-dimensional object included in the image by using two images for three-dimensional objects with different parallax imaged at the same timing by the camera 2a and the camera 2b. The three-dimensional object extraction unit 7 acquires the three-dimensional information of all the three-dimensional objects having extractable areas. The three-dimensional information includes three-dimensional position data and three-dimensional shape data of the three-dimensional object included in the three-dimensional object image. The three-dimensional shape data does not indicate the detailed shape of the three-dimensional object, but is information indicating the height, width, and depth when the three-dimensional object is approximated as a rectangular parallelepiped or a cylindrical shape, for example. The three-dimensional position data is information indicating, for example, the position of the center of gravity (center point) of the shape indicated by the three-dimensional shape data as the representative position of the three-dimensional object. In the following description, as a coordinate system for defining three-dimensional information, the own vehicle is the origin, the width direction of the own vehicle is the X-axis direction, the vertical direction is the Y-axis direction, and orthogonal to the X-axis and the Y-axis. The direction to perform is the Z-axis direction.

  The three-dimensional object extraction unit 7 cuts out a small area from the reference image captured by the camera 2a (one of the three-dimensional object images captured at the same timing) and captured by the camera 2b corresponding to the small area. A small area of the reference image (the other of the three-dimensional object images) is searched. For the search, for example, similarity of luminance patterns such as SAD (Sum of Absolute Difference) which is an example of pattern matching processing can be used. The three-dimensional object extraction unit 7 uses the search result to obtain a shift amount (shift pixel number) between the small area of the reference image and the small area of the reference image corresponding to the small area. The three-dimensional object extraction unit 7 calculates the parallax using the number of shifted pixels, the focal length of each camera obtained by referring to the data recording unit 4, and the relative position between the cameras. Then, the three-dimensional object extraction unit 7 groups parallaxes, tracks these groups, and calculates a three-dimensional position related to the three-dimensional object.

  The three-dimensional object information also includes information on the absolute speed of the three-dimensional object. The absolute speed of the three-dimensional object is determined by a known method using the latest three-dimensional object image, the three-dimensional object image acquired in the past by one or a predetermined number of frames, and the vehicle speed and yaw rate obtained from the vehicle sensor 3. Calculated. Further, the output of a millimeter wave sensor or the like included in the vehicle sensor can be used.

  The candidate point extraction unit 8 uses the white line image to obtain information on candidate points constituting the white line edge (hereinafter referred to as “white line candidate point information”), and obtains the acquired white line candidate point information and the white line image. The data is output to the area specifying unit 9. Therefore, the white line candidate point information acquired in the candidate point extraction unit 8 is not the information indicating the white line itself, but is constituted by points extracted from the white line candidate point information. The candidate point extraction unit 8 extracts candidate points by edge processing. In the edge processing, a well-known edge processing method using the luminance value of the pixels constituting the white line image is used. The candidate point extraction unit 8 may obtain a three-dimensional position of the candidate point by a stereo matching method based on the two white line images captured at the same timing. Note that the candidate point extraction unit 8 may extract all candidate points constituting the edge, not limited to the candidate points constituting the white line edge. That is, the white line candidate points may include candidate points for edges of solid objects other than the white lines.

  The candidate point extraction unit 8 scans from the left side to the right side of the white line image using the luminance value of the pixels constituting the white line image as a determination value. Then, the candidate point extraction unit 8 acquires a portion that has changed from a region smaller than the threshold value to a region that is larger than the threshold value as a point indicating an upstream edge, and indicates a portion that has changed from a region that is larger than the threshold value to a region that is smaller than the threshold value. Get as a point. A region between the rising edge point and the falling edge point is a white line. The candidate point extraction unit 8 obtains the downlink edge candidate information and the uplink edge candidate information from each of the two white line images, and then obtains three-dimensional position information using an image processing method such as a stereo matching method. Therefore, the candidate point information acquired by the candidate point extraction unit 8 includes the uplink edge candidate information and the downlink edge candidate information in which the three-dimensional position is specified.

  The candidate point extraction unit 8 may perform a fitting process on a plurality of candidate points. The candidate point extraction unit 8 handles a set including a plurality of upstream edge points as an upstream edge candidate point group, and handles a set including a plurality of downstream edge points as a downstream edge candidate point group. Then, for example, processing based on temporal continuity is performed on each candidate point group. In this process, noise is removed from the candidate points by determining whether or not the candidate point to be processed is close to another candidate point detected in the past, and a more probable candidate point is determined. select. There is also a process based on geometric continuity. In this process, the candidate point to be processed is removed from the candidate point by determining whether or not the candidate point is closer to the virtual line segment expected from the runway pattern, etc. Select. According to these processes, the accuracy of information output from the candidate point extraction unit 8 can be improved.

  The area specifying unit 9 obtains area information and outputs the acquired area information to the detecting unit 11. The area information is information indicating an area occupied by the three-dimensional object in the white line image. The three-dimensional object information is three-dimensional information related to a three-dimensional object. The white line candidate point information is three-dimensional information indicating candidate points of information indicating a white line included in the solid object image. Therefore, these can be handled in common in a virtual space by selecting a common coordinate system.

  Incidentally, the three-dimensional object information is extracted from the three-dimensional object image acquired at the first timing. Further, the white line candidate point information is extracted from the white line image acquired at the second timing. Then, the relationship between the three-dimensional object and the white line may not be accurate if the three-dimensional object information and the white line candidate point information are simply combined. The first timing and the second timing are not the same.

  Therefore, the area specifying unit 9 estimates the position of the three-dimensional object at the second timing using the speed acquired from the vehicle sensor 3, the yaw rate, and the time difference between the first and second timings. And the area | region specific part 9 projects the solid object shown by the coordinate system which prescribes | regulates white line candidate point information on XY plane, and obtains the area which occupies in XY plane. By this processing, area information indicating the area occupied by the three-dimensional object in the white line image is obtained.

  The area specifying unit 9 estimates the position of the three-dimensional object on the horizontal plane (XZ plane). This estimation is performed for each direction of the X axis and the Z axis forming a horizontal plane (XZ plane). Specifically, the region specifying unit 9 calculates the amount of movement of the three-dimensional object generated by the time difference by multiplying the absolute speed of the three-dimensional object calculated by the three-dimensional object extraction unit 7 by the time difference. The movement amount is applied to the position acquired from the three-dimensional object image to obtain an estimated position in which the shift caused by the movement of the three-dimensional object is corrected. Subsequently, the area specifying unit 9 calculates the amount of movement of the host vehicle caused by the time difference. For the movement of the host vehicle, the vehicle speed and yaw rate obtained from the vehicle sensor 3 are used. The area specifying unit 9 applies the movement amount to the estimated position described above, and obtains an estimated position in which the displacement of the position caused by the movement of the three-dimensional object and the movement of the host vehicle is corrected. Subsequently, the region specifying unit 9 restores the three-dimensional shape of the three-dimensional object using the three-dimensional shape data included in the three-dimensional object information at the estimated position of the three-dimensional object. Subsequently, the region specifying unit 9 obtains information on the region occupied by the three-dimensional object in the white line image by calculating the planar shape of the three-dimensional object projected on the XY plane.

  Note that the area specifying unit 9 may include not only the area occupied by the three-dimensional object at the second timing when the white line image is acquired, but also the part where the three-dimensional object previously existed in the area information. Specifically, the area information may include an area occupied by the three-dimensional object on the white line image when the three-dimensional object moves from the first timing to the second timing. When the three-dimensional object is a moving body such as an oncoming vehicle, the region information may include a region obtained by projecting the region occupied by the three-dimensional object on the XY plane on the horizontal plane (XZ plane). On the horizontal plane, a region occupied by a three-dimensional object such as another vehicle or a utility pole is mainly a region in the lane or outside the lane, and it can be considered that there is generally a high possibility that no white line exists. Therefore, by adding a region where there is a high possibility that no white line exists to the region information, it is possible to reduce the possibility of erroneously detecting road dirt or the like as a white line.

  The detection unit 11 obtains white line information and outputs the white line information to the travel path parameter estimation unit 5. White line information is information that is likely to indicate a white line selected from white line candidate point information. Candidate information indicating candidates for white line edges on the XY plane and area information indicating areas occupied by the three-dimensional object on the same XY plane are input to the detection unit 11 from the area specifying unit 9. Here, it is considered that the white line is not reflected in the area occupied by the three-dimensional object. Then, it can be assumed that there are no white line candidate points in the region occupied by the three-dimensional object. Therefore, when a white line candidate point exists in the area occupied by the three-dimensional object, it is considered that there is a high possibility that it is noise.

  Therefore, the detection unit 11 determines whether white line candidate points are included in the area occupied by the three-dimensional object, using the area information and the white line candidate point information. When the detection unit 11 determines that the white line candidate point is included in the region occupied by the three-dimensional object, the detection unit 11 determines that the white line candidate point is highly likely to be noise, and does not adopt the white line information. On the other hand, when the detection unit 11 determines that the white line candidate point is not included in the region occupied by the three-dimensional object, the detection unit 11 determines that the white line candidate point is highly likely to be true information indicating the white line, and employs the white line candidate point as white line information. To do. That is, the detection unit 11 detects the white line candidate point as a point constituting the edge of the white line. The detection unit 11 performs the above determination on all three-dimensional objects extracted by the three-dimensional object extraction unit 7. The detection unit 11 detects a point constituting a white line edge from the white line candidate points, and detects a white line using the point constituting the white line edge.

  Next, the operation of the white line detection device will be described with reference to FIGS.

  The white line detection device 1 uses the in-vehicle camera 2 and the image acquisition unit 6 to acquire the three-dimensional object image at the first timing (step S1). The three-dimensional object image G1 shown in FIG. 4A is an image captured when the host vehicle is traveling in a traveling lane that is one lane on one side. The solid object image G1 includes white lines Wa and Wb and a vehicle center line C. The three-dimensional object image G1 includes an oncoming vehicle T (three-dimensional object) traveling in the oncoming lane. The oncoming vehicle T is clearly shown in the three-dimensional object image G1.

  Subsequently, the white line detection device 1 uses the in-vehicle camera 2 and the image acquisition unit 6 to acquire a white line image at the second timing (step S2). The white line image G2 shown in FIG. 4B is an image captured when the host vehicle is traveling in a traveling lane that is one lane on one side. The white line image G2 includes white lines Wa and Wb and a vehicle center line C. The white line image G2 includes an oncoming vehicle T traveling in the oncoming lane. In the white line image G2, the white lines Wa and Wb and the vehicle center line C are clearly shown. The white line image G2 is captured after the three-dimensional object image G1. Therefore, the position of the oncoming vehicle T is different from the position of the oncoming vehicle T in the three-dimensional object image G1.

  Subsequently, the white line detection device 1 acquires travel data using the vehicle sensor 3 (step S3). The travel data includes the vehicle speed of the host vehicle and the yaw rate of the host vehicle. And the white line detection apparatus 1 acquires setting data using the data recording part 4 (process S4). The setting data includes a camera parameter of the in-vehicle camera 2, an initial value of likelihood described later, a change value of likelihood described later, and the like.

  The white line detection device 1 uses the three-dimensional object extraction unit 7 to extract three-dimensional object information from the three-dimensional object image (step S5). The number of three-dimensional objects included in the three-dimensional object information is at least one. The total number of three-dimensional objects included in the three-dimensional object information is N (N is an integer of 1 or more). When N is 2 or more, for example, the first three-dimensional object, the second three-dimensional object, etc. are attached with numbers. For example, as illustrated in FIG. 4C, the three-dimensional object extraction unit 7 uses the oncoming vehicle T as the nth three-dimensional object, and extracts data regarding the three-dimensional position and the three-dimensional shape of the oncoming vehicle T.

  The white line detection device 1 acquires white line candidate point information using the candidate point extraction unit 8 (step S6). The number of white lines included in the candidate point information is at least one, and may be two or three or more. For example, as illustrated in FIG. 4D, the candidate point extraction unit 8 extracts data relating to the three-dimensional positions and shapes of the white lines Wa and Wb and the vehicle center line C. At this time, it is assumed that a part of the oncoming vehicle T is erroneously extracted as a part of the white line Wa (line segment W1). In step S6, initial values of likelihood are set for all candidate points. The initial value of likelihood is 0.5, for example. The likelihood is an evaluation value representing the probability that the candidate point is a point that actually constitutes a white line edge.

  The white line detection device 1 calculates a time difference (step S7). This step S7 is executed by the region specifying unit 9. The area specifying unit 9 calculates a difference between time data included in the three-dimensional object image G1 and the white line image G2 as a time difference. The time difference is, for example, 50 milliseconds.

  The white line detection device 1 estimates the position of the oncoming vehicle T (step S8). This step S8 is executed by the region specifying unit 9. In this process S8, the coordinate system that is the reference for the three-dimensional object information and the coordinate system that is the reference for the candidate point information are common. If the coordinate system that is the reference for the three-dimensional object information is different from the coordinate system that is the reference for the candidate point information, coordinate conversion may be performed so that either one is based on the other coordinate system. Subsequently, as shown in FIG. 4E, the white line detection device 1 calculates a region A occupied by the oncoming vehicle T in the white line image G2 (step S9). This step S9 is executed by the region specifying unit 9.

  The white line detection device 1 acquires white line information. This step is executed by the detection unit 11. Specifically, the detection unit 11 determines whether or not the likelihood at the white line candidate point to be processed is equal to the initial value (0.5) (step S10). According to this step S10, it is possible to omit processing of white line candidate points that have already been determined to be included in the region occupied by the three-dimensional object in the iterative processing. Subsequently, processing is performed on white line candidate point information constituting the white line Wa. It is determined whether candidate points are included in the area A occupied by the oncoming vehicle T (step S11). If the candidate point is included (step S11: YES), the likelihood of the candidate point is changed to a predetermined value (for example, 0.1) (step S12). For example, as shown in FIG. 4 (e), since the candidate point P1 constituting the line segment W1 is included in the area A occupied by the oncoming vehicle T, the likelihood is changed from the initial value 0.5 to the changed value. 0.1. Then, the process proceeds to the next step S13. If the candidate point is not included (step S11: NO), the process proceeds to step S13 without changing the likelihood of the candidate point. For example, the candidate point P2 constituting the line segment Wa1 is not included in the area A occupied by the oncoming vehicle T, and the likelihood is left at the initial value of 0.5.

  Subsequently, it is determined whether or not the processing has been completed for all candidate points (step S13). If the process has not been completed (step S13: NO), steps S10 to S12 are performed again. When the process is completed (step S13: YES), the process proceeds to the next step S14. By executing the steps S10 to S13, the process for one three-dimensional object is completed.

  Subsequently, it is determined whether or not processing has been completed for all three-dimensional objects (step S14). When the processing is not completed (step S14: NO), the counter (n) is added (step S15), and the above-described steps S10 to S14 are performed on the next (n + 1) th three-dimensional object. . When the process is completed (step S14: YES), the process proceeds to step S16.

  Subsequently, white line information is detected (step S16). In the candidate information after performing the above-described Steps S10 to S14, weighting by likelihood is performed for each candidate point. For example, as shown in FIG. 4F, the candidate points constituting the line segment W1 included in the region A in the white line Wa have a likelihood of 0.1. On the other hand, in the white line Wa, the candidate points constituting the line segments Wa1 and Wa2 not included in the region A have a likelihood of 0.5. Therefore, the detection unit 11 selects a candidate point having a likelihood equal to or greater than a predetermined value (0.5 or more) as a point constituting an edge of a white line. A white line is detected by performing the above steps S1 to S16.

  In the above-described operation, the operation of extracting the three-dimensional object information (step S5) and the operation of acquiring the white line image (step S2) may be performed in parallel. Similarly, the operation of extracting candidate point information (step S6) and the operation of acquiring the three-dimensional object image (step S1) may be performed in parallel. In the above-described operation, the combination of the three-dimensional object image and the white line image used for the processing may be any combination as long as the time difference between the three-dimensional object image and the white line image used for the processing is acquired. Can do.

  As described above, in the white line detection device 1, the image acquisition unit 6 acquires the three-dimensional object image G1 at the first timing, and acquires the white line image G2 at the second timing not simultaneously with the first timing. Yes. Since the timing for acquiring the three-dimensional object image G1 and the white line image G2 is shifted in time, the vehicle-mounted camera 2 can be set to a condition suitable for capturing each image. Therefore, the three-dimensional object extraction unit 7 and the candidate point extraction unit 8 can acquire the three-dimensional object information and the white line candidate point information with high accuracy. This white line candidate point information may include a noise component (for example, a line segment W1 in FIG. 4D) due to the presence of the oncoming vehicle T that is a three-dimensional object. Therefore, the region specifying unit 9 acquires region information indicating the region A occupied by the oncoming vehicle T in the white line image G2 using the three-dimensional object information. In the acquisition of the area information, the temporal position shift between the images is corrected using the time difference between the vehicle speed and direction of the host vehicle and the first and second timings. Accordingly, it is possible to accurately specify the region A occupied by the oncoming vehicle T in the white line image G2. Since the candidate point P1 included in the area A occupied by the oncoming vehicle T is highly likely to be noise, the detection unit 11 detects the candidate point P2 and the like not included in the area A occupied by the oncoming vehicle T as white line information. . Then, since the detected white line information does not include the candidate point P2 or the like that may be noise that reduces the detection accuracy, the detection accuracy of the white line can be improved.

  An image processing system 100A according to the second embodiment will be described. The image processing system 100A pays attention to the curb installed on the outside of the traveling lane among the three-dimensional objects, and suppresses the decrease in the accuracy of white line detection due to the presence of the curb. Further, in the correction relating to the time difference between the imaging timing of the three-dimensional object image and the imaging timing of the white line image, the curb position is estimated by the flow detection process based on the image, not the estimation process based on the traveling data of the own vehicle.

  As shown in FIG. 5, the image processing system 100 </ b> A includes a white line detection device 1 </ b> A, an in-vehicle camera 2, a data recording unit 4, and a runway parameter estimation unit 5. The same elements as those in the first embodiment are denoted by the same reference numerals, and redundant description is omitted.

  The white line detection device 1 </ b> A includes an image acquisition unit 6, a three-dimensional object extraction unit 7, a candidate point extraction unit 8, a position estimation unit 12, and a detection unit 13. The image acquisition unit 6 has the same configuration as the image acquisition unit 6 of the first embodiment and operates in the same manner.

  Similarly to the above-described embodiment, the three-dimensional object extraction unit 7 acquires curb information (three-dimensional object information) related to the curb in the image using the three-dimensional image, and sends the curb information to the position estimation unit 12. Output. The curb information is information indicating a three-dimensional position and a three-dimensional shape of the curb. For example, stereo matching processing can be used to acquire this information. The parallax between the two images included in the three-dimensional object image is calculated by the stereo matching process. Subsequently, a boundary between the road surface and the curb is detected as a curb edge by using a step of the curb (height information included in information indicating a change in parallax or a three-dimensional shape). A pattern matching process can also be used to detect the curb edge. According to the pattern matching process, it is possible to suppress a decrease in the parallax accuracy of the curb at a position far from the host vehicle. In the above-described embodiment, the three-dimensional object extraction unit 7 may not perform until the detection of the curb edge.

  Similar to the above-described embodiment, the candidate point extraction unit 8 uses the white line image to acquire candidate point information regarding the white line in the image, and outputs the candidate point information to the detection unit 11. Similar to the candidate point extraction unit 8, the candidate point extraction unit 8 extracts a plurality of candidate points from the white line image by edge processing.

  The position estimation unit 12 estimates the position of the curb at the timing of the second timing from the position of the curb at the first timing using the time difference between the image for the three-dimensional object and the image for the white line. Then, the position estimation unit 12 outputs information on the estimated curb position (estimated curb information) to the detection unit 13. The imaging timing is shifted between the solid object image from which the curb information is extracted and the white line image from which the candidate point information is extracted. That is, the shutter timing in the vehicle-mounted camera 2 is different (see FIG. 2). For example, when the solid object image is acquired first and then the white line image is acquired, the position of the curb in the solid object image appears to be delayed from the position of the curb in the white line image. Since the time when the three-dimensional object image is acquired and the time when the white line image is acquired are known, the position estimating unit 12 calculates a time difference (delay time) from these times. Subsequently, the position estimation unit 12 uses the latest 3D object image and the 3D object image (for example, 100 milliseconds before) for the curb optical flow amount (pixel movement amount per unit time). ) Is calculated. And the position estimation part 12 estimates the position in the time which acquired the image for white lines from the positional information on the curb acquired from the image for solid objects using the said delay time and the amount of optical flows.

  In addition, the position estimation part 12 converts the position of the curb estimated to the expression using the coordinate system which prescribes candidate point information as needed. Further, the position estimation unit 12 may perform parameter estimation using the estimated position information of the curb as necessary. In the parameter estimation process, edge information of the curb is calculated from the estimated curb information by an approximation process such as a least square method using a model such as a straight line model or a clothoid model. This parameter estimation process is effective when the curb steps are not accurately calculated when the parallax between the two images acquired by the stereo camera is small, and the curb edge shape indicated by the curb information is not continuous. . The parameter estimation process is effective when a region corresponding to a predetermined region of the standard image is not found from the reference image in the stereo matching process.

  The detection unit 13 performs weighting processing on the white line candidate point information using the estimated curb information, and detects white line information from the weighted white line candidate point information. The weighting process of white line candidate point information using estimated curb information will be described. In this weighting process, the positional difference along the X-axis direction from the curb position indicated in the estimated curb information to the white line edge candidate indicated in the white line candidate point information is used as a parameter. FIG. 6A shows the relationship between the positional difference and the probability (likelihood) that is a white line. The horizontal axis indicates the position difference, and the vertical axis indicates the likelihood. The position difference and the likelihood are represented by a predetermined function, and are assumed to be a linear function here. As shown in the graph, since the white line edge candidate is further away from the curb as the positional difference is larger, it is defined that the white line edge candidate is more likely to be a white line (increase the likelihood). On the other hand, since the curbstone and the white line edge candidate are closer to each other as the positional difference is smaller, the white line edge candidate is defined to have a lower probability of being a white line (decrease the likelihood). Note that the graph shown in FIG. 6A is appropriately set so that optimum control is performed.

  FIG. 6B is a plan view of the white line edge candidate Wc indicated by the white line candidate line information and the curb edge E indicated by the estimated curb information viewed from the Y-axis direction. The detection unit 13 divides the white line edge candidate Wc and the curb edge E into three regions B1, B2, and B3 along the depth direction (Z-axis direction). For example, it is assumed that the lengths of the regions B1, B2, and B3 are 5 meters. Note that the number of divisions along the depth direction may be three or more. When the number of divisions is large, an accurate positional difference can be obtained in a curved traveling lane. Subsequently, in the first region B1, a position difference da on the near side in the lateral direction (X-axis direction) between the curb edge E and the white line edge candidate Wc is calculated. Subsequently, the position difference db on the back side is calculated in the first region B1. The positional difference d1 between the curb edge E and the white line edge candidate Wc in the first area B1 is defined as (d1 = (da + db) / 2). Subsequently, the positional difference d is converted into the likelihood L1 using the graph shown in FIG. The processing for calculating the positional difference d and the processing for converting the positional difference into likelihood are also performed for the second region B2 and the third region B3. As a result, the likelihood of the second region B2 is L2, and the likelihood of the third region B3 is L3.

  Next, the detection unit 13 detects white line information. The detection unit 11 selects, as a white line, a line segment of a region having a weight (likelihood) equal to or greater than a threshold for the first to third regions B1, B2, and B3 that are weighted. For example, the detection unit 13 employs a region whose likelihood is Ls or more as white line information. Then, as shown in FIG. 6A, white line candidate information indicating the regions B1 and B2 is adopted as the white line information.

  According to the white line detection device 1A, similarly to the white line detection device 1, it is possible to reduce the influence of noise caused by the presence of the curb and improve white line detection accuracy.

  A white line detection device 1B in the third embodiment will be described. The white line detection device 1B according to the third embodiment is different from the white line detection device 1 according to the first embodiment in that white line detection device 1B according to the first embodiment is different from the white line detection device 1 according to the first embodiment in that the white line detection accuracy caused by the presence of the curb is suppressed. Common to 1A. Further, the correction regarding the time difference between the imaging timings in the three-dimensional object image and the white line image is the same as the white line detection device 1 of the first embodiment in that the traveling data of the host vehicle is used, and the white line detection of the second configuration is performed. Different from the device 1A.

  As shown in FIG. 7, the image processing system 100 </ b> B includes a white line detection device 1 </ b> B, an in-vehicle camera 2, a vehicle sensor 3, a data recording unit 4, and a runway parameter estimation unit 5. The white line detection device 1B includes an image acquisition unit 6, a three-dimensional object extraction unit 7, a candidate point extraction unit 8, a position estimation unit 12A, and a detection unit 13.

  The position estimation unit 12A estimates the position at the timing of the second timing from the position of the curb at the first timing, using the time difference between the solid object image and the white line image. In this estimation, the position estimation unit 12A uses a method using the vehicle speed and yaw rate acquired from the vehicle sensor 3 as in the first embodiment, instead of the method using the optical flow. Specifically, the position estimation unit 12A calculates the amount of movement of the host vehicle caused by the time difference. The estimated position of the curb is obtained by applying the amount of movement to the position of the curb. In addition, in the area | region specific part 9 of 1st Embodiment, the movement amount resulting from the motion of a solid object was calculated. However, because the curb is a fixed object, its own movement is zero. Therefore, the position estimation unit 12A does not need processing related to the movement of the three-dimensional object itself.

  According to the white line detection device 1B, similarly to the white line detection device 1, it is possible to reduce the influence of noise caused by the presence of the curb and improve white line detection accuracy.

DESCRIPTION OF SYMBOLS 1,1A, 1B ... White line detection apparatus, 2 ... Car-mounted camera, 2a, 2b ... Camera, 2c ... Camera control part, 3 ... Vehicle sensor, 4 ... Data recording part, 5 ... Runway parameter estimation part, 6 ... Image acquisition part 7 ... Solid object extraction unit, 8 ... Candidate point extraction unit, 9 ... Region specifying unit, 11, 13 ... Detection unit, 12 ... Position estimation unit, 12A ... Position estimation unit, 100, 100A, 100B ... Image processing system, A ... Area occupied by oncoming vehicle, C ... Vehicle center line, E ... Curb edge, G1 ... Solid object image, G2 ... White line image, L1, L2, L3 ... Likelihood, P1, P2 ... Candidate point, T ... Oncoming vehicle, Wa, Wb ... white line, Wc ... white line edge candidate.

Claims (1)

A white line detection device that detects a white line of a traveling lane in which the host vehicle travels using an image captured by an in-vehicle camera mounted on the host vehicle,
An image acquisition unit that acquires a three-dimensional object image captured by the vehicle-mounted camera at a first timing and a white line image captured by the vehicle-mounted camera at a second timing that is not simultaneous with the first timing;
A three-dimensional object extraction unit that obtains information of a three-dimensional object in the three-dimensional object image by stereo processing using the three-dimensional object image;
A candidate point extraction unit that obtains information of candidate points that constitute edges in the white line image by edge processing using the white line image;
Using the information of the three-dimensional object, the vehicle speed and direction of the host vehicle, and the time difference between the first timing and the second timing, information on the area occupied by the three-dimensional object in the white line image is obtained. An area identification unit to obtain;
A detection unit that detects the candidate points that are not included in the region occupied by the three-dimensional object as points constituting the edge of the white line using the information of the candidate point and the information of the region occupied by the three-dimensional object; A white line detection device provided.

Images