JP7446830B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method that estimate parking spaces and driving lanes provided on a road surface based on image signals output from an imaging device that images the road surface around a vehicle.

2. Description of the Related Art Recently, vehicle parking support devices have been put into practical use that automatically detect a target parking space and park the vehicle automatically when the vehicle is parked in a predetermined parking lot. When detecting a parking space, parking lot lines are detected based on an image taken around the vehicle using a camera mounted on the vehicle, and an area surrounded by the parking lot lines is detected as a parking space. In addition, a driving support device has also been put into practical use that automatically detects the lane boundary lines that separate driving lanes based on images taken of the surrounding area of the vehicle while the vehicle is driving, and allows the vehicle to drive automatically. has been done.

Technologies for detecting white lines such as parking lot lines and lane boundary lines include white line detection devices and white line detection methods that detect edges from images taken around the vehicle and detect white lines based on the detected edges. It has been disclosed (for example, see Patent Documents 1 and 2).

Japanese Patent Application Publication No. 8-167023 Japanese Patent Application Publication No. 2007-179386

However, in the conventional technology described above, the edge enhancement function of the camera emphasizes edges such as the boundary of a long shadow on the road surface, the boundary between sunlight and shade, and white lines also apply to areas other than parking lot lines and lane boundaries. It was sometimes detected as.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image processing device and an image processing method that can detect boundary lines dividing a road surface, such as parking lot lines and lane boundary lines, with high accuracy.

In order to achieve the above object, an image processing device of the present invention is an image processing device that detects a boundary line that partitions a predetermined area of a road surface. An image based on an image signal output from an imaging device that images the road surface around the vehicle is scanned in a predetermined direction, and pixels whose brightness included in the image signal changes more than a threshold value are detected. an edge detection unit that detects as an edge a portion where a pixel arrangement is longer than a predetermined length; a luminance of a pixel located on a predetermined direction side of a first edge detected by the edge detection unit; and the first edge. and the luminance of a pixel on the side opposite to the predetermined direction side of the first edge of a second edge different from The present invention is characterized by comprising a comparison unit that determines that an edge and the second edge are a pair of edges forming a boundary line.

The image processing device of the present invention configured as described above detects pixels whose brightness changes more than a threshold value from a captured image, and defines a portion where the detected pixel arrangement has a length longer than a predetermined value as an edge. Detected as. Then, the difference in brightness between a pixel on the side of a predetermined direction of the detected first edge and a pixel on the side of the second edge in the opposite direction to the side of the predetermined direction of the first edge is a predetermined value. In the following cases, the first edge and the second edge are determined to be a pair of edges forming a boundary line. This makes it possible to prevent the shadows of trees and poles extending on the road surface from being recognized as boundary lines.

This makes it possible to detect boundary lines that partition the road surface, such as parking lot lines and lane boundary lines, with high precision. As a result, it becomes possible to detect parking spaces provided on the road surface of a parking lot or the like and to detect driving lanes divided by lane boundary lines with high precision.

1 is a block diagram showing a schematic configuration of a parking assistance device to which an image processing device according to an embodiment of the present invention is applied. It is a figure showing an example of the arrangement position of the imaging device of the parking assistance device which is an embodiment. 1 is a functional block diagram showing a schematic configuration of an image processing device according to an embodiment. FIG. 3 is a flowchart for explaining an example of the operation of the image processing apparatus according to the embodiment. FIG. 2 is a diagram for explaining an example of the operation of the image processing device according to the embodiment, and shows an example of a vehicle and a parking lot line drawn on the road surface of a parking lot. 2A and 2B are diagrams for explaining an example of the operation of the image processing device according to the embodiment, in which (a) is a diagram schematically showing an overhead view image, and (b) is a diagram schematically showing an edge detected from the overhead view image. FIG. FIG. 6 is a diagram for explaining an example of the operation of the image processing apparatus according to the embodiment, and is an enlarged view of area A in FIG. 6(b). FIG. 6 is a diagram for explaining an example of the operation of the image processing apparatus according to the embodiment, and is an enlarged view of region B in FIG. 6(b).

(Schematic configuration of parking assist device)
Embodiments of the present invention will be described below based on the drawings. FIG. 1 is a block diagram showing a schematic configuration of a parking assistance device to which an image processing device according to an embodiment of the present invention is applied. FIG. 2 is a diagram showing an example of the arrangement position of the imaging device of the parking assist device. Note that although a parking assistance device will be described below, the device to which the image processing device according to the embodiment of the present invention is applied is not limited to the parking assistance device; It can also be applied to driving support devices and the like.

As shown in FIG. 1, the parking assistance device 1 is mounted on a vehicle V (see FIG. 2) and performs a parking assistance operation. Specifically, the parking support device 1 recognizes a parking slot in which this vehicle V can park. Then, the parking support device 1 controls the vehicle V to park the vehicle V in the recognized parking slot.

As shown in FIG. 2, a plurality of small cameras (imaging devices) are provided at the front, rear, left and right sides of the vehicle V.

Specifically, a front camera 20a is mounted on the front bumper or front grill of the vehicle V so as to face the front of the vehicle V. A rear camera 20b is attached to the rear bumper or rear garnish of the vehicle V, facing toward the rear of the vehicle V. A left side camera 20c is mounted on the left door mirror of the vehicle V to face the left side of the vehicle V. A right side camera 20d is mounted on the right door mirror of the vehicle V, facing toward the right side of the vehicle V.

The front camera 20a, the rear camera 20b, the left side camera 20c, and the right side camera 20d are each equipped with a wide-angle lens or a fisheye lens that can observe a wide range. It is now possible to observe the entire area including the road surface. These cameras 20a to 20d constitute an imaging device that images the road surface around the vehicle V. In the following description, when the individual cameras (imaging devices) 20a to 20d are described without distinction, they will be simply referred to as the camera 20.

Returning to FIG. 1, the parking assist device 1 includes a front camera 20a, a rear camera 20b, a left camera 20c, a right camera 20d, a camera ECU 21, a navigation device 30, a wheel speed sensor 32, and a steering angle sensor 33. and has.

The camera ECU 21 is mainly composed of a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The camera ECU 21 controls the camera 20, and uses information detected by the camera 20 to perform processes for generating an overhead image, detection processing for detecting a parking frame, and determining whether the vehicle V can be parked in the detected parking frame. Performs judgment processing etc. to judge.

The navigation device (display device) 30 has a monitor 31 having an image display function. The navigation device 30 has a storage unit that stores map data for route guidance and the like. The navigation device 30 provides route guidance to the target point set by the operator of the navigation device 30 based on the map data and the current position of the vehicle V detected by a GPS device (not shown) or the like. Various images during the route guidance operation are displayed on the monitor 31.

The wheel speed sensor 32 is a sensor that detects the wheel speed of the vehicle V. Detection information (wheel speed) detected by the wheel speed sensor 32 is input to the vehicle control ECU 40.

The steering angle sensor 33 detects the steering angle of the steering wheel of the vehicle V. The steering angle when the vehicle V runs straight is set as a neutral position (0 degrees), and the rotation angle from the neutral position is output as the steering angle. Detection information (steering angle) detected by the steering angle sensor 33 is input to the vehicle control ECU 40.

Further, the parking assist device 1 includes a vehicle control ECU 40, a steering control unit 50, a throttle control unit 60, and a brake control unit 70.

The vehicle control ECU 40 is mainly composed of a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The vehicle control ECU 40 executes various processes to support parking of the vehicle V based on each detection information input from the camera ECU 21, the wheel speed sensor 32, and the steering angle sensor 33.

That is, for example, when the driver turns on an automatic parking start switch (not shown) to activate the parking support device 1, the vehicle control ECU 40 automatically parks the vehicle V in the parking slot that the camera ECU 21 determines is available for parking. Executes automatic parking processing.

The steering control unit 50 controls the steering angle of the vehicle V by driving the power steering actuator 51 based on vehicle control information determined by the vehicle control ECU 40.

The throttle control unit 60 controls the throttle of the vehicle V by driving the throttle actuator 61 based on vehicle control information determined by the vehicle control ECU 40.

The brake control unit 70 controls the brakes of the vehicle V by driving the brake actuator 71 based on vehicle control information determined by the vehicle control ECU 40.

The camera ECU 21, wheel speed sensor 32, and steering angle sensor 33 are connected to the vehicle control ECU 40 by a sensor information CAN (registered trademark) (Controller Area Network) 80, which is an in-vehicle LAN (Local Area Network). .

Further, the steering control unit 50, throttle control unit 60, and brake control unit 70 are connected to the vehicle control ECU 40 by a vehicle information CAN (registered trademark) 81, which is an in-vehicle LAN.

In the parking assist device 1 having the above configuration, the image processing device 100 of this embodiment is mainly configured by the camera ECU 21.

(Functional configuration of image processing device)
FIG. 3 is a functional block diagram showing a schematic configuration of image processing apparatus 100 according to this embodiment. Image processing apparatus 100 according to this embodiment includes a control section 110 and a storage section 120. The control unit 110 is mainly composed of the CPU of the camera ECU 21, and the storage unit 120 is mainly composed of the ROM, RAM, flash memory, etc. of the camera ECU 21.

The control unit 110 controls the entire image processing apparatus 100. In addition, the control unit 110 determines whether the vehicle V can park based on parking lot lines and parking spaces that demarcate parking spaces detected and estimated by an edge detection unit 111, a comparison unit 112, and a parking space detection unit 113, which will be described later. In order to have the vehicle control ECU 40 execute automatic parking processing to automatically park this vehicle V in the parking slot determined as It is sent to the ECU 40.

The vehicle control ECU 40 adjusts the power based on the information provided from the control unit 110 and further based on the detection information detected by the wheel speed sensor 32 and the steering angle sensor 33 (only sensors are shown in FIG. 3). The steering actuator 51, the throttle actuator 61, and the brake actuator 71 (only actuators are shown in FIG. 3) are drive-controlled.

The control unit 110 includes arithmetic elements such as a CPU, a programmable logic device such as an FPGA, and an integrated circuit such as an ASIC.

A control program (not shown) is stored in the storage unit 120 of the image processing device 100, and this control program is executed by the control unit 110 when the image processing device 100 is started up, and the image processing device 100 operates as shown in FIG. It will have the functional configuration as shown. In particular, since the image processing apparatus 100 of this embodiment performs high-speed image processing as described later, it is preferable to include an arithmetic element capable of high-speed calculation, such as an FPGA.

As shown in FIG. 3, the control section 110 includes an edge detection section 111, a comparison section 112, a parking frame detection section 113, and a display control section 114. Note that when the image processing device 100 and the image processing method are applied to a driving support device and a driving support method, a driving lane detection unit can be used instead of the parking frame detection unit 113 in FIG. An image processing device and an image processing method for detecting are obtained.

The edge detection unit 111 detects edges of parking lot lines on the road surface of a parking lot or the like by edge detection based on an image signal output from the camera 20 that images the road surface R around the vehicle V. Furthermore, when applied to a driving support device, the edge detection unit 111 detects edges such as lane boundary lines on the road surface. The parking lot line here refers to a line drawn as a boundary line (straight line) that mainly divides a parking area provided on the road surface R. FIG. 5 shows an example of a vehicle V and a parking lot line 200 drawn on the road surface R of the parking lot P where the vehicle V is attempting to park. Between the parking lot lines 200 is a parking frame 201 representing a parking space. Further, FIG. 6(a) is a diagram schematically showing an overhead image G generated by combining image signals taken by the camera 20, and FIG. FIG. 3 is a diagram schematically showing edges.

In addition, lane boundary lines are mainly drawn as boundaries that separate driving lanes on the road surface, and are either unbroken solid lines (straight or curved lines) or continuous short line segments of a predetermined length. A broken line (straight or curved).

Parking lane lines and lane lane lines are generally indicated by white lines, but may also be drawn by lines of a color other than white, such as yellow lines, for example. Therefore, the parking lot line or lane boundary line detected by the edge detection unit 111 is not limited to a "white line", but in general, a parking lot line or lane boundary line that has contrast with the road surface is used. It is sufficient to detect it as a line.

The edge detection unit 111 scans the image in a predetermined direction to detect pixels whose brightness or color parameters (for example, RGB, RGBA, etc.) included in the image signal change more than a threshold value. A portion where the length of the pixel arrangement is longer than a predetermined length is detected as an edge. Scanning here refers to selecting pixels one by one in a predetermined direction and comparing brightness or color parameters between adjacent pixels. The detected edge is defined as a first edge or a second edge depending on the direction (trend) of change in the brightness or color parameter.

Note that the scanning direction is preferably set in a direction perpendicular to the parking lot line drawn on the road surface R. That is, as shown in FIG. 5, when the parking lot line 200 extends in a direction perpendicular to the traveling direction of the vehicle V, scanning is performed on the overhead image G shown in FIG. 6(a) along the traveling direction. is desirable. On the other hand, when the parking lot line 200 extends along the traveling direction of the vehicle V, it is desirable to scan the overhead image G in a direction perpendicular to the traveling direction. Generally, since the direction in which the parking lot line 200 extends is unknown, the edge detection unit 111 detects the direction in which the parking lot line 200 extends in two steps, respectively along the traveling direction of the vehicle V and the direction perpendicular thereto on the overhead image G. It is desirable to scan.

Then, the edge detection unit 111 selects an edge where the luminance difference or color parameter difference between adjacent pixels is larger than a predetermined value in the positive direction from the plurality of detected edges as a first edge ("plus edge", " An edge where the luminance difference or color parameter difference between adjacent pixels is larger than a predetermined value in the negative direction is detected as a second edge (also called a "minus edge" or "falling edge"). ) is detected.

Here, when extracting edges based on brightness, there is a change (in the positive direction) from a low brightness and dark pixel (for example, a black pixel) to a high brightness and bright pixel (for example, a white pixel) with a larger difference than the threshold value. The sequence of pixels that have changed) is detected as a plus edge. In other words, this indicates that the scanning position has been switched from the road surface R to what is presumed to be the parking lot line. Further, a row of pixels that changes from a bright pixel with high brightness to a dark pixel with low brightness with a difference greater than a threshold value (changes in the negative direction) is detected as a negative edge. In other words, this indicates that the scanning position has been switched from what is estimated to be the parking lot line to the road surface R.

On the other hand, when extracting edges based on color parameters, the road surface color parameter and the parking lot line color parameter are compared. The edge detection unit 111 detects as a negative edge the arrangement of pixels in which the value of the color parameter changes in the direction of increasing (changes in the negative direction), and detects as a negative edge the arrangement of pixels in which the value of the color parameter changes in the direction of decreasing (changes in the positive direction). ) is detected as a plus edge. Furthermore, when the brightness of the parking lot line is lower than that of the road surface (or the color parameter is larger), the changes in the brightness and color parameters are reversed. In either case, a plus edge and a minus edge are detected on both sides of a boundary line such as a parking lot line, so it is possible to extract pairs as described below.

By repeating the above scanning for multiple lines (rows), a line segment (pixel arrangement, pixel column) consisting of continuous plus edges in the direction intersecting the scanning direction is Extract as minutes. Furthermore, a line segment (pixel column) composed of consecutive negative edges is extracted as a line segment of negative edges (second edge). The extracted plus edge line segments and minus edge line segments are filtered by length according to the reference length, and plus edge line segments and minus edge line segments that are less than the reference length are discarded. In addition to the length, filtering may also be performed based on the direction (angle) in which the line segment extends.

Next, the edge detection unit 111 calculates the positions (coordinates) of each end point of the plus edge line segment and the minus edge line segment that remain as a result of the filtering, and based on this position, detects adjacent plus edges at a predetermined interval. Extract pairs of line segments with a negative edge and a line segment with a negative edge.

By the way, depending on the camera 20 used, the edge enhancement function may be used to detect long parts other than the boundary between the parking lot line and the road surface, such as the boundary between the shadow of a pole etc. and the road surface, the boundary between a shaded area and a sunny area, and the road surface texture. etc. may appear white. In particular, when an image is converted into a bird's-eye view, an image signal captured at a position far from the camera 20 is easily affected by edge enhancement. As a result, unnatural white lines (hereinafter referred to as "false white lines") that do not originally exist may appear in the image. edges are also detected and may be mistakenly recognized as parking lot lines, etc. As a result, the accuracy of parking frame detection and lane detection may be affected.

In order to avoid such misrecognition, the comparison unit 112 compares the brightness included in the image signals around the edges of each pair extracted by the edge detection unit 111, and selects the parking lot line from among the plurality of pairs. Only the pairs of edges that make up are extracted, and the other pairs are discarded as pairs of false white line edges.

For this reason, the comparison unit 112 calculates, for each pair, the brightness of a pixel on the positive edge side in a predetermined direction and the brightness of a pixel on the negative edge side in the opposite direction from the positive edge side in the predetermined direction. compare. More specifically, the luminance of a pixel offset by a predetermined offset value from the plus edge toward the scanning source side is compared with the luminance of a pixel offset by a predetermined offset value from the minus edge toward the scanning destination side. If the brightness difference is less than or equal to a predetermined value, the plus edge and minus edge are determined to be a pair of edges forming a parking lot line. Pairs exceeding a predetermined value are discarded as forming edges of line segments other than parking lot lines. The offset here refers to a parallel movement from a pixel forming a plus or minus edge in a predetermined direction (a plus edge to the scanning source side and a minus edge to the scanning destination side) by an amount equal to the offset value. The pixels of this parallel movement destination are compared.

When comparing the luminances, at least one pixel may be selected and compared from among the pixel columns to which the plus edge and the minus edge are parallelly moved. More preferably, a plurality of pixels are selected from the pixel row at arbitrary or predetermined intervals, or pixels at a plurality of division points obtained by dividing the pixel row at equal intervals are selected, and the average values of their luminances are compared with each other. do. This improves the accuracy of brightness comparison and improves the accuracy of determining pairs forming parking lot lines.

The outside of the parking lot line is usually the road surface, so the brightness of the pixel offset from the positive edge detected from the parking lot line toward the scanning source side and the brightness of the pixel offset from the negative edge toward the scanning destination side are: Both are road surface brightness. Therefore, there is almost no difference in brightness between these pixels. Therefore, if the brightness difference is less than or equal to a predetermined value, it is determined that the pair constitutes a parking lot line.

On the other hand, for example, if a false white line appears at the boundary between the shadow of a pole or the like and the road surface due to the edge enhancement function of the camera 20, and a pair of edges is detected, pixels offset from one edge Luminance is obtained. The luminance of the shadow, which is darker than the road surface, is obtained from the pixels offset from the other edge. Therefore, the difference in brightness between these pixels becomes large. Therefore, if the brightness difference exceeds a predetermined value, the pair is determined to constitute a false white line other than a parking lot line. By discarding a pair of edges that constitute such a false white line, it is possible to improve the accuracy of parking frame detection by the parking frame detection unit 113, which will be described later.

In this embodiment, the brightness is specifically compared in the following procedure. First, the pixel row (line segment) that constitutes the plus edge is offset to the scanning source side based on the offset value, and the pixel row (line segment) is divided equally along the extending direction. The luminance is acquired and the average value Yp is calculated.

In addition, the pixel row (line segment) offset from the pixel row (line segment) constituting the minus edge to the scanning destination side based on the offset value is equally divided along the extending direction, and the pixels at multiple division points are The luminance is acquired and the average value Ym is calculated.

Then, when the calculated average values Yp and Ym satisfy the following formula (1), the comparison unit 112 determines that the plus edge and the minus edge are a pair of edges forming a parking lot line.

|Yp - Ym| ≦ Threshold (1)

Further, offset pixels may be compared based on color parameters instead of brightness. Further, the threshold value for the luminance difference or the color parameter difference is not particularly limited, and can be set as appropriate depending on the state of the image signal of the camera 20 used. Further, the offset value is not particularly limited, and may be set at a position where the brightness or color parameters of the portion near the edge can be appropriately compared depending on the degree of edge emphasis, resolution, etc. of the camera 20. You can set the value that can be offset as appropriate. The offset value can be, for example, the number of pixels. Further, the number of pixels used for comparison of brightness or color parameters is not particularly limited, and can be set as appropriate depending on resolution and the like. Furthermore, the number of pixels used for comparison of brightness or color parameters can be increased or decreased depending on the length of the line segment (number of pixels) or the like.

In this embodiment, pixels are compared based on brightness, and the threshold value of the brightness difference (absolute value) is set to 45. Further, the offset value is set to 2 pixels. Furthermore, the number of pixels whose brightness is compared is 10 (10 points). That is, a line segment at a position offset from the line segment of each edge is divided into 9 equal parts, the brightness of 10 pixels corresponding to 10 dividing points including the starting point and ending point of the line segment is obtained, and the average value is calculated. is being calculated.

Further, the number of pixels and the number of divisions used for comparison of threshold values, offset values, and brightness are set in advance in the storage unit 120 as parameter data 122. These parameter data 122 may be set fixedly, but may also be set arbitrarily by the manufacturer or the user. For example, the specifications may be set from the initial setting screen of the navigation device 30, or the specifications may be set when updating the software or data. Since settings can be made appropriately according to the specifications of the camera 20, for example, only the image processing device 100 of this embodiment needs to be newly installed in the vehicle V, and the existing camera 20 etc. can be used, increasing versatility. An excellent image processing device 100 can be provided.

The parking frame detection unit 113 determines the possibility of forming a parking frame from among the plus edge line segments and minus edge line segments of the pair determined by the comparison unit 112 to be a pair of edges forming a parking lot line. Detect line segments of adjacent edges with . A parking frame (that is, a parking space divided by a parking frame) is detected based on the distance between the detected edge line segments.

More specifically, the parking frame detection unit 113 first detects two adjacent lines that may constitute a parking space among the positive edge lines and negative edge lines of a plurality of pairs of parking lot lines. Select. The two lines selected here are the lines that constitute the left and right ends of the pair of parking lot lines that separate the parking spaces, and are the negative edge of the predetermined parking lot line (for example, K1 shown in FIG. 6(b)). (Em1) and a positive edge line (Ep2) of the parking lot line (K2 shown in FIG. 6(b)) facing this.

Then, the parking frame detection unit 113 calculates the distance between the lines of the two selected edges (inner dimensions of adjacent parking lot lines) based on the coordinates of the end points of each edge, and the calculated distance is , determine whether it is within a predetermined range. If this distance is within the range of the predetermined parking space width±threshold value, the area partitioned by the two edge lines is detected as a parking space. The width of the parking space is preferably 2m to 3m if it is for a regular car or small cargo vehicle. If it is a parking space for a large freight vehicle or bus, a height of 3.3 m or more is desirable.

Since edges detected from false white lines other than parking lot lines are discarded by the comparison unit 112, the parking space detection unit 113 can detect parking spaces with high precision. That is, it is possible to prevent the area between the parking lot line and the shadow (for example, between the edge line segments Em3 and Ep4 in FIG. 6(b)) from being mistakenly detected as a parking space.

Then, the parking frame detection unit 113 estimates the shape of the parking space, that is, the parking frame (201 shown in FIG. 5) based on the detected parking space, and stores the coordinate value in the storage unit 120 as the parking frame registration data 121. Register.

The display control unit 114 displays a road surface image around the vehicle V captured by the camera 20 and an image indicating a parking space detected and estimated by the parking space detection unit 113, overlapping this road surface image as appropriate, or performing navigation alone. A display control signal for display on the monitor 31 of the device (display device) 30 is sent to the navigation device 30.

The storage unit 120 includes a storage medium such as a large capacity storage medium such as a hard disk drive, or a semiconductor storage medium such as ROM or RAM. The storage unit 120 temporarily or non-temporarily stores various data used during various operations in the control unit 110.

Furthermore, as described above, the storage unit 120 stores parking space registration data 121 and parameter data 122. As parameter data 122, in addition to thresholds, offset values, the number of pixels and the number of divisions used for brightness comparison, reference lengths of boundary lines such as parking lot lines and lane boundary lines, parking space widths, and their thresholds are included. It can also be stored. Furthermore, various parameters used by the image processing apparatus 100, such as the width of the boundary line and the angle of the extending direction, can also be stored. In addition, a plurality of parameters are stored corresponding to the country and region where the parking assistance device 1 is used, the shape and size of the parking space (parking frame), the distance between driving lanes, the shape of the lane boundary line, etc. It may also be configured to select appropriate parameters.

(Operation of image processing device)
Next, an example of the operation of the image processing apparatus 100 according to this embodiment will be described with reference to the flowchart of FIG. 4 and FIGS. 5 to 8.

FIG. 4 is a flowchart for explaining the operation of the image processing apparatus 100. The operation shown in the flowchart of FIG. 4 starts when the driver inputs an instruction to start automatic parking by operating an automatic parking start switch (not shown).

In step S1, the control unit 110 of the image processing device 100 acquires an image signal of the road surface R around the vehicle V captured by the camera 20.

In step S2, based on the image signal acquired in step S1, the control unit 110 generates a signal by combining these image signals. The signal synthesized in step S2 is a signal for causing the navigation device 30 to display an image (overhead image) as if a camera was installed above the vehicle V and looked down directly below. Techniques for generating such a bird's-eye view image are well known, and examples thereof include techniques disclosed in Japanese Patent Laid-Open Nos. 3-99952 and 2003-118522.

FIG. 6(a) is an example of an overhead image G based on the combined signals. This bird's-eye view image G is converted into bird's-eye view images g1, g2, g3, and g4 that look down on the vehicle V from directly above, respectively, based on the image signals taken by the cameras 20a to 20d, and each bird's-eye view image g1 is further synthesized. This is an image generated by In the center of the bird's-eye view image G, an icon I indicating a state in which the vehicle V is viewed from directly above is displayed.

Note that the image compositing operation in step S2 may not be performed, or the image compositing operation in step S2 may be performed after the extraction of plus edges and minus edges in the next step S3. However, the processing load on the image processing apparatus 100 can be reduced by performing the work of extracting the plus edges and minus edges after generating the bird's-eye view image G.

In step S3 (edge detection step), as described above, the edge detection unit 111 scans the overhead image G synthesized in step S2 in a predetermined direction, and detects positive edges and Extract negative edges.

As an example, as shown in FIG. 6(a), the X-axis of the bird's-eye view image G (in this case, the The Y-axis (in this case, the direction in which the parking lot line 200 extends) is set in the vertical direction in the diagram. The edge detection unit 111 scans the bird's-eye view image G in a direction perpendicular to the traveling direction of the vehicle V, from left to right in the figure (X-axis positive direction). As a result, positive (+) edges and negative (-) edges in the image are detected. Note that when pixels are scanned from right to left in the figure, that is, in the negative direction of the X axis, the plus edges and minus edges are reversed. Further, plus edges and minus edges may be detected based on information on color parameters (for example, RGB, RGBA, etc.) included in the image signal. In this case, these are detected based on changes in the size (gradation) of predetermined colors.

In FIG. 6(b), positive edges detected in the overhead image G are schematically shown with thick lines, and negative edges are schematically shown with thick broken lines. As shown in FIG. 6(b), line segments Ep1, Ep2, Ep3, and Ep6 with positive edges are detected on the left side (scanning source) and on the right side (scanning destination) in the outer peripheral portion of the four parking lot lines. ), negative edge line segments Em1, Em2, Em3, and Em6 were detected. In addition, due to the effect of the edge enhancement function, plus-edge line segments Ep4 and Ep5 and minus-edge line segments Em4 and Em5 were detected on both sides of the shadow of the elongated object (pole), respectively.

In addition to this, when there is light reflection on the road surface, dust, dirt, etc., these edges may also be detected as noise (short edges). These noises are discarded by filtering in the next step S4.

In the next step S4, the edge detection unit 111 performs filtering of the plus edges and minus edges detected in step S3. Although this filtering can be performed after pair extraction in the next step S5, image processing can be speeded up by performing it before pair extraction to remove noise.

First, the edge detection unit 111 extracts a plus edge line segment and a minus edge line segment that have a length equal to or greater than a predetermined reference length and extend in a predetermined direction. The reference length is, for example, the length of the vehicle V (for example, 5 m), but if the parking lot line is short, the reference length is set to be shorter than the vehicle length. The angle is set in consideration of the traveling direction of the vehicle V, the direction of the camera 20 that took the image, and the like. In the case of FIG. 6, the parking lot line is a straight line extending substantially perpendicularly toward the parking space with respect to the driving direction, so the angle is set to 90°±threshold.

Through this process, plus edge line segments Ep1 to Ep6 and minus edge line segments Em1 to Em6 are extracted. On the other hand, short edges detected due to reflections on the road surface, the presence of dust, etc., and long edge line segments that extend in a direction other than the vertical direction are not extracted and are discarded as noise.

In the next step S5, the edge detection unit 111 extracts a pair of adjacent plus edge line segments and minus edge line segments from the extracted multiple edge line segments. At this time, the distance between a plus edge and a minus edge adjacent to each other on the road surface is calculated based on the bird's-eye view image G, and if this distance is within the range of a predetermined line width±threshold value, these are determined to be a pair.

In the example of FIG. 6A, a pair of a plus-edge line segment Ep1 and a minus-edge line segment Em1 is extracted. Similarly, Ep2 and Em2, Ep3 and Em3, Ep4 and Em4, Ep5 and Em5, and Ep6 and Em6 are extracted as pairs. In other words, edges of false white lines other than parking lot lines are also extracted as candidates for pairs of edges constituting parking lot lines.

Therefore, in the parking lot line extraction loop process (comparison step) in the next steps S6 to S11, only the pairs of edges that make up the parking lot lines are extracted, and the pairs of edges that make up the false white lines are discarded. The processing of steps S6 to S11 ends when it is determined that all pairs have been processed.

First, in step S7, the luminance of 10 pixels offset to the scanning source side from the plus edge line segment of the pair to be processed is acquired, and the average value Yp is calculated. Next, in step S8, the brightness of ten pixels offset toward the scanning destination from the line segment of the negative edge of the pair to be processed is acquired, and the average value Ym is calculated.

Next, in step S9, it is determined whether the difference between the average luminance values Yp and Ym is less than or equal to a threshold value, that is, whether the above-mentioned formula (1) is satisfied. If formula (1) is satisfied (the difference is less than or equal to the threshold), it is determined that the pair constitutes a parking lot line, and the process proceeds to step S10, where the line sandwiched between the pair of plus edges and minus edges is is registered in the storage unit 120 as a parking lot line. Thereafter, if there is a next pair to be processed, the process returns to step S7, and if there is no pair to process, the process ends the loop and proceeds to step S12.

On the other hand, if formula (1) is not satisfied (the difference exceeds the threshold), it is determined that the pair constitutes a false white line other than the parking lot line, and step S10 is skipped and the pair is discarded. . This removes false white lines. Thereafter, if there is a next pair to be processed, the process returns to step S7, and if there is no pair to process, the process ends the loop and proceeds to step S12.

A specific example of the processing in steps S7 to S10 will be described below with reference to FIGS. 7 and 8. FIG. 7 is an enlarged view of region A in FIG. 6(b), showing a pair of a plus-edge line segment Ep1 and a minus-edge line segment Em1 detected on the outer periphery of the parking lot line. FIG. 8 is an enlarged view of region B in FIG. 6(b), showing a pair of plus edge line segment Ep4 and minus edge line segment Em4 detected on the outer periphery of the false white line, and a pair of plus edge line segment Ep5 and A pair of line segments Em5 with negative edges is shown.

In the example shown in FIG. 7, line segment Lp1 is a pixel column that is parallel shifted (offset) by 2 pixels (2 pixels) from the plus edge line segment Ep1 toward the scanning source side, and 2 pixels (2 pixels) from the minus edge line segment Em1 toward the scanning destination side. ) Extract the line segment Lm1 of the pixel row that has been translated (offset). Each line segment Lp1, Lm1 is divided into nine equal parts, and the brightness of 10 pixels Q including both ends (start point and end point) of each line segment Lp1, Lm1 is obtained. A plurality of acquired luminances are averaged to calculate average values Yp and Ym (hereinafter, the same numbers as the line segments are given to distinguish them, such as "Yp1" and "Ym1").

Similarly, in the example shown in FIG. 8, line segments Lp4 and Lm4 are extracted based on the pair of plus-edge line segment Ep4 and minus-edge line segment Em4, and 10 points are obtained by dividing these into 9 equal parts. The average luminance values Yp4 and Ym4 of the pixel Q are calculated. In addition, line segments Lp5 and Lm5 are extracted based on the pair of the plus edge line segment Ep5 and the minus edge line segment Em5, and the average value Yp5 of the brightness of the 10 pixels Q obtained by dividing these into 9 equal parts. , Ym5 are calculated.

Since the pair of line segments Ep1 and Em1 shown in FIG. 7 constitute a parking lot line, the road surface brightness is obtained for the line segments Lp1 and Lm1, and the difference between these average values Yp1 and Ym1 is less than or equal to the threshold value. Become. Therefore, the edge line segments Ep1 and Em1 that satisfy equation (1) are determined to be a pair of edges forming a parking lot line, and are registered in the storage unit 120 as a parking lot line.

On the other hand, the pair of line segments Ep4 and Em4 and the pair of line segments Ep5 and Em5 shown in FIG. 8 are false white line edges that appear around the shadow due to the edge enhancement function. Therefore, the brightness of the road surface is acquired for the line segments Lp4 and Lm5, and the brightness of a shadow darker than the road surface is acquired for the line segments Lm4 and Lp5. Therefore, the differences between the average value Yp4 and the average value Ym4 and between the average value Yp5 and the average value Ym5 become large, and the equation (1) is not satisfied, and these pairs are discarded.

From the above, in the example of FIG. 6(b), the parking lot line K1 is composed of pairs of plus edge line segments and minus edge line segments (Ep1 and Em1, Ep2 and Em2, Ep3 and Em3, and Ep6 and Em6). ~K4 is registered in the storage unit 120.

Returning to FIG. 4, after passing through the parking lot line extraction loop processing of steps S6 to S11, in the next step S12, the parking frame detection unit 113 detects the parking lot line registered in the storage unit 120 (see FIG. 6(b)). In the example, parking frames and parking spaces are detected based on parking lot lines K1 to K4). To do this, as described above, the distance between the opposing positive edge line segment and negative edge line segment of adjacent parking lot lines K1 to K4 that may constitute a parking space is calculated. If this distance is within the range of the predetermined parking space width±threshold value, it is determined that the area between these two parking lot lines is a parking space. In the example of FIG. 6B, the areas between the line segment Em1 and the line segment Ep2, between the line segment Em2 and the line segment Ep3, and between the line segment Em3 and the line segment Ep6, which have positive or negative edges, are detected as parking spaces. Then, the parking frame is a rectangular frame whose long side is a line along the opposite positive edge line segment and negative edge line segment that make up the detected parking space, and whose short side is a line connecting the opposing ends. 201 (see FIG. 5).

In step S13, the parking frame detection unit 113 calculates the coordinate values of the end points of the opposing plus-edge line segments and minus-edge line segments that constitute each parking frame 201, and stores them in the storage unit 120 as parking frame registration data 121. register. At this time, by registering the coordinate values of at least the two end points of the parking frame 201 on the side closer to the vehicle V, the parking frame 201 can be identified while minimizing the storage capacity. However, if the coordinate values of the four points are registered, Good too. Further, the angle (extending direction) of the parking lot line 200 and other information necessary for automatic parking processing can also be added to the parking space registration data 121.

Furthermore, the parking frame detection unit 113 performs a determination process and the like to determine whether or not the vehicle V can be parked in the detected parking frame. For example, if there are other vehicles or obstacles in the detected parking space, it is determined that parking is not possible and the parking space is not registered in the storage unit 120 as the parking space registration data 121. Moreover, the parking frame detection unit 113 can also determine a parking frame that is close to the vehicle V or a parking frame that is easy to park in as a parking frame that can be parked, and register it in the storage unit 120 as the parking frame registration data 121.

The parking space registration data 121 detected as described above is sent to the vehicle control ECU 40, and various processes for supporting parking of the vehicle V are executed.

(Effect of image processing device)
In the image processing device 100 according to the present embodiment configured as described above, the edge detection unit 111 converts an image based on an image signal output from the camera 20 that images the road surface R around the vehicle V in a predetermined direction. The image signal is scanned to detect pixels whose brightness included in the image signal changes more than a threshold value, and a portion where the detected pixel arrangement has a length longer than a predetermined value is detected as an edge. Then, the comparison unit 112 compares the luminance of the pixel located on the predetermined direction side of the first edge detected by the edge detection unit 111 with the luminance of the pixel located on the side of the second edge in the predetermined direction opposite to the predetermined direction side of the first edge. The brightness of the pixel is compared, and if the brightness difference is less than or equal to a predetermined value, the first edge and the second edge are determined to be a pair of edges forming a boundary line.

In this embodiment, the first edge and the second edge are a positive edge where the difference in brightness between adjacent pixels is greater than a predetermined value in the positive direction, and a positive edge where the difference in brightness between adjacent pixels is increased in the negative direction. This is a negative edge that is larger than a predetermined value.

As a result, pairs of edges constituting shadows of trees or poles extending on the road surface whose edges have been emphasized by the edge enhancement function of the camera 20 are excluded, and it is possible to suppress the pair of edges from being determined as a boundary line. As a result, it is possible to provide an image processing device and an image processing method that can detect boundary lines that partition a road surface, such as parking lot lines and lane boundary lines, with high precision. As a result, it is possible to provide an image processing device 100 and an image processing method that can be applied to any vehicle V equipped with any camera 20, regardless of the functions of the camera 20.

In addition, by including the image processing device 100 or the image processing method, a parking support device, a parking support method, a driving support device, and a driving support method that can detect parking spaces and driving lanes with high accuracy can be provided. Can be provided.

Furthermore, the comparison unit 112 compares the luminance of a pixel on the side of a predetermined direction of the first edge (plus edge) with the luminance of a pixel on the side of the second edge (minus edge) in the predetermined direction and the opposite side of the first edge. When comparing the brightness of pixels, it is desirable to use pixels that are offset by a predetermined amount from the first edge and the second edge. Thereby, for example, it is possible to compare the brightness of a road surface portion with clearly different brightness and the brightness of a shadow portion, and it is possible to further improve the detection accuracy of the boundary line.

Further, in the present embodiment, when the following formula is satisfied, the comparison unit 112 determines that the first edge and the second edge are a pair of edges forming a boundary line. As a result, pairs of edges other than the boundary line, such as shadows, are discarded, and only the pair of edges forming the boundary line can be extracted with high precision.

|Yp − Ym| ≦ Threshold In the above formula, Yp is the average value of the luminances at a plurality of division points obtained by equally dividing the row of pixels on the predetermined direction side of the plus edge along the extending direction. Ym is an average value of the luminances of pixels at a plurality of division points obtained by equally dividing a row of pixels on the side of the minus edge in the opposite direction from the predetermined direction side of the plus edge along the extending direction.

Further, the edge detection unit 111 may detect edges based on information on color parameters (for example, RGB, RGBA, etc.) included in the image signal instead of the brightness, and even in this case, the first edge (plus edge) and a second edge (minus edge) different from this can be detected faster and more accurately.

Further, the configuration may include a storage unit 120 in which parameters such as an offset value and various threshold values used in the edge detection unit 111 and the comparison unit 112 are set in advance. With this configuration, offset values and threshold values can be set based on the functions of the camera 20 used and the conditions of parking lots and driving lanes (presence of poles, trees, buildings, etc. that tend to create long shadows, weather, season, time, etc.). can be arbitrarily set or updated, the boundary line can be detected more appropriately, and the versatility of the image processing apparatus 100 and the image processing method is also improved.

In addition, when the boundary line is a parking lot line that divides parking spaces, adjacent pairs are detected from among the pairs of edges that constitute the boundary line determined by the comparison unit 112, and By having a configuration including a parking frame detection unit 113 that detects a parking frame based on distance, a parking frame can be detected with higher accuracy. Therefore, the image processing device 100 and the image processing method can be suitably used in a parking support device and a parking support method.

In addition, when the boundary line is a lane boundary line that divides a driving lane, the driving lane detection section detects the driving lane based on the pair of edges forming the boundary line determined by the comparison section 112. With this configuration, the driving lane can be detected with high accuracy. Therefore, the image processing device 100 and the image processing method can be suitably used in a driving support device and a driving support method.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments and examples, and design changes can be made to the extent that they do not depart from the gist of the present invention. are included in the present invention.

For example, in the image processing apparatus 100 according to the above-described embodiment, information on brightness and color parameters (for example, RGB, RGBA, etc.) of an image is based on the magnitude of change and the direction of change (positive direction or negative direction). However, the present invention is not limited to this, and edges may be detected based on the magnitude and direction of change in other information included in the image signal.

20 Camera (imaging device) 20a Front camera (imaging device)
20b Rear camera (imaging device) 20c Left side camera (imaging device)
20d Right side camera (imaging device) 100 Image processing device 111 Edge detection unit 112 Comparison unit 113 Parking frame detection unit 120 Storage unit 200 Parking lane line 201 Parking frame Em1 to Em6 Negative edge line segment (second edge)
Ep1 to Ep6 Plus edge line segment (first edge)
G Overhead image (image) R Road surface V Vehicle Ym Average value (second edge)
Yp average value (first edge)

Claims (9)

An image processing device that detects a boundary line dividing a predetermined area of a road surface, the image processing device comprising:
An image based on an image signal output from an imaging device that images the road surface around the vehicle is scanned in a predetermined direction, and pixels whose brightness included in the image signal changes more than a threshold value are detected. an edge detection unit that detects a portion where the pixel arrangement is longer than a predetermined length as an edge;
The average value of the luminance of a plurality of pixels selected from a pixel column located on the predetermined direction side of the first edge detected by the edge detection unit and the second edge of the second edge different from the first edge. and the average value of the brightness of a plurality of pixels selected from the pixel row on the side opposite to the predetermined direction of the first edge.If the difference in brightness is less than a predetermined value, An image processing device comprising: a comparison unit that determines an edge and the second edge to be a pair of edges forming a boundary line. The first edge and the second edge include a positive edge where the brightness difference between adjacent pixels is larger than a predetermined value in a positive direction, and a positive edge where the brightness difference between adjacent pixels is larger than a predetermined value in a negative direction. The image processing apparatus according to claim 1, wherein the negative edge becomes larger. The comparison unit compares the brightness of a pixel on a predetermined direction side of the first edge with the brightness of a pixel on the second edge in a direction opposite to the predetermined direction side of the first edge. 3. The image processing apparatus according to claim 1, wherein the image processing apparatus performs the comparison using pixels offset by a predetermined amount from the first edge and the second edge. An average value of the luminance of the pixels at a plurality of division points obtained by equally dividing the row of pixels on the side of the first edge in a predetermined direction along the extending direction is set as Yp,
An average of the luminances of the pixels at a plurality of division points obtained by equally dividing a row of pixels on the side of the second edge in the opposite direction from the predetermined direction side of the first edge. When the value is Ym,
The comparison unit may determine the first edge and the second edge to be a pair of edges constituting the boundary line if the following formula |Yp − Ym| ≦ threshold is satisfied. The image processing device according to any one of claims 1 to 3. The image processing device according to claim 1, wherein the edge detection unit detects the edge based on a color parameter included in the image signal instead of brightness. The image processing apparatus according to any one of claims 1 to 5, further comprising a storage unit in which parameters and threshold values used in the edge detection unit and the comparison unit are set in advance. The boundary line is a parking lot line dividing a parking space,
A parking frame detection unit that detects adjacent pairs of edges constituting the boundary line determined by the comparison unit, and detects a parking frame based on the distance between the adjacent pairs, The image processing apparatus according to any one of claims 1 to 6, further comprising an image processing apparatus. The boundary line is a lane boundary line that divides driving lanes,
7. The vehicle according to claim 1, further comprising a driving lane detection unit that detects the driving lane based on a pair of edges forming the boundary line determined by the comparison unit. The image processing device described. An image processing method for detecting a boundary line dividing a predetermined area of a road surface, the method comprising:
An image based on an image signal output from an imaging device that images the road surface around the vehicle is scanned in a predetermined direction, and pixels whose brightness included in the image signal changes more than a threshold value are detected. an edge detection step of detecting a portion where the pixel arrangement is longer than a predetermined length as an edge;
The average value of the luminance of a plurality of pixels selected from a pixel column located on the predetermined direction side of the first edge detected in the edge detection step, and the second edge of the second edge different from the first edge. and the average value of the brightness of a plurality of pixels selected from the pixel row on the side opposite to the predetermined direction of the first edge.If the difference in brightness is less than a predetermined value, a comparison step of determining the edge and the second edge to be a pair of edges forming a boundary line;
An image processing method characterized by comprising: