JP2016004446A - Motion estimation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve motion estimation accuracy for an object in a distant area.SOLUTION: A motion estimation device comprises: an imaging device 10 outputting picked-up images (luminance images) in time series; an image information storage unit 22 storing information on the picked-up images so as to be associated with imaging time; a distance information generation unit 23 calculating depth distance information from the imaging device 10 to pixels of each picked-up image; a search image setting unit 24 setting the picked-up image farther from the picked-up image serving as a motion estimation object as a search image corresponding to the pixels of the picked-up image serving as the motion estimation object if the pixels are present farther from an own vehicle, and setting the picked-up image closer in the imaging time to the picked-up image serving as the motion estimation object as the search image corresponding to the pixels of the picked-up image serving as the motion estimation object if the pixels are present closer to the own vehicle; and a point-of-interest estimation unit 25 and a three-dimensional motion estimation unit 26 setting each of the pixels of the picked-up image serving as the motion estimation object as a point of interest, searching a corresponding point corresponding to the point of interest from onto the search image corresponding to the pixel as the point of interest, and estimating a motion of the point of interest.

Description

  The present invention relates to a motion estimation device that estimates the motion of an object around a host vehicle based on a captured image.

  Conventionally, this kind of motion estimation apparatus is known. For example, Patent Literature 1 below discloses a periphery monitoring device that calculates a two-dimensional optical flow of a target object around a host vehicle from a time-series image captured by a stereo camera by corresponding point search processing. In the technique of Patent Document 1, a candidate area where a target object is highly likely to exist is extracted based on the two-dimensional optical flow. In this technique, the three-dimensional information of the candidate area is acquired from the image of the stereo camera, and the collision between the host vehicle and the target object is based on the positional relationship between the three-dimensional information and the periphery monitoring device having the stereo camera. Judging the possibility.

JP 2010-286926 A

  By the way, an object that is far from the own vehicle (for example, an object that exists in the vicinity of the vanishing point on the captured image) has a more apparent movement than an object near the own vehicle between frames with a short imaging interval. small. For this reason, it is difficult to calculate a two-dimensional optical flow for an object in such a far region as compared to an object near the own vehicle. Therefore, in the far field, although the image capturing timing is the same, the candidate area is likely to have a lower detection accuracy of a moving object and the presence of the target object than in the vicinity of the host vehicle. This may cause a decrease in extraction accuracy.

  Accordingly, an object of the present invention is to provide a motion estimation device that can improve the disadvantages of the conventional example and can accurately estimate the motion of an object in a far region.

  In order to achieve the above object, the present invention captures an imaging region around a host vehicle and outputs a captured image, which is a collection of pixels including luminance value information, in time series, and the captured image An image information storage unit that stores the information in association with the imaging time, a distance information generation unit that calculates depth distance information from the imaging device to the pixel of the captured image based on luminance value information of the captured image, and movement As the pixels of the captured image to be estimated are located farther from the own vehicle, the captured image of the image information storage unit that is further away from the captured image of the motion estimation target While setting as a search image associated with a pixel, the closer the pixel of the captured image of the motion estimation target is to the vehicle, the closer the imaging time is to the captured image of the motion estimation target Image information A search image setting unit that sets the captured image of the unit as a search image associated with the pixel, a pixel of the captured image that is the motion estimation target is set as a target point for each pixel, and a corresponding point corresponding to the target point And a motion estimation unit that estimates the motion of the target point from the search image associated with the pixel of the target point.

  Here, the search image setting unit has a newer imaging time in the captured image of the image information storage unit as the estimated value of the two-dimensional motion vector in the pixel of the captured image of the motion estimation target increases. Is preferably set in the search image.

  In order to achieve the above object, the present invention captures an imaging region around a host vehicle and outputs a captured image, which is a collection of pixels including luminance value information, in time series, and the captured image An image information storage unit that stores the information in association with the imaging time, a distance information generation unit that calculates depth distance information from the imaging device to the pixel of the captured image based on luminance value information of the captured image, and movement An area setting unit that divides the captured image to be estimated into a plurality of areas, and the area that is located farther from the host vehicle is more distant from the captured image that is the motion estimation object While the captured image of the image information storage unit is set as a search image associated with the region, the region existing closer to the vehicle has a longer imaging time than the captured image of the motion estimation target. Close said A search image setting unit that sets the captured image of the image information storage unit as a search image associated with the region, and a pixel of the captured image that is the motion estimation target is set as a target point for each pixel, and corresponds to the target point And a motion estimator that searches for the corresponding point from the search image associated with the region including the target point and estimates the motion of the target point.

  Here, the area setting unit includes position information of the imaging target on the captured image of the motion estimation target, and position information of an area obtained by dividing the own lane, the oncoming lane, and the roadside on the captured image of the motion estimation target. , The distance from the vanishing point to each pixel on the captured image of the motion estimation target, the depth distance of each pixel of the captured image of the motion estimation target, and the two-dimensional motion of the pixel of the captured image of the motion estimation target It is desirable to divide the captured image as the motion estimation target into a plurality of regions based on at least one of the estimated values of the vector sizes.

  Since the motion estimation apparatus according to the present invention adaptively sets a search image for estimating the motion of the attention point for each pixel of the captured image of the motion estimation target or for each region obtained by dividing the captured image, With respect to an imaging target (such as an object) in a distant area with a small apparent motion between captured images with a short imaging interval, the motion can be accurately estimated.

FIG. 1 is a block diagram showing an example of the configuration of an embodiment of a motion estimation apparatus according to the present invention. FIG. 2 is an example of a captured image at time t. FIG. 3 is an example of a captured image at time t-1. FIG. 4 is an example of a captured image at time t-3. FIG. 5 is a diagram for explaining the block matching method. FIG. 6 is a flowchart illustrating motion estimation according to the embodiment. FIG. 7 is a block diagram showing an example of the configuration of a modified example of the motion estimation apparatus according to the present invention. FIG. 8 is a flowchart illustrating motion estimation according to a modification.

  Embodiments of a motion estimation device according to the present invention will be described below in detail with reference to the drawings. The present invention is not limited to the embodiments.

[Example]
An embodiment of a motion estimation apparatus according to the present invention will be described with reference to FIGS.

  Reference numeral 1 in FIG. 1 indicates an object detection system having the motion estimation apparatus of the present embodiment. The object detection system 1 is a system capable of detecting an object or background (such as the sky) around the host vehicle based on a captured image, and determining whether the object is moving or stationary. It is. Therefore, the object detection system 1 includes an imaging device 10 that captures the periphery of the host vehicle, and an electronic control device that performs image processing for object detection based on the captured image (hereinafter referred to as “image processing ECU”). 20). In the object detection system 1, in addition to the imaging device 10, an image information acquisition unit 21, an image information storage unit 22, a distance information generation unit 23, a search image setting unit 24, an attention point motion estimation unit 25, and a three-dimensional image, which will be described later. A motion estimation unit 26 and an imaging target detection unit 27 are provided as a configuration of the motion estimation device. In particular, in this illustration, the attention point motion estimation unit 25, the three-dimensional motion estimation unit 26, and the imaging target detection unit 27 constitute a motion estimation unit that estimates the motion of the imaging target.

  As the imaging device 10, a device capable of acquiring a luminance image and a distance image is used. The captured images captured by the left camera and the right camera of the stereo camera are a collection of pixels each including luminance value information, and can be acquired as a luminance image having luminance value information for each pixel. In addition, since the stereo camera can obtain a captured image (luminance image) in the same imaging area for each of the left camera and the right camera, the parallax for each pixel in the imaging area can be obtained from the information of each captured image. Value information can be obtained, and a distance image can be generated based on the parallax value information. For this reason, a stereo camera having a right camera 11 and a left camera 12 is used for the illustrated imaging apparatus 10. The captured image may be monochrome or color. The imaging wavelength band may be in the visible region or the near infrared region. Moreover, the imaging device 10 should just be what can acquire a brightness | luminance image and a distance image. For this reason, a distance image sensor such as a TOF (Time Of Flight) camera may be used as the imaging device 10. In the following, images having the luminance value information (captured images, luminance images) are collectively referred to as captured images.

  The imaging device 10 captures an imaging target around the own vehicle with the right camera 11 and the left camera 12, respectively, and captures information on the captured image in the imaging region in time series (that is, for each captured frame). It outputs to process ECU20. The imaging targets include road surfaces, lane markings, three-dimensional objects (dynamic objects such as other vehicles and static objects such as buildings), backgrounds (such as the sky), and the like. The image processing ECU 20 passes the received captured image information to the image information acquisition unit 21 of the image processing ECU 20.

  The image information acquisition unit 21 stores the information of the received captured image in the image information storage unit 22 in time series. Further, the image information acquisition unit 21 stores the information of the captured image in the image information storage unit 22 so that information captured by the right camera 11 and information captured by the left camera 12 can be distinguished. That is, the image information storage unit 22 stores information of the captured image in association with the imaging time. The image information storage unit 22 is, for example, a temporary storage device included in the image processing ECU 20. The image information storage unit 22 may store only the number of frames necessary for estimating the motion of an object described later. For this reason, when the number of frames is exceeded, buffering may be performed so that captured images that are no longer necessary are deleted in order from the oldest imaging time. The image information acquisition unit 21 passes the received captured image information and the captured image information of the image information storage unit 22 to the distance information generation unit 23 and the search image setting unit 24 of the image processing ECU 20.

  The distance information generation unit 23 calculates the above-described parallax value information for each pixel based on the information of the captured images of the right camera 11 and the left camera 12 having the same imaging time, and based on the parallax value information for each pixel. The depth distance information of each pixel is calculated, and a distance image representing the depth distance of each position (each pixel) in the imaging region is generated. Therefore, the information on the distance image includes depth distance information for each pixel. The distance image is generated by a well-known method in this technical field.

  Here, the depth distance information is information indicating the depth distance from the imaging device 10 to the imaging target in the captured image. The direction of the depth distance is, for example, a direction orthogonal to the XY plane when the horizontal direction (lateral direction) and the vertical direction (vertical direction) of the captured image are defined as an X coordinate and a Y coordinate, respectively, This is a direction away from the imaging device 10 in the imaging direction of the device 10. For this reason, the imaging device 10 captures at least the traveling direction of the host vehicle. For the calculation of the depth distance information, for example, a dense stereo technique such as SGM method or ELAS method may be used.

  The distance information generation unit 23 passes information on the distance image having the depth distance information to the search image setting unit 24 and the three-dimensional motion estimation unit 26 of the image processing ECU 20.

  When estimating the motion of the imaging target on the captured image, the captured image (hereinafter referred to as “search”) having a corresponding point corresponding to the point of interest on the captured image of the motion estimation target (hereinafter referred to as “estimation target image”). "Image")) is searched for from the captured images having a different imaging time from the estimation target image. In principle, the search image is searched for all the pixels of the estimation target image. In this motion estimation device, for example, when a block matching method described later is applied, the two-dimensional motion of the attention point on the estimation target image is estimated between the estimation target image and the search image. In this example, the latest captured image is the estimation target image. The corresponding point is a movement source of the attention point of the estimation target image in the search image. For this reason, when there is an opening of time t0 in the imaging interval between the estimation target image and the search image, the movement destination of the corresponding point after the time t0 has passed becomes the attention point.

  Here, the imaging target that exists far away from the host vehicle may be, for example, a moving dynamic object or a static object whose relative positional relationship changes as the host vehicle travels. The apparent movement as seen from the own vehicle is smaller as it is located farther from the own vehicle. For this reason, such a distant imaging target is more noticeable as it is located farther from the own vehicle between two captured images (between frames) continuously taken while the host vehicle is traveling. Small movement. On the other hand, the imaging target that is present in the vicinity of the host vehicle, whether it is a dynamic object or a static object, is more apparent from the host vehicle as it is closer to the host vehicle. The movement is big. For this reason, the movement of the attention point increases as the nearby imaging target is located closer to the host vehicle between two captured images continuously captured while the host vehicle is traveling. Therefore, it is more difficult to estimate the presence or absence of movement and the movement of an imaging object far away from the own vehicle is more distant from the own vehicle than the imaging object near the own vehicle. Then, it is difficult to estimate the size and direction of the motion vector.

  For example, FIG. 2 shows the latest captured image (time t), and FIG. 3 shows the captured image captured immediately before (time t−1). When the distant vehicle C1 (the attention point set in the pixel representing the vehicle C1) and the nearby vehicle C2 (the attention point set in the pixel representing the vehicle C2) are compared between the two captured images, However, in the case of a distant vehicle C1, it is difficult to discriminate the position difference that can estimate the movement. The distant vehicle C1 exists near the vanishing point on the captured image. The vanishing point is a point at infinity in the traveling direction of the host vehicle on the captured image and is a point where there is no apparent movement.

  Therefore, the search image setting unit 24 of the image processing ECU 20 according to the present embodiment sets the pixel (attention point) for each pixel based on the positional relationship between the host vehicle (the imaging device 10) and the pixel of the estimation target image. A search image having corresponding points is set. That is, the search image setting unit 24 sets, for each pixel, a search image associated with the pixel that is the estimation target of the motion in the estimation target image.

  Between the two captured images (estimation target image and search image) that are compared with each other when estimating the motion of the imaging target, the relative motion amount of the imaging target increases as the imaging interval increases. It becomes easy to estimate the size and direction of the two-dimensional motion vector of the pixel in the imaging target. For this reason, it is desirable to widen the imaging interval between the estimation target image and the search image for estimating the movement of a distant imaging target with a small apparent motion existing at or near the vanishing point. . Therefore, based on the depth distance information of the pixel (attention point) of the estimation target image, this pixel (attention point) exists in the search image setting unit 24 at a distance from the own vehicle (imaging device 10). The captured image of the image information storage unit 22 that is further away from the estimation target image is set as a search image associated with the pixel (attention point).

  On the other hand, for an imaging target that exists near the host vehicle, the apparent motion between the estimation target image and the search image can be estimated without increasing the imaging interval between the estimation target image and the search image. . For this reason, in the search image setting unit 24, based on the depth distance information of the pixel (attention point) of the estimation target image, the closer this pixel (attention point) is to the own vehicle (imaging device 10). Then, the captured image of the image information storage unit 22 having a shorter imaging time than the estimation target image is set as a search image associated with the pixel (attention point).

  For example, in the search image setting unit 24, a search image (that is, the pixel is associated with the pixel when estimating the motion of the pixel according to the relative magnitude of the two-dimensional motion vector in the pixel of the estimation target image). A search image to be associated with the estimation target image when it becomes the attention point is determined from past captured images in the image information storage unit 22.

  Specifically, the search image setting unit 24 determines the estimation target when the estimation result (two-dimensional optical flow) of each pixel of the captured image one frame before the current estimation target image has not yet been obtained. The size of the two-dimensional motion vector of each pixel in the image is estimated. For example, for each pixel of the estimation target image, the search image setting unit 24 sets the depth distance information of the pixel and the amount of movement of the host vehicle (for example, the host vehicle from the time of imaging one frame before the estimation target image is captured). The size of the two-dimensional motion vector of the pixel is estimated on the basis of the movement direction and the movement amount). In the estimation, it is assumed that the imaging target represented by the pixel is a stationary object such as a static object or a road surface. When obtaining an estimated value of the size of the two-dimensional motion vector of the pixel, a three-dimensional momentum described later estimated by the three-dimensional motion estimation unit 26 may be taken into consideration.

  On the other hand, the search image setting unit 24 acquires the estimation result (two-dimensional optical flow) of each pixel of the captured image one frame before the current estimation target image from the attention point motion estimation unit 25. If it is, the estimated value of the size of the two-dimensional motion vector of each pixel in the estimation target image can be obtained based on the estimation result.

  Here, from the estimated value of the size of the two-dimensional motion vector of the pixel, it can be predicted how much the pixel exists in the estimation target image with respect to the own vehicle (imaging device 10). . For this reason, the search image setting unit 24 uses the estimated value of the size of the two-dimensional motion vector in the pixel of the estimation target image to set a search image associated with the pixel when estimating the motion of the pixel. Can do. The search image is set for each pixel of the estimation target image.

  Here, the larger the estimated value of the size of the two-dimensional motion vector of the pixel (that is, the closer the pixel is to the host vehicle), the newer the imaging time than the estimation target image. For example, the imaging time for the estimated value may be set stepwise so that the image is set as the search image. In this case, it is assumed that the imaging interval of the imaging device 10 is constant. In addition, here, the magnitude of the two-dimensional motion vector of the pixel is set such that the larger the estimated value of the magnitude of the two-dimensional motion vector of the pixel is, the newer the captured image is set as the search image. It may be defined as a function of

  For example, FIG. 4 shows a captured image captured three times before (time t−3) the captured image (estimation target image) at time t in FIG. 2 described above. The search image setting unit 24 sets the captured image at time t−1 in FIG. 3 stored in the image information storage unit 22 as a search image for each pixel representing the nearby vehicle C2 on the estimation target image. On the other hand, for each pixel representing the distant vehicle C1 on the estimation target image, the captured image at time t-3 in FIG. 4 stored in the image information storage unit 22 is set as a search image.

  In addition, on the picked-up image, there are picked up objects having no texture (pattern) over a wide area, such as freshly paved asphalt, the sky, the wall of a new building, and a blackened area under the shade. In some cases. In such a region, since it is difficult to distinguish between pixels, there is a possibility that the motion of each pixel cannot be estimated well. For this reason, the search image setting unit 24 may be configured not to set the search image for each pixel in a wide area having no texture that hinders such motion estimation. As a result, for the pixels of the estimation target image for which the search image is not set, the motion estimation by the attention point motion estimation unit 25 is omitted. Therefore, in that case, the calculation processing load can be reduced. Further, the search image setting unit 24 uses the depth distance information of the pixel together with the relative size of the two-dimensional motion vector in the pixel of the estimation target image in order to increase the accuracy when determining the search image. Also good.

  The search image setting unit 24 sends the association information between the pixel of the estimation target image and the search image to the attention point motion estimation unit 25 of the image processing ECU 20 for each pixel of the estimation target image.

  The attention point motion estimation unit 25 sets an attention point on the estimation target image, and specifies a search image associated with the attention point based on the association information. The attention point motion estimation unit 25 reads the identified search image from the captured image of the image information storage unit 22 and estimates the two-dimensional motion of the attention point. The attention point motion estimation unit 25 does not set pixels of the estimation target image having no association information as attention points.

  The attention point motion estimation unit 25 estimates the two-dimensional motion of the attention point by a well-known method in this technical field. For example, a gradient method or a block matching method can be used for estimating the movement of the attention point.

  When the gradient method is applied, the optical flow constraint equation of the following Equation 1 is configured from each pixel in the local region. The local area is a predetermined area including the point of interest, and includes a pixel of the point of interest and a plurality of pixels around the point of interest. The local area is, for example, a rectangular area having N × N pixels or M × N pixels (N, M: integer). In this conventional gradient method, the motion of the attention point is estimated by solving the simultaneous equations of Equation 2 in which the constraint equations for each pixel are simultaneously provided.

Let I (x, y, t) be the luminance at the time t of the pixel at the coordinates (x, y) of the pixel of interest (pixel of interest) on the captured image. “U (= dx / dt)” is the x component of the motion vector of the pixel at the coordinates (x, y). “V (= dy / dt)” is the y component of the motion vector of the pixel at the coordinates (x, y). “I _x (= ∂I / ∂x)” and “I _y (= ∂I / ∂y)” represent a spatial luminance gradient. “I _t (= ∂I / ∂t)” represents a temporal luminance gradient. “F” and “A” in Expression 2 are represented by the following Expressions 3 and 4, respectively.

  In the block matching method, an image at time t (an image captured at the time t) (an image captured at the time t before the time t or an image captured at the time t) (an image captured at the time t before the time t) (an image captured at the time t). A corresponding point in the image (in the captured image or in the distance image) corresponding to the attention point in the captured image or the distance image is directly searched. In this block matching method, the probability of the corresponding point is evaluated using the following formula 5 or formula 6.

  FIG. 5 is a diagram for explaining the block matching method of Equation 5. Here, (u, v) at which the SAD of Equation 5 is minimized is estimated as the movement of the point of interest (two-dimensional optical flow).

  The point-of-interest motion estimation unit 25 obtains a two-dimensional motion estimation result {two-dimensional optical flow (the size and direction of a two-dimensional motion vector)} of the point of interest, which is a target of motion estimation, from the three-dimensional image processing ECU 20. This is sent to the motion estimation unit 26.

  Based on the depth distance information of the pixel (target point) of the estimation target image and the estimation result of the two-dimensional motion of the pixel, the three-dimensional motion estimation unit 26 calculates the momentum (three-dimensional momentum) of the pixel in the three-dimensional space. Is estimated. The three-dimensional momentum is the magnitude and direction of the motion vector in the three-dimensional space of the pixel. The three-dimensional motion estimation unit 26 performs the estimation of the three-dimensional momentum on all the pixels for which both the two-dimensional optical flow and the parallax value are obtained. The three-dimensional motion estimation unit 26 sends the estimation result of the three-dimensional motion amount to the imaging target detection unit 27 of the image processing ECU 20.

  The imaging target detection unit 27 classifies each imaging target on the estimation target image based on the depth distance information of the pixel of the estimation target image and the estimation result of the three-dimensional momentum of the pixel. Specifically, the depth distance and the three-dimensional momentum between adjacent pixels are equal or similar to each other in the same group, and this is performed on each pixel of the estimation target image, so that each imaging target Perform classification. Thereby, this imaging target detection unit 27 detects each imaging target of the estimation target image together with information on the position and relative speed with respect to the host vehicle on the estimation target image. In the image processing ECU 20, the detection result is sent to the control system at the subsequent stage. In the control system, the detection result is used for collision determination with the own vehicle, creation of a collision avoidance route, and the like.

  FIG. 6 shows a flowchart relating to the arithmetic processing operation of the motion estimation apparatus.

  The image information acquisition unit 21 acquires information of a captured image (luminance image) in which each imaging target in the imaging region is projected from the imaging device 10 (step ST1). Each time the image information acquisition unit 21 receives information on a luminance image from the imaging device 10, the image information acquisition unit 21 temporarily stores the information in the image information storage unit 22 in time series in order of imaging time (step ST2). The information is output to the distance information generation unit 23 and the search image setting unit 24.

  The distance information generation unit 23 calculates the depth distance information of each pixel based on the luminance value information of each pixel included in the luminance image, and generates a distance image having the depth distance information for each pixel (step ST3). ).

  The search image setting unit 24 selects a search image to be associated with the pixel when estimating the motion of the pixel based on the estimated value of the size of the two-dimensional motion vector in the pixel of the latest captured image (estimation target image). Set (step ST4). This setting is performed for each pixel of the estimation target image in principle.

  The attention point motion estimation unit 25 sets an attention point from each pixel of the estimation target image (step ST5), and reads a search image associated with the attention point from the captured image of the image information storage unit 22. Thus, the two-dimensional movement of the attention point is estimated (step ST6).

  The point-of-interest motion estimation unit 25 estimates the motion of all points of interest (that is, pixels excluding those having no association information described above from all the pixels of the estimation target image) on the estimation target image. It is determined whether or not it is finished (step ST7). The attention point motion estimation unit 25 returns to step ST5 and estimates the motion of another attention point when the estimation of the motions of all the attention points to be estimated has not been completed. Then, when the estimation of the motion of all the points of interest to be estimated has been completed, the three-dimensional motion estimation unit 26 calculates the three-dimensional momentum of each target point based on the depth distance information and the estimation result of the two-dimensional motion. Is estimated (step ST8).

  The imaging target detection unit 27 classifies each imaging target on the estimation target image based on the depth distance information of each attention point and the estimation result of the three-dimensional momentum of each attention point. Are detected (step ST9). The image processing ECU 20 outputs information on the position and relative speed of each imaging target with respect to the host vehicle to the subsequent control system (step ST10).

  As described above, the motion estimation device according to the present exemplary embodiment applies a pixel to each pixel of the estimation target image when estimating the motion of the pixel based on the distance from the own vehicle on the estimation target image of the pixel. The search image to be associated is set adaptively. That is, in this motion estimation apparatus, the closer the pixel of the estimation target image is to the host vehicle (imaging apparatus 10), the closer the captured image of the image information storage unit 22 is to the estimation target image. Is set as a search image associated with the pixel. On the other hand, in this motion estimation device, as the pixel of the estimation target image is located farther from the own vehicle (imaging device 10), the image information storage unit is farther away from the estimation target image. 22 captured images are set as search images associated with the pixels. For this reason, this motion estimation apparatus can accurately estimate the motion of an imaging target in a distant region with a small apparent motion between captured images with a short imaging interval.

[Modification]
The motion estimation apparatus according to this modification is obtained by changing the following points from the motion estimation apparatus according to the above-described embodiment. FIG. 7 shows an object detection system 2 having a motion estimation device of this modification. This object detection system 2 is obtained by replacing the image processing ECU 20 with an image processing ECU 120 in the object detection system 1 of the embodiment.

  The image processing ECU 120 includes the same image information acquisition unit 21, image information storage unit 22, distance information generation unit 23, search image setting unit 24, attention point motion estimation unit 25, and three-dimensional motion estimation unit 26 as the image processing ECU 20 of the embodiment. And an imaging target detection unit 27, and a road region recognition unit 28, a roadside region recognition unit 29, a vanishing point detection unit 30, and a region setting unit 31. In the present modification, these configurations are included in the motion estimation device of the present modification. However, the search image setting unit 24, the point-of-interest motion estimation unit 25, and the three-dimensional motion estimation unit 26 of this modification are changed as described later in accordance with the new configuration. Further, with respect to the imaging target detection unit 27 of the present modification, the detection result (the detection result of each imaging target of the estimation target image, the information of the position and relative speed of each imaging target with respect to the own vehicle) is searched for and set as the search image setting unit 24. Send to.

  The road region recognition unit 28, the roadside region recognition unit 29, the vanishing point detection unit 30, and the region setting unit 31 are interposed between the distance information generation unit 23 and the search image setting unit 24. The result is sent to the search image setting unit 24.

  The road area recognition unit 28 recognizes the road area on the estimation target image (in the three-dimensional space when each imaging target is recognized three-dimensionally), and faces the road area to the own lane area. It is divided into lane areas.

  The recognition of the road area may be performed by a well-known method in this technical field. For example, the road area recognition unit 28 can determine the road area on the estimation target image based on the depth distance information of each pixel of the estimation target image received from the distance information generation unit 23. When recognizing the road region, the luminance value information of the estimation target image (luminance image) may be used.

  The own lane region is a region occupied by the lane in which the host vehicle is traveling on the road region of the estimation target image. For example, when an imaging target moving in the same direction as the host vehicle on the road region (that is, an imaging target with a small relative momentum with respect to the host vehicle) is detected, the road region recognition unit 28 determines that the imaging target is the host vehicle. It can be determined that the vehicle is in the same lane as another vehicle or the like, and the region where the imaging target exists can be determined as the own lane region.

  The oncoming lane area is an area occupied by the lane in which the oncoming vehicle travels on the road area of the estimation target image. For example, when an imaging target moving in the opposite direction to the own vehicle (that is, an imaging target having a large relative momentum with respect to the own vehicle) is detected on the road region, the road area recognition unit 28 detects the opposite lane. It is determined that the vehicle is an oncoming vehicle or the like traveling, and the region where the imaging target exists is determined as the oncoming lane region.

  When the road area recognition unit 28 can detect the white line, the road area recognition unit 28 can identify the own lane area and the opposite lane area based on the white line.

  The roadside area recognition unit 29 recognizes the roadside area on the estimation target image (in the three-dimensional space when each imaging target is recognized three-dimensionally). The roadside area is an area that occupies both sides of the road, and is an area where static objects such as sidewalks, buildings, and plants are present. The stationary object such as a sidewalk or a static object has a relative momentum corresponding to the momentum of the own vehicle (that is, the same movement amount as that of the own vehicle in the direction opposite to the movement direction of the own vehicle). For this reason, for example, when an imaging target having a relative momentum corresponding to the momentum of the host vehicle is detected on the estimation target image, the roadside region recognition unit 29 determines the region where the imaging target exists as the roadside region. judge. Here, in this roadside region, the size of the two-dimensional motion vector of each pixel may be estimated based on the speed of the host vehicle. On the estimation target image, the background such as the sky also has a relative momentum corresponding to the momentum of the host vehicle. For this reason. The roadside area recognition unit 29 may recognize the background as an area equivalent to the roadside area.

  The vanishing point detection unit 30 detects the vanishing point on the estimation target image (in the three-dimensional space when each imaging target is recognized three-dimensionally). For example, the vanishing point detection unit 30 is based on the depth distance information and luminance value information of each pixel of the estimation target image, and further based on the recognition result of the road region recognition unit 28 and the road side region recognition unit 29, on the estimation target image. Straight line components such as white lines of roads and ridgelines of roadside area buildings are extracted. And this vanishing point detection part 30 calculates | requires the intersection which extended those linear components, or the point corresponded to the said intersection, and detects this intersection etc. as a vanishing point. The vanishing point detection unit 30 may detect the vanishing point using each captured image obtained in time series. For example, when the imaging device 10 monitors the front of the host vehicle, a vanishing point generally appears at or near the center of the captured image. As described above, the apparent movement between captured images is small at or near the vanishing point.

  The area setting unit 31 detects the position information of the imaging target on the estimation target image detected by the imaging target detection unit 27 and the area recognized by the road area recognition unit 28 and the roadside area recognition unit 29 (opposite the own lane area). Lane area and roadside area), the distance from the vanishing point to each pixel on the estimation target image, the depth distance of each pixel of the estimation target image, and the size of the two-dimensional motion vector of each pixel The estimation target image is divided into a plurality of regions based on at least one of the estimated values. Specifically, based on the distance from the vanishing point of each pixel, the depth distance of each pixel, and the estimated value of the two-dimensional motion vector of each pixel, the area for each imaging target and the own lane area are opposed to each other. The lane area and the roadside area are further divided into the following first to third areas.

  The first region is a distant region with respect to the host vehicle (imaging device 10), and is a vanishing point and a region around it. The area around the vanishing point (that is, the distance from the vanishing point) is, for example, an area in which the apparent movement is smaller than a predetermined value in two captured images that are continuously captured. The predetermined value is, for example, a size that cannot be used to estimate the motion well. The second region is a neighborhood region with respect to the host vehicle (imaging device 10), and is a region whose depth distance to the host vehicle (imaging device 10) is equal to or less than a predetermined distance. The predetermined distance is a distance at which the apparent movement is clear and the movement can be estimated well in two captured images that are successively captured. For example, this second region may be the remaining region excluding the far region on the estimation target image. The third region is a region that is set after the size of the two-dimensional motion vector of the pixel can be estimated in the estimation target image, and is a region where the pixels that can be estimated to be the same imaging target based on the estimated value It is. That is, the third region is a region having an estimated value that is the same size and substantially the same size. When the third area can be specified, even if the third area exists on the first area or the second area, the third area is set as the third area instead of the first area or the second area.

  The search image setting unit 24 of the embodiment sets a search image for each pixel of the estimation target image. On the other hand, the search image setting unit 24 according to the present modification sets the search image for each region of the estimation target image divided into a plurality.

  For example, in the search image setting unit 24, in the case of the first region, the captured image of the image information storage unit 22 that has a longer imaging time than the estimation target image is associated with the first region. It is set as the attached search image. In other words, in the case of the second area, as compared with the first area, the captured image of the image information storage unit 22 whose imaging time is closer to the estimation target image is used as a search image associated with the second area. Let it be set. That is, the search image setting unit 24 associates the captured image of the image information storage unit 22 that is more distant from the estimation target image in the region farther away from the host vehicle with the region. On the other hand, as the area closer to the host vehicle is set as the search image, the captured image of the image information storage unit 22 whose imaging time is closer to the estimation target image is set as the search image associated with the area. To do. In the case of the third region, the search image setting unit 24 sets the search image at the same imaging interval as when the estimated value of the size of the two-dimensional motion vector related to the third region is obtained. That is, in the case of the third area, the captured image of the image information storage unit 22 that is further away from the time of capturing the estimation target image by the imaging interval is set as a search image associated with the third area.

  The search image setting unit 24 is based on the position information of the areas recognized by the road area recognition unit 28 and the roadside area recognition unit 29 (the own lane area, the oncoming lane area, and the roadside area) and the depth distance information of each pixel. A search image may be set as follows. For example, in the case of the own lane region, the search image setting unit 24 sets the image information that is farther from the estimation target image than the estimation target image if the depth distance is the same. The captured image in the storage unit 22 is set as a search image.

  The search image setting unit 24 sends association information between a region set on the estimation target image and the search image to the attention point motion estimation unit 25 for each region.

  The attention point motion estimation unit 25 of the present modification sets an attention point on the estimation target image, and specifies a search image associated with the region including the attention point based on the association information. The attention point motion estimation unit 25 reads the identified search image from the captured image of the image information storage unit 22 and estimates the two-dimensional motion of the attention point. The attention point motion estimation unit 25 does not set pixels of the estimation target image having no association information as attention points.

  The three-dimensional motion estimation unit 26 according to the present modification estimates the three-dimensional motion amount of each pixel of the estimation target image, as in the embodiment. The three-dimensional motion estimation unit 26 smoothes the three-dimensional motion amount for each region set by the search image setting unit 24.

  FIG. 8 shows a flowchart relating to the arithmetic processing operation of this motion estimation apparatus.

  In the present modification, as in the embodiment, information of the captured image (luminance image) is acquired from the imaging device 10 (step ST11), and is temporarily stored in the image information storage unit 22 in time series (step ST12). . Further, in the present modification, a distance image is generated based on the captured image (luminance image) as in the embodiment (step ST13).

  In this modification, the vanishing point detection unit 30 detects a vanishing point on the estimation target image (step ST14), and the region setting unit 31 divides the estimation target image into a plurality of regions (step ST15).

  The search image setting unit 24 sets a search image associated with the area for each area (step ST16).

  The attention point motion estimation unit 25 sets an attention point from among the pixels of the estimation target image (step ST <b> 17), and captures a search image associated with the region including the attention point in the image information storage unit 22. The image is read from the image, and the two-dimensional movement of the attention point is estimated (step ST18).

  The point-of-interest motion estimation unit 25 estimates the motion of all points of interest (that is, pixels excluding those having no association information described above from all the pixels of the estimation target image) on the estimation target image. It is determined whether or not it is finished (step ST19). The attention point motion estimation unit 25 returns to step ST <b> 17 when the estimation of the motions of all the attention points to be estimated has not been completed, and estimates the motion for another attention point. Then, when the estimation of the motion of all the points of interest to be estimated has been completed, the three-dimensional motion estimation unit 26 calculates the three-dimensional momentum of each target point based on the depth distance information and the estimation result of the two-dimensional motion. Is estimated (step ST20). At this time, the three-dimensional motion estimation unit 26 smoothes the three-dimensional momentum for each region.

  The imaging target detection unit 27 classifies each imaging target on the estimation target image based on the depth distance information of each attention point and the estimation result of the three-dimensional momentum of each attention point. Are detected (step ST21). The image processing ECU 120 outputs information on the position and relative speed of each imaging target with respect to the host vehicle to the subsequent control system (step ST22).

  As described above, the motion estimation apparatus according to the present modification divides the estimation target image into a plurality of parts under a predetermined condition, and adaptively sets a search image for each region. For this reason, this motion estimation apparatus can accurately estimate the motion of an imaging target in a distant region with a small apparent motion between captured images with a short imaging interval. In addition, since this motion estimation device smoothes the estimated three-dimensional momentum for each region, it is possible to reduce the variance of motion estimation accuracy. In addition, the motion estimation apparatus can estimate the motion from the information of the surrounding pixels even for pixels for which the motion could not be directly estimated based on the depth distance information by such smoothing. It is possible to reduce the number of pixels that cannot be estimated and obtain a motion estimation result with high density in the region. In addition, this motion estimation device compares the search image for each pixel in the same region when estimating the motion of each pixel in the same region as compared with the case where the search image is switched in units of pixels as in the embodiment. Therefore, it is possible to reduce calculation processing time and calculation cost.

DESCRIPTION OF SYMBOLS 1, 2 Object detection system 10 Imaging device 20,120 Image processing ECU
DESCRIPTION OF SYMBOLS 21 Image information acquisition part 22 Image information storage part 23 Distance information generation part 24 Search image setting part 25 Attention point motion estimation part 26 Three-dimensional motion estimation part 27 Imaging target detection part 28 Road area recognition part 29 Roadside area recognition part 30 Disappearance inspection Outlet 31 Area setting part

Claims (4)

An imaging device that images an imaging region around the host vehicle and outputs a captured image that is an aggregate of pixels including luminance value information in time series;
An image information storage unit for storing information of the captured image in association with an imaging time;
A distance information generator that calculates depth distance information from the imaging device to pixels of the captured image based on luminance value information of the captured image;
As the pixels of the captured image that is the target of motion estimation are located farther from the host vehicle, the captured image of the image information storage unit that is further away from the captured image of the target of motion estimation is the imaging time. While set as a search image associated with the pixel, the closer the pixel of the captured image of the motion estimation target is to the vehicle, the closer the imaging time is to the captured image of the motion estimation target A search image setting unit that sets the captured image of the image information storage unit as a search image associated with the pixel;
A pixel of the captured image of the motion estimation target is set as a point of interest for each pixel, a corresponding point corresponding to the point of interest is searched from the search image associated with the pixel of the point of interest, and the point of interest A motion estimator for estimating motion;
A motion estimation apparatus comprising:   The search image setting unit searches for the captured image of the image information storage unit with a newer imaging time as the estimated value of the size of a two-dimensional motion vector in the pixel of the captured image of the motion estimation target increases. The motion estimation device according to claim 1, wherein the motion estimation device is set to an image. An imaging device that images an imaging region around the host vehicle and outputs a captured image that is an aggregate of pixels including luminance value information in time series;
An image information storage unit for storing information of the captured image in association with an imaging time;
A distance information generator that calculates depth distance information from the imaging device to pixels of the captured image based on luminance value information of the captured image;
An area setting unit that divides the captured image of the motion estimation target into a plurality of areas;
The search image that associates the captured image of the image information storage unit that is more distant from the captured image of the motion estimation target in the region farther away from the host vehicle with the region. On the other hand, the search that associates the captured image of the image information storage unit with the region closer to the motion estimation target in the image information storage unit, the closer to the region that is present near the host vehicle A search image setting unit to be set as an image;
A pixel of the captured image of the motion estimation target is set as a point of interest for each pixel, and a corresponding point corresponding to the point of interest is searched from the search image associated with the region including the point of interest. A motion estimation unit for estimating the motion of the attention point;
A motion estimation apparatus comprising:   The area setting unit includes position information of an imaging target on the captured image of the motion estimation target, position information of an area obtained by dividing the own lane, the oncoming lane, and the roadside on the captured image of the motion estimation target, and the motion The distance from the vanishing point to each pixel on the captured image of the estimation target, the depth distance of each pixel of the captured image of the motion estimation target, and the magnitude of the two-dimensional motion vector of the pixel of the captured image of the motion estimation target The motion estimation apparatus according to claim 3, wherein the captured image to be the motion estimation target is divided into a plurality of regions based on at least one of the estimated values.

Images