JP6972797B2 - Information processing device, image pickup device, device control system, mobile body, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an image pickup device, a device control system, a mobile body, an information processing method, and a program.

In terms of automobile safety, the body structure of automobiles has been developed from the viewpoint of how to protect pedestrians and occupants when they collide with pedestrians or automobiles. However, in recent years, with the development of information processing technology and image processing technology, technology for detecting people, automobiles, and the like at high speed has been developed. Automobiles that apply these technologies to automatically apply the brakes before a collision to prevent a collision have already been released.

In order to automatically apply the brakes, it is necessary to measure the distance to an object such as a person or another vehicle, and for that purpose, measurement using an image of a stereo camera has been put into practical use (see, for example, Patent Document 1). ..

In the measurement using the image of this stereo camera, an object such as a vehicle in front of the own vehicle is detected in the parallax image of a certain frame, and then the object is tracked in the parallax image of the subsequent frames. The technology is known.

However, in the prior art, for example, one object detected in the previous frame and one object detected in the current frame due to an error due to a change in ambient illuminance, a change in the moving speed or a moving direction of the object, etc. There is a problem that other objects may be erroneously determined to be the same object. Also, for example, even when pedestrians cross each other or multiple vehicles of the same color and shape are running side by side, one object detected in the previous frame and this frame There is a problem that other detected objects may be erroneously determined to be the same object.

Therefore, it is an object of the present invention to provide a technology capable of continuing highly accurate tracking.

An acquisition unit that acquires information in which a vertical position, a horizontal position, and a depth position of an object are associated with each other at a plurality of time points, and a predetermined value in the information previously acquired by the acquisition unit. Based on the position of the object, the prediction unit that predicts the position of the predetermined object in the information acquired this time by the acquisition unit, and the predetermined condition according to the position of the predetermined object from the information this time. A plurality of objects to be satisfied are extracted, and among the plurality of objects in the present information, the predetermined object in the previous information is based on the similarity between each image of the plurality of objects and the image of the predetermined object. The plurality of objects satisfy the predetermined conditions according to the position of the predetermined object, and the predetermined object is calculated by the prediction unit. An information processing apparatus in which the plurality of objects are located within a predetermined distance according to the type of the predetermined object from the position of.

According to the disclosed technology, it is possible to continue highly accurate tracking.

It is a figure which shows the structure of the device control system which concerns on embodiment. It is a figure which shows the structure of the image pickup unit and the image analysis unit which concerns on embodiment. It is a figure for demonstrating the principle of calculating a distance from a parallax value by using the principle of triangulation. It is a figure which shows an example of the functional block diagram of a device control system. It is a figure for demonstrating the parallax image data, and the V map generated from the parallax image data. It is a figure which shows the image example of the photographed image as a reference image imaged by one image pickup part, and the V map corresponding to the photographed image. It is a figure which shows the image example which represented the example of the reference image schematically. It is a figure which shows the U map corresponding to the image example. It is a figure which shows the real U map corresponding to the U map. It is a figure for demonstrating the method of obtaining the value of the horizontal axis of a real U map from the value of the horizontal axis of a U map. It is a flowchart which shows an example of the isolated area detection process. It is a figure which shows the real frequency U map which set the rectangular area inscribed by the isolated area detected by the isolated area detection part. It is a figure which shows the parallax image which set the scanning range corresponding to a rectangular area. It is a figure which shows the parallax image which searched the scanning range and set the object area. It is a flowchart which shows the flow of the process performed in the corresponding area detection part and the object area extraction part of a parallax image. It is a figure which shows an example of the table data for classifying an object type. It is a flowchart which shows an example of the object tracking process. It is a flowchart which shows the detailed example of the object tracking process. It is a figure explaining the range of the actual distance according to the object type of the target object. It is a flowchart which shows the other detailed example of the object tracking process. It is a figure explaining the degree of overlap.

Hereinafter, a device control system having an image processing device according to the embodiment will be described.

<Device control system configuration>
FIG. 1 is a diagram showing a configuration of a device control system according to an embodiment.

This device control system 1 is mounted on an own vehicle 100 such as a moving vehicle, and includes an image pickup unit 101, an image analysis unit 102, a display monitor 103, and a vehicle travel control unit 104. Then, the imaging unit 101 detects and tracks an object in front of the own vehicle from a plurality of captured image data (frames) in the front region (imaging region) in the traveling direction of the own vehicle, which is an image of the front of the moving body, and the tracking result. Is used to control mobile objects and various in-vehicle devices. Control of the moving body includes, for example, warning notification, control of the steering wheel of the own vehicle 100 (self-moving body), or braking of the own vehicle 100 (self-moving body).

The image pickup unit 101 is installed near, for example, a rearview mirror (not shown) of the windshield 105 of the own vehicle 100. Various data such as captured image data obtained by the imaging of the imaging unit 101 are input to the image analysis unit 102 as an image processing means.

The image analysis unit 102 analyzes the data transmitted from the image pickup unit 101, and on the traveling road surface in front of the own vehicle with respect to the road surface portion (the road surface portion located directly under the own vehicle) on which the own vehicle 100 is traveling. The relative height (position information) at each point is detected, and the three-dimensional shape of the traveling road surface in front of the own vehicle is grasped. It also recognizes objects to be recognized such as other vehicles, pedestrians, and various obstacles in front of the own vehicle.

The analysis result of the image analysis unit 102 is sent to the display monitor 103 and the vehicle travel control unit 104. The display monitor 103 displays the captured image data and the analysis result obtained by the imaging unit 101. The display monitor 103 may not be provided. The vehicle travel control unit 104 notifies, for example, a warning to the driver of the own vehicle 100 based on the recognition result of the recognition target object such as another vehicle, a pedestrian, and various obstacles in front of the own vehicle by the image analysis unit 102. It also performs driving support control such as controlling the steering wheel and brakes of the own vehicle.

<Structure of image pickup unit 101 and image analysis unit 102>
FIG. 2 is a diagram showing a configuration of an image pickup unit 101 and an image analysis unit 102 according to an embodiment.

The image pickup unit 101 is composed of a stereo camera provided with two image pickup units 110a and 110b as image pickup means, and the two image pickup units 110a and 110b are the same. Each of the image pickup units 110a and 110b is an analog output from the sensor boards 114a and 114b including the image pickup lenses 111a and 111b and the image sensors 113a and 113b in which the light receiving elements are two-dimensionally arranged, and the sensor boards 114a and 114b, respectively. It is composed of signal processing units 115a and 115b that generate and output captured image data obtained by converting an electric signal (an electric signal corresponding to the amount of light received by each light receiving element on the image sensors 113a and 113b) into a digital electric signal. ing. Luminance image data and parallax image data are output from the image pickup unit 101.

Further, the image pickup unit 101 includes a processing hardware unit 120 including an FPGA (Field-Programmable Gate Array) or the like. In order to obtain a parallax image from the brightness image data output from the imaging units 110a and 110b, the processing hardware unit 120 determines the parallax value of the corresponding image portion between the captured images captured by the imaging units 110a and 110b, respectively. A parallax calculation unit 121 is provided as a parallax image information generation means for calculation.

The parallax value referred to here is a comparison image with respect to an image portion on the reference image corresponding to the same point in the imaging region, with one of the captured images captured by each of the imaging units 110a and 110b as a reference image and the other as a comparison image. The amount of positional deviation of the upper image portion is calculated as a parallax value of the image portion. By using the principle of triangulation, it is possible to calculate the distance from this parallax value to the same point in the imaging region corresponding to the image portion.

FIG. 3 is a diagram for explaining the principle of calculating the distance from the parallax value by using the principle of triangulation. In the figure, f is the focal length of each of the image pickup lenses 111a and 111b, and D is the distance between the optical axes. Further, Z is a distance (distance in a direction parallel to the optical axis) from the image pickup lenses 111a and 111b to the subject 301. In this figure, the image formation positions in the left and right images with respect to the point O on the subject 301 have distances from the image formation center of Δ1 and Δ2, respectively. The parallax value d at this time can be defined as d = Δ1 + Δ2.

Returning to the description of FIG. The image analysis unit 102 includes a storage means 122 composed of an image processing board or the like and including a RAM or ROM for storing brightness image data and parallax image data output from the image pickup unit 101, and recognition processing of an identification target. It includes a CPU (Central Processing Unit) 123 that executes a computer program for performing disparity calculation control and the like, a data I / F (interface) 124, and a serial I / F 125.

The FPGA constituting the processing hardware unit 120 performs processing that requires real-time performance for image data, for example, gamma correction, distortion correction (parallelization of left and right captured images), and parallax calculation by block matching to perform a parallax image. Information is generated and written to the RAM of the image analysis unit 102. The CPU of the image analysis unit 102 is responsible for controlling the image sensor controllers of the image pickup units 110A and 110B and overall control of the image processing board, as well as detecting the three-dimensional shape of the road surface, guard rails, and various other objects (objects). A program that executes detection processing, etc. is loaded from the ROM, various processes are executed by inputting the brightness image data and the parallax image data stored in the RAM, and the processing results are obtained from the data I / F 124 or serial I / F 125. Output to the outside. When executing such processing, the data I / F124 is used to input vehicle operation information such as vehicle speed, acceleration (mainly acceleration generated in the front-rear direction of the own vehicle), steering angle, yaw rate, etc. of the own vehicle 100, and various types are input. It can also be used as a processing parameter. The data output to the outside is used as input data for controlling various devices of the own vehicle 100 (brake control, vehicle speed control, warning control, etc.).

The image pickup unit 101 and the image analysis unit 102 may be configured as an image pickup device 2 which is an integrated device.

<Object detection processing>
Next, with reference to FIG. 4, a function for performing object detection processing realized by the processing hardware unit 120 and the image analysis unit 102 in FIG. 2 will be described. FIG. 4 is a diagram showing an example of a functional block diagram of the device control system 1. Hereinafter, the object detection process in this embodiment will be described.

Luminance image data is output from the two image pickup units 110a and 110b constituting the stereo camera. At this time, when the imaging units 110a and 110b are in color, color luminance conversion for obtaining a luminance signal (Y) from the RGB signal is performed using, for example, the following equation [1].

Y = 0.3R + 0.59G + 0.11B ... Equation [1]
<< Parallax image generation processing >>
Next, in the parallax image generation unit 132 configured by the parallax calculation unit 121, the parallax image data (parallax image information. “The vertical position of the detection object, the horizontal position, and the depth direction correspond to each other. An example of "attached information".) Performs a parallax image generation process to generate). In the parallax image generation process, first, the brightness image data of one of the two image pickup units 110a and 110b is used as the reference image data, and the brightness image data of the other image pickup unit 110b is used as the comparison image data. The difference between the two is calculated using the difference, and the difference image data is generated and output. This parallax image data shows a parallax image in which the pixel value corresponding to the parallax value d calculated for each image portion on the reference image data is represented as the pixel value of each image portion.

<< V map generation process >>
Next, in the V map generation unit 134, the parallax image data is acquired from the parallax image generation unit 132, and the V map generation process for generating the V map is executed. Each parallax pixel data included in the parallax image data is represented by a set (x, y, d) of the x-direction position, the y-direction position, and the parallax value d. This is converted into three-dimensional coordinate information (d, y, f) in which d is set on the X-axis, y is set on the Y-axis, and frequency f is set on the Z-axis, or the three-dimensional coordinate information (d, y, f). Three-dimensional coordinate information (d, y, f) limited to information exceeding a predetermined frequency threshold is generated as disparity histogram information. The disparity histogram information of the present embodiment consists of three-dimensional coordinate information (d, y, f), and the three-dimensional histogram information distributed in the XY two-dimensional coordinate system is a V map (distortion histogram map). , V-disparity map).

Specifically, the V-map generation unit 134 calculates the parallax value frequency distribution for each row region of the parallax image data obtained by dividing the image into a plurality of parts in the vertical direction. The information indicating the parallax value frequency distribution is the parallax histogram information.

FIG. 5 is a diagram for explaining the parallax image data and the V map generated from the parallax image data. Here, FIG. 5A is a diagram showing an example of the parallax value distribution of the parallax image, and FIG. 5B is a diagram showing a V map showing the parallax value frequency distribution for each row of the parallax image of FIG. 5A.

When the parallax image data having the parallax value distribution as shown in FIG. 5A is input, the V-map generation unit 134 calculates the parallax value frequency distribution which is the distribution of the number of data of each parallax value for each row. This is output as parallax histogram information. The information of the parallax value frequency distribution of each row obtained in this way is on a two-dimensional Cartesian coordinate system in which the y-direction position (vertical position of the captured image) on the parallax image is taken on the Y-axis and the parallax value is taken on the X-axis. By representing in, a V map as shown in FIG. 5B can be obtained. This V-map can also be expressed as an image in which pixels having pixel values corresponding to the frequency f are distributed on the two-dimensional Cartesian coordinate system.

FIG. 6 is a diagram showing an image example of a captured image as a reference image captured by one of the imaging units and a V map corresponding to the captured image. Here, FIG. 6A is a photographed image, and FIG. 6B is a V map. That is, the V map shown in FIG. 6B is generated from the captured image as shown in FIG. 6A.

In the image example shown in FIG. 6A, the road surface 401 on which the own vehicle is traveling, the preceding vehicle 402 existing in front of the own vehicle, and the utility pole 403 existing outside the road are projected. Further, in the V map shown in FIG. 6B, there are a road surface 501, a preceding vehicle 502, and a utility pole 503 corresponding to an image example.

<< Road surface shape detection processing >>
Next, in the present embodiment, the road surface shape detection unit 135 detects the three-dimensional shape of the road surface in front of the own vehicle 100 from the V map information (parallax histogram information) generated by the V map generation unit 134. The process is executed.

In the image example shown in FIG. 6A, the road surface in front of the own vehicle 100 is relatively flat, that is, the surface in which the front road surface of the own vehicle 100 is parallel to the road surface portion directly under the own vehicle 100 is extended to the front of the own vehicle. This is the case when it matches the virtual reference road surface (virtual reference moving surface) obtained. In this case, in the lower part of the V map corresponding to the lower part of the image, the high frequency points (road surface 501) are distributed in a substantially straight line with an inclination so that the parallax value d becomes smaller toward the upper part of the image. Pixels showing such a distribution are pixels that are present at almost the same distance in each row on the parallax image, have the highest occupancy rate, and project a detection target whose distance is continuously increased toward the upper part of the image. It can be said that.

Since the image pickup unit 110A captures the area in front of the vehicle, the parallax value d of the road surface becomes smaller toward the upper side of the image as shown in FIG. 6B. Further, in the same line (horizontal line), the pixels projecting the road surface have substantially the same parallax value d. Therefore, the high-frequency points (road surface 501) distributed in a substantially straight line on the V-map correspond to the characteristics of the pixels that project the road surface (moving surface). Therefore, it can be estimated that the pixels of the points distributed on or near the approximate straight line obtained by linearly approximating the high frequency points on the V map are the pixels projecting the road surface with high accuracy. Further, the distance to the road surface portion projected on each pixel can be obtained with high accuracy from the parallax value d of the corresponding point on the approximate straight line. Since the height of the road surface can be obtained by estimating the road surface, the height of the object on the road surface can be obtained. This can be calculated by a known method. For example, a linear equation representing the estimated road surface is obtained, and the corresponding y-coordinate y0 when the parallax value d = 0 is set as the height of the road surface. Then, for example, when the parallax value is d and the y coordinate is y', the height from the road surface when y'−y0 is the parallax value d is shown. The height H of the above-mentioned coordinates (d, y') from the road surface can be obtained by the arithmetic expression H = (z × (y'−y0)) / f. In this calculation formula, "z" is the distance calculated from the parallax value d (z = BF / (d-offset)), and "f" is the focal length of the imaging units 10a and 10b (y'-y0). It is a value converted to the same unit as the unit of. Here, BF is a value obtained by multiplying the baseline length B of the imaging units 10a and 10b by the focal length f, and offset is the parallax when an object at infinity is photographed.

<< U map generation process >>
Next, the U map generation unit 137 executes a frequency U map generation process and a height U map generation process as the U map generation process for generating the U map (U-disparity map).

In the frequency U map generation process, the set (x, y, d) of the x-direction position, the y-direction position, and the difference value d in each difference pixel data included in the difference image data is x on the X-axis and d on the Y-axis. , Set the frequency on the Z axis and create XY two-dimensional histogram information. This is called a frequency U map. In the U map generation unit 137 of the present embodiment, the frequency U map is created only for the points (x, y, d) of the parallax image in which the height H from the road surface is in a predetermined height range (for example, 20 cm to 3 m). .. In this case, an object existing in the predetermined height range can be appropriately extracted from the road surface.

Further, in the height U map generation process, the set (x, y, d) of the x-direction position, the y-direction position, and the difference value d in each difference pixel data included in the difference image data is x, Y on the X-axis. Set the d on the axis and the height from the road surface on the Z axis to create XY two-dimensional histogram information. This is called a height U map. The height value at this time is the highest from the road surface.

FIG. 7 is an image example schematically showing an example of a reference image captured by the image pickup unit 110a, and FIG. 8 is a U map corresponding to the image example of FIG. 7. Here, FIG. 8A is a frequency U map, and FIG. 8B is a height U map.

In the image example shown in FIG. 7, guardrails 413 and 414 are present on both the left and right sides of the road surface, and as other vehicles, one preceding vehicle 411 and one oncoming vehicle 412 are present. At this time, in the frequency U map, as shown in FIG. 8A, the high frequency points corresponding to the left and right guardrails 413 and 414 are substantially linear 603,604 extending upward from both left and right ends toward the center. It is distributed in. On the other hand, the high-frequency points corresponding to the preceding vehicle 411 and the oncoming vehicle 412 are distributed between the left and right guardrails in the states of lines 601 and 602 extending in parallel in the substantially X-axis direction. In a situation where the side surface portions of these vehicles are projected in addition to the rear portion of the preceding vehicle 411 or the front portion of the oncoming vehicle 412, the parallax is generated in the image area in which the same other vehicle is projected. Occurs. In such a case, as shown in FIG. 8A, the high frequency points corresponding to other vehicles are in a state where a line segment extending parallel to the substantially X-axis direction and a line segment inclined with respect to the substantially X-axis direction are connected. The distribution of is shown.

Further, in the height U map, points having the highest height from the road surface in the left and right guardrails 413 and 414, the preceding vehicle 411, and the oncoming vehicle 412 are distributed in the same manner as in the frequency U map. Here, the height of the point distribution 701 corresponding to the preceding vehicle and the point distribution 702 corresponding to the oncoming vehicle is higher than the height of the point distribution 703 and 704 corresponding to the guardrail. Thereby, the height information of the object in the height U map can be used for the object detection.

<< Real U map generation process >>
Next, the real U map generation unit 138 will be described. In the real U map generation unit 138, as the U map generation process for generating a real U-disparity map (an example of "distribution data"), a real frequency U map generation process and a real height U map generation process are performed. Run.

In the real U map, the horizontal axis in the U map is converted from the pixel unit of the image to the actual distance, and the parallax value on the vertical axis is converted into the thinned parallax having the thinning rate according to the distance.

In the real frequency U-map generation process, the real U-map generation unit 138 sets a set (x, y, d) of the x-direction position, the y-direction position, and the difference value d in each difference pixel data included in the difference image data. Two-dimensional histogram information of XY is created by setting the actual distance in the horizontal direction on the X-axis, the thinning difference on the Y-axis, and the frequency on the Z-axis. The real U-map generation unit 138 of the present embodiment, like the U-map generation unit 137, is only for the points (x, y, d) of the parallax image in which the height H from the road surface is within a predetermined height range. Create a real frequency U map. The real U map generation unit 138 may be configured to generate a real U map based on the U map generated by the U map generation unit 137.

FIG. 9 is a diagram showing a real U map (hereinafter referred to as a real frequency U map) corresponding to the frequency U map shown in FIG. 8A. As shown in the figure, the left and right guardrails are represented by vertical linear patterns 803,804, and the preceding vehicle and the oncoming vehicle are also represented by patterns 801 and 802 close to the actual shape.

The thinning parallax on the vertical axis is not thinned out for long distances (here, 50 m or more), thinned out to 1/2 for medium distances (20 m or more and less than 50 m), and 1 / for short distances (10 m or more and less than 20 m). It is thinned out to 3 and 1/8 for short distances (10 m or more and less than 20 m).

In other words, the farther away, the smaller the amount of thinning. The reason is that the object is small in the distance, so the parallax data is small, and the distance resolution is small, so the thinning is small. To increase.

An example of a method of converting the horizontal axis from a pixel unit of an image to an actual distance and a method of obtaining (X, d) of a real U map from (x, d) of a U map will be described with reference to FIG.

The width of 10 m on each side when viewed from the camera, that is, 20 m is set as the object detection range. Assuming that the width of one pixel in the horizontal direction of the real U map is 10 cm, the horizontal size of the real U map is 200 pixels.

Let f be the focal length of the camera, p be the lateral position of the sensor from the center of the camera, Z be the distance from the camera to the subject, and X be the lateral position from the center of the camera to the subject. Assuming that the pixel size of the sensor is s, the relationship between x and p is represented by "x = p / s". Further, due to the characteristics of the stereo camera, there is a relationship of "Z = Bf / d".

Further, since there is a relationship of "x = p * Z / f" from the figure, it can be represented by "X = sxB / d". Although X is an actual distance, since the width of one pixel in the horizontal direction on the real U map is 10 cm, the position of X on the real U map can be easily calculated.

A real U map (hereinafter referred to as a real height U map) corresponding to the height U map shown in FIG. 8B can also be created by the same procedure.

The real U map has the advantage that the processing speed can be increased because the vertical and horizontal lengths can be made smaller than the U map. In addition, since the horizontal direction is independent of the distance, it is possible to detect the same object with the same width both in the distance and in the vicinity. There is also an advantage that (width threshold processing) becomes easy.

The vertical length in the U-map is determined by how many meters the shortest measurable distance is. That is, since "d = Bf / Z", the maximum value of d is determined according to the shortest measurable Z. Further, since the parallax value d handles a stereo image, it is usually calculated in pixel units, but since it includes a small number, a parallax value obtained by multiplying the parallax value by a predetermined value and rounding off a decimal part is used.

When the shortest measurable Z is halved, d is doubled, so the U-map data becomes huge. Therefore, when creating a real U-map, the closer the distance is, the more pixels are thinned out to compress the data, and the amount of data is reduced as compared with the U-map.
Therefore, object detection by labeling can be performed at high speed.

《Isolated area detection》
Next, the isolated area detection process performed by the isolated area detection unit 139 will be described. FIG. 11 is a flowchart showing an example of the isolated area detection process. The isolated area detection unit 139 first smoothes the information of the real frequency U map generated by the real U map generation unit 138 (step S111).

This is to make it easier to detect effective isolated areas by averaging the frequency values. That is, since the parallax value has a variance due to a calculation error and the like, and the parallax value is not calculated for all the pixels, the real U map is different from the schematic diagram shown in FIG. 9 and has noise. Includes. Therefore, in order to remove noise and to make it easier to separate the object to be detected, the real U map is smoothed. Similar to image smoothing, this can be considered as noise by applying a smoothing filter (for example, a simple average of 3 x 3 pixels) to the frequency value of a real U map (real frequency U map). The frequency is reduced, and the frequency is higher than the surroundings in the object part, forming a cohesive group, which has the effect of facilitating the isolated area detection process in the subsequent stage.

Next, the threshold value for binarization is set (step S112). Initially, the smoothed real U map is binarized using a small value (= 0) (step S113). After that, the coordinate with a value is labeled to detect the isolated region (step S114).

In these two steps, the real frequency U-map detects isolated areas (called islands) whose frequency is higher than their surroundings. For detection, the real frequency U map is first binarized (step S113). Initially, binarization is performed with a threshold value of 0. This is a countermeasure against the fact that some islands are isolated and some are connected to other islands because of the height of the object, its shape, and separation from the road surface parallax. That is, by binarizing the real frequency U map from a small threshold, first isolated islands of appropriate size are detected, and then by increasing the threshold, the connected islands are separated and isolated. It made it possible to detect the island as an island of appropriate size.

Labeling is used as a method for detecting the binarized islands. Coordinates that are black after binarization (coordinates whose frequency value is higher than the binarization threshold) are labeled based on their connectivity, and the areas with the same label are designated as islands.

The size of each of the detected isolated regions is determined (step S115). This is because the detection target is from a pedestrian to a large vehicle, so it is determined whether or not the width of the isolated region is within the range of the size. If the size is large (step S115: YES), the binarization threshold is incremented by 1 (step S112), and binarization is performed only within the isolated region of the real frequency U map (step S113). Then, labeling is performed, a smaller isolated region is detected (step S114), and the size thereof is determined (step S115).

The labeling process is repeated from the above threshold setting to detect an isolated region of a desired size. If an isolated region of a desired size can be detected (step S115: NO), then peripheral region removal is performed (step S116). This is a distant object, the accuracy of road surface detection is poor, the parallax of the road surface is introduced in the real U map, and the height of the left, right, and near when the parallax between the object and the road surface is detected as a mass. Is a process of deleting the area near the road surface (peripheral part in the isolated area). If the removed area exists (step S117: YES), labeling is performed again to reset the isolated area (step S114).

<< Detection of corresponding area of parallax image and extraction of object area >>
Next, the corresponding area detection unit 140 and the object area extraction unit 141 of the parallax image will be described. FIG. 12 is a diagram showing a real frequency U map in which a rectangular area inscribed by the isolated area detected by the isolated area detection unit is set, and FIG. 13 is a parallax in which a scanning range corresponding to the rectangular area in FIG. 12 is set. It is a figure which shows the image, and FIG. 14 is a figure which shows the parallax image which searched the scanning range in FIG. 13 and set the object area.

Regarding the isolated area determined as the object candidate area by the isolated area detection unit 139, as shown in FIG. 12, the first vehicle 801 as the isolated area, the first detection island 811 and the first detection island 811 as the rectangular area inscribed by the second vehicle 802. When the second detection island 812 is set, the width of this rectangular region (length in the X-axis direction on the U map) corresponds to the width of the identification object (object) corresponding to the isolated region. Further, the height of the set rectangular area corresponds to the depth (length in the traveling direction of the own vehicle) of the identification object (object) corresponding to the isolated area. On the other hand, the height of the identification object (object) corresponding to each isolated area is unknown at this stage. The corresponding area detection unit 140 of the parallax image detects the corresponding area on the parallax image corresponding to the isolated area in order to obtain the height of the object corresponding to the isolated area related to the object candidate area.

The corresponding area detection unit 140 of the disparity image is the position, width and minimum of the first detection island 811 and the second detection island 812 islands detected from the real U map based on the information of the isolated area output from the isolated area detection unit 139. From the parallax, the x-direction range (xmin, xmax) of the first detection island corresponding region scanning range 481 and the second detection island corresponding region scanning range 482, which are the ranges to be detected in the parallax image shown in FIG. 13, can be determined. Further, from the height and position of the object in the parallax image (y-coordinate corresponding to the maximum height from the road surface at the time of maximum parallax dmax), ymax = "y indicating the height of the road surface obtained from the maximum parallax dmax". Up to) can be determined.

Next, in order to detect the exact position of the object, a pixel that scans the set scanning range and has a value in the range of the rectangular depth (minimum parallax dmin, maximum parallax dmax) detected by the isolated area detection unit 139 as the parallax. Is extracted as a candidate pixel. Then, among the extracted candidate pixel groups, a line having a predetermined ratio or more in the lateral direction with respect to the detection width is defined as an object candidate line.

Next, it scans vertically, and if there are other object candidate lines around a certain object candidate line of interest at a predetermined density or higher, the object candidate line of interest is determined as an object line. ..

Next, the object area extraction unit 141 searches for an object line in the search area of the parallax image, determines the lowermost and uppermost ends of the object line, and as shown in FIG. 14, the circumscribing rectangle 461 of the object line group. 462 is determined as the regions 451 and 452 of the objects (first vehicle, second vehicle) in the parallax image.

FIG. 15 is a flowchart showing the flow of processing performed by the corresponding area detection unit 140 and the object area extraction unit 141 of the parallax image. First, the search range in the x-axis direction with respect to the parallax image is set from the position, width, and minimum parallax of the island in the real U map (step S161).

Next, the maximum search value ymax in the y-axis direction with respect to the parallax image is set from the relationship between the maximum parallax dmax of the island and the road surface height (step S162). Next, the minimum search value ymin in the y-axis direction for the parallax image is obtained and set from the maximum height of the island in the real height U map and ymax and dmax set in step S172, so that the y-axis for the parallax image is set. The search range of the direction is set (step S163).

Next, the parallax image is searched in the set search range, and the pixels within the range of the minimum parallax dmin and the maximum parallax dmax of the island are extracted and used as object candidate pixels (step S164). When the object candidate pixels are at a certain ratio or more in the horizontal direction, the line is extracted as an object candidate line (step S165).

The density of the object candidate line is calculated, and if the density is larger than a predetermined value, the line is determined as the object line (step S166). Finally, the circumscribed rectangle of the object line group is detected as the object area in the parallax image (step S167).

Thereby, the identification object (object, object) can be recognized.

《Object type classification》
Next, the object type classification unit 142 will be described.

From the height (yomax-yomin) of the object area extracted by the object area extraction unit 141, the identification object (object) projected on the image area corresponding to the object area is obtained from the following equation [2]. The actual height Ho can be calculated. However, "zo" is the distance between the object corresponding to the object area calculated from the minimum parallax value d in the object area and the own vehicle, and "f" is the focal length of the camera (yomax-yomin). It is a value converted to the same unit as the unit of.

Ho = zo × (yomax-yomin) / f ... Equation [2]
Similarly, from the width (xomax-xomin) of the object area extracted by the object area extraction unit 141, the identification object (object) projected in the image area corresponding to the object area from the following equation [3]. The actual width Wo of can be calculated.

Wo = zo × (xomax-xomin) / f ... Equation [3]
Further, the depth Do of the identification object (object) projected in the image area corresponding to the object area is calculated by the following equation [4] from the maximum parallax dmax and the minimum parallax dmin in the isolated area corresponding to the object area. ] Can be calculated.

Do = BF × {(1 / (dmin-offset) -1 / (dmax-offset)} ... Equation [4]
The object type classification unit 142 classifies the object type from the height, width, and depth information of the object corresponding to the object area that can be calculated in this way. The table shown in FIG. 16 shows an example of table data for classifying object types. In the example of FIG. 16, for example, if the width is less than 1100 mm, the height is less than 250 mm, and the depth is more than 1000 mm, it is determined to be a "motorcycle, bicycle". If the width is less than 1100 mm, the height is less than 250 mm, and the depth is 1000 mm or less, it is determined to be a "pedestrian". According to this, it is possible to distinguish whether the identification object (object) existing in front of the own vehicle is a pedestrian, a bicycle or a motorcycle, a small vehicle, a truck, or the like.

<< 3D position determination >>
Next, the processing of the three-dimensional positioning unit 143 will be described. The three-dimensional position determination unit 143 determines the relative three-dimensional position of the identification object (object) with respect to the own vehicle 100.

The three-dimensional positioning unit 143 is a three-dimensional object based on the distance to the object corresponding to the detected object area and the distance on the image between the image center of the parallax image and the center of the object area on the parallax image. The center position in the coordinates (real space) is calculated by, for example, the following formula.

When the center coordinates of the object area on the disparity image are (region_centerX, region_centerY) and the image center coordinates of the disparity image are (image_centerX, image_centerY), the laterality of the identification object (object) relative to the image pickup units 110a and 110b. The center position Xo in the direction and the center Yo position in the height direction can be calculated from the following equations [5] and [6].

Xo = Z × (region_centerX-image_centerX) / f ... Equation [5]
Yo = Z × (region_centerY-image_centerY) / f ... Equation [6]
《Object Tracking》
Next, the object tracking unit 144 will be described. The object tracking unit 144 executes a process of tracking an object (object) detected from a previous (past) frame in the current (current) frame.

The object tracking unit 144 has a prediction unit 1441 and an extraction unit 1442.

The prediction unit 1441 predicts the position of the predetermined object in the current frame based on the position of the predetermined object detected based on the previous frame. Specifically, the prediction unit 1441 identifies the relative movement speed and movement direction between the object and the own vehicle 100 by using the positions of the objects in the previous plurality of frames, and the movement speed and the movement direction. Based on, the position of each object with respect to the parallax image of this frame is predicted. A known technique can be applied to this object tracking process.

The extraction unit 1442 extracts a plurality of objects satisfying the conditions relating to the position predicted by the prediction unit 1441 from this frame. Further, the extraction unit 1442 extracts the same object as the predetermined object from the plurality of objects based on the similarity between each image of the plurality of objects and the image of the predetermined object.

Next, with reference to FIG. 17, the object tracking process by the object tracking unit 144 will be described. FIG. 17 is a flowchart showing an example of the object tracking process. Note that FIG. 17 describes the processing of one object (target object) among the objects extracted by the object area extraction unit 141. Therefore, the process of FIG. 17 is performed on each object extracted by the object area extraction unit 141.

In step S201, the prediction unit 1441 of the object tracking unit 144 predicts the position of the target object with respect to the current frame based on the position of the target object determined by the three-dimensional position determination unit 143 based on the previous plurality of frames. .. For example, the prediction unit 1441 uses the difference between the position of the target object in the previous frame and the position of the target object in the previous frame as the relative speed with respect to the own vehicle. Then, the position moved by the relative speed from the position of the target object in the previous frame is predicted to be the position of the target object in the current frame. That is, for example, when the relative velocity is 1 m / frame in the lateral direction (X-axis direction) and 2 m / frame in the depth direction (Y-axis direction) on the real U map, the relative velocity is moved 1 m in the lateral direction and 2 m in the depth direction. The position is predicted to be the position of the target object in this frame. In addition, other known techniques may be used for the prediction of the position in step S201.

Subsequently, the extraction unit 1442 of the object tracking unit 144 extracts one or more objects whose positions predicted for the target object in step S201 and the positions detected based on the current frame satisfy a predetermined condition ( Step S202). The threshold value for whether or not the predetermined condition is satisfied may be changed, for example, according to the setting. Further, the extraction unit 1442 satisfies the predetermined condition, for example, when the number of objects detected based on the current frame is a predetermined number or more, or when the number of objects being tracked is a predetermined number or more. The threshold value may be changed so as to be difficult. As a result, for example, when traveling on a road where a relatively large number of obstacles are detected, one object detected in the current frame is the same as another object detected in the previous frame. It is possible to reduce the possibility of being erroneously determined.

Subsequently, the extraction unit 1442 of the object tracking unit 144 extracts an object that matches the target object from the extracted objects based on the characteristics of the target object and the characteristics of each extracted object (step S203). .. As a result, the same object is grasped as the same object in a plurality of frames.

In step S203, for example, an image feature amount may be used as the feature of the object. In this case, for example, a known method such as a higher-order local autocorrelation (HLAC) or a HOG (Histogram of Oriented Gradient) may be used to calculate the feature amount of the image. ..

HLAC extends the autocorrelation of the function g (x, y), which represents the shading of an image, to the Nth order, and g (x, y) g (x + a1, y + b1) ‥ g (x + aN, y + This is a method of extracting the features of an image by limiting the displacements ai and bi of bN) to the local region around the reference pixel (x, y). When the degree of correlation is up to the second order (three-point correlation) and the displacement is limited to the local area of 3 × 3 pixels, in the case of a black-and-white binary image, a 25-dimensional pattern is obtained, and the pixel value indicated by this 25-dimensional pattern is obtained. Is the feature amount of the image.

HOG is a method of calculating an orientation histogram for each block region as a feature amount for an edge image calculated from an input image.

The extraction unit 1442 calculates the similarity between the images by comparing the feature amount of the image of the target object in the previous frame with the feature amount of the image of each extracted object. For example, the Euclidean distance between the 25-dimensional feature quantities of the two images calculated using HLAC or the like is calculated as the degree of similarity between the two images. Then, the extraction unit 1442 determines, for example, that the object having the highest degree of similarity is the target object in the current frame.

As the feature of the object, not only the feature amount of the image but also the position of the object may be used. For example, a score may be used in which the value becomes higher as the feature amount of the image is closer to the target object and the position of the object is closer to the predicted position.

Further, as a feature of the object, the degree of overlap of the area of the object in the parallax image or the reference image may also be used. For example, even if a score is used in which the feature amount of the image is closer to the target object, the position of the object is closer to the predicted position, and the overlap degree of the object area in the parallax image or the reference image is higher, the higher the value is. good.

Next, with reference to FIG. 18, a more detailed example of the object tracking process of FIG. 17 will be described. FIG. 18 is a flowchart showing a detailed example of the object tracking process.

In step S2001, the prediction unit 1441 is one or more candidates for the target object for the current frame based on the positions of one or more candidates for the target object determined by the three-dimensional positioning unit 143 based on a plurality of previous frames. Predict each position (predicted position) of. The candidates for the target object will be described later.

Subsequently, the extraction unit 1442 acquires the object type of the target object determined by the object type classification unit 142 (step S2002).

Subsequently, the extraction unit 1442 was detected based on the current frame from the position predicted for each of the one or more candidates of the target object in step S2001 within the range of the actual distance according to the object type of the target object. It is determined whether or not there are a plurality of objects (step S2003). Here, the range according to the object type is, for example, within a radius of 2 m centered on the predicted position if the object type is "pedestrian" or the like, and within a radius of 3 m centered on the predicted position if the object type is "small car" or the like. It may be in the range of. FIG. 19 is a diagram illustrating a range of actual distances according to the object type of the target object. FIG. 19A shows an example of a range 902a of the actual distance from the predicted position 901a when the object type of the target object is “pedestrian” or the like. FIG. 19B shows an example of the range 902b of the actual distance from the predicted position 901b when the object type of the target object is "small car" or the like. As described above, in the case of the object type in which the relative speed with respect to the own vehicle is considered to be relatively large, the range of the actual distance from the predicted position is set to be relatively large.

If there are no more than one (NO in step S2003), the process ends. In this case, one object existing in the range may be determined to be an object matching the target object and may be tracked in subsequent frames.

When a plurality of objects exist (YES in step S2003), the extraction unit 1442 calculates the similarity of the image with the target object for each of the plurality of objects (step S2004). As the image of the target object, the average value of the images in the region of the target object in the luminance image or the parallax image of the previous plurality of frames may be used. Alternatively, the image when the target object is tracked may be used.

Subsequently, the extraction unit 1442 determines that the object having the highest image similarity with the target object among the plurality of objects is an object that matches the target object (step S2005).

Subsequently, the extraction unit 1442 selects, among the plurality of objects, the objects having a predetermined number or less in descending order of the degree of similarity of the image with the target object as the candidate objects matching the target object (step S2006).
As a result, in the subsequent frames, the candidate objects that match the target object are also subject to determination as to whether or not they are the target objects, as with the objects that are determined to match the target object in step S2005. Therefore, even if the similarity of the image is calculated to be low due to, for example, temporary reflection of light or temporary shadow of some object in this frame, and another object is tracked as the target object. , The target object can be detected correctly in the subsequent frames.

<Modification example>
Next, with reference to FIG. 20, another more detailed example of the object tracking process of FIG. 17 will be described. FIG. 20 is a flowchart showing another detailed example of the object tracking process.

In step S3001, the prediction unit 1441 predicts the position of the target object with respect to the current frame based on the position of the target object determined by the three-dimensional position determination unit 143 based on the previous plurality of frames.

Subsequently, the extraction unit 1442 calculates an area in the parallax image of the target object expected for the current frame based on the predicted position calculated in step S3001 (step S3002). For example, the extraction unit 1442 calculates the region in the parallax image of the target object by the same processing as the parallax image corresponding region detection unit 140 described above.

Subsequently, the extraction unit 1442 has the area in the parallax image of the target object expected for the current frame calculated in step S3002, and the object of each object in the current frame extracted by the object area extraction unit 141. The degree of overlap with the region is calculated (step S3003).

Here, the degree of overlap is a value indicating the degree of coincidence between the two regions. For example, the area of the region in the parallax image of the target object expected for the current frame is L, and one object in the current frame. When the area of the object area of is K and the area of the area where the two areas overlap is M, the ratio of M to K and L may be used. In this case, the overlap degree W can be calculated by, for example, the following equation [7].

W = M / {(K + L) / 2} ... Equation [7]
FIG. 21 is a diagram illustrating the degree of overlap. FIG. 21 shows an example of regions 914a and 914b in which the region 911 in the parallax image of the target object expected for the current frame and the object regions 913a and 913b of the objects 912a and 912b in the current frame overlap. Has been done.

Subsequently, the extraction unit 1442 determines whether or not there is an object whose overlap degree is equal to or higher than the first threshold value among the objects in the current frame (step S3004).

When there is an object whose overlap degree is equal to or higher than the first threshold value (YES in step S3004), the extraction unit 1442 determines that the object is an object that matches the target object (step S3005), and performs processing. finish.

When there is no object whose overlap degree is equal to or higher than the first threshold value (NO in step S3004), the extraction unit 1442 has a second object whose overlap degree is lower than the first threshold value among the objects in the current frame. Extract one or more objects that are equal to or greater than the threshold value of (step S3006). If there is no object that is equal to or greater than the second threshold value, the process may be terminated. In this case, for example, if an object matching the target object cannot be detected in the subsequent predetermined number of frames, the target object may be excluded from the tracking target.

Subsequently, the extraction unit 1442 calculates the similarity of the image with the target object for each of the one or more objects extracted in step S3006 (step S3007).

Subsequently, the extraction unit 1442 matches the object having the same image similarity with the target object as the predetermined value or more and having the highest degree of overlap calculated in step S3003 among the one or more objects with the target object. It is determined that the object is to be used (step S3008). In step S3008, the extraction unit 1442 selects, for example, the object having the highest value obtained by multiplying the similarity of images by the weighting coefficient according to the degree of overlap among the one or more objects, as an object that matches the target object. It may be determined that. As a result, even if there are no objects with a certain degree of overlap and the target object cannot be detected, the target object is detected under different conditions after relaxing the conditions and extracting the object candidates. High tracking can be continued.

As in FIG. 18, a candidate for an object that matches the target object selected in this frame is selected, and it is determined that the candidate also matches the target object in step S3005 or step S2008 in the subsequent frames. Like an object, it may be a target for determining whether or not it is a target object. In this case, for example, objects having a predetermined number or less are selected as candidate objects matching the target object in descending order of the degree of similarity of the image with the target object to the predetermined value or more and the degree of overlap calculated in step S3003. May be done.

<Summary>
According to the above-described embodiment, the predicted position of the predetermined object detected in the previous frame is calculated at this time. Then, among the plurality of objects satisfying the conditions corresponding to the predicted positions, the same object as the predetermined object is extracted based on the degree of similarity between each image of the plurality of objects and the image of the predetermined object. Will be done. As a result, highly accurate tracking can be continued.

Since the distance value (distance value) and the parallax value can be treated equivalently, the parallax image is described as an example of the distance image in the present embodiment, but the present invention is not limited to this. For example, a distance image may be generated by integrating the parallax image generated by using a stereo camera with the distance information generated by using a detection device such as a millimeter wave radar or a laser radar. Further, a stereo camera and a detection device such as a millimeter wave radar or a laser radar may be used in combination and combined with the detection result of an object by the stereo camera described above to further improve the detection accuracy.

The system configuration in the above-described embodiment is an example, and it goes without saying that there are various system configuration examples depending on the application and purpose. It is also possible to combine some or all of the above-described embodiments.

For example, the functional unit that performs processing at least a part of each functional unit of the processing hardware unit 120 and the image analysis unit 102 may be realized by cloud computing composed of one or more computers.

Further, in the above-described embodiment, an example in which the device control system 1 is mounted on an automobile as the own vehicle 100 has been described, but the present invention is not limited thereto. For example, as an example of another vehicle, it may be mounted on a vehicle such as a motorcycle, a bicycle, a wheelchair, or an agricultural cultivator. Further, it may be mounted not only on a vehicle as an example of a moving body but also on a moving body such as a robot.

Further, each functional unit of the processing hardware unit 120 and the image analysis unit 102 may be configured to be realized by hardware, or may be configured to be realized by the CPU executing a program stored in the storage device. good. The program may be recorded and distributed on computer-readable recording media in installable or executable files. Examples of the recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), and Blu-ray Disc. Further, this program may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. In addition, this program may be configured to be provided or distributed via a network such as the Internet.

1 Equipment control system 100 Own vehicle 101 Imaging unit 102 Image analysis unit (an example of "information processing device")
103 Display monitor 104 Vehicle driving control unit (an example of "control unit")
110a, 110b Imaging unit 120 Processing hardware unit 132 Parallax image generation unit 134 V map generation unit (an example of "acquisition unit")
135 Road surface shape detection unit 137 U map generation unit 138 Real U map generation unit 139 Isolated area detection unit 140 Parallax image corresponding area detection unit 141 Object area extraction unit 142 Object type classification unit 143 Three-dimensional position determination unit (“calculation unit”) An example)
144 Object tracking unit 1441 Prediction unit 1442 Extraction unit 2 Imaging device

Japanese Unexamined Patent Publication No. 2009-122786

Claims (10)

An acquisition unit that acquires information in which the vertical position, the horizontal position, and the depth position of the object are associated with each other at a plurality of time points.
A prediction unit that predicts the position of the predetermined object in the information acquired this time by the acquisition unit based on the position of the predetermined object in the information previously acquired by the acquisition unit.
From the information this time, a plurality of objects satisfying a predetermined condition according to the position of the predetermined object are extracted, and based on the degree of similarity between each image of the plurality of objects and the image of the predetermined object, It is provided with an extraction unit for extracting the same object as the predetermined object in the previous information among the plurality of objects in the information this time.
Satisfaction of the predetermined condition according to the position of the predetermined object means that the plurality of objects satisfy the predetermined distance from the position of the predetermined object calculated by the prediction unit according to the type of the predetermined object. The plurality of objects are located within the range.
Information processing device. An acquisition unit that acquires information in which the vertical position, the horizontal position, and the depth position of the object are associated with each other at a plurality of time points.
A prediction unit that predicts the position of the predetermined object in the information acquired this time by the acquisition unit based on the position of the predetermined object in the information previously acquired by the acquisition unit.
From the information this time, a plurality of objects satisfying a predetermined condition according to the position of the predetermined object are extracted, and based on the degree of similarity between each image of the plurality of objects and the image of the predetermined object, It is provided with an extraction unit for extracting the same object as the predetermined object in the previous information among the plurality of objects in the information this time.
The fact that the plurality of objects satisfy the predetermined conditions according to the position of the predetermined object means that, in the present information, the region in which the predetermined object exists and the plurality of objects in the vertical direction and the horizontal direction. The degree of overlap with the region where each of the objects is present is equal to or greater than a predetermined threshold value.
Information processing device. The extraction unit extracts, among the plurality of objects, the object having the highest degree of similarity to the image of the predetermined object as the same object as the predetermined object.
The information processing apparatus according to claim 1 or 2. The extraction unit is based on the similarity and the distance from the position of a predetermined object predicted by the prediction unit to each position of the plurality of objects in the information this time. Of these, the same object as the predetermined object is extracted.
The information processing apparatus according to any one of claims 1 to 3. The extraction unit is a predetermined object according to the position of an object that is determined not to be the same object as the predetermined object among the plurality of objects in the information this time in the information acquired thereafter by the acquisition unit. When the information of the object satisfying the condition exists, the same object as the predetermined object is extracted based on the similarity between the image of the predetermined object and the image of the object.
The information processing apparatus according to any one of claims 1 to 4. With multiple imaging units
A generation unit that generates the information based on a plurality of images taken by the plurality of imaging units, respectively.
The information processing apparatus according to any one of claims 1 to 5.
An image pickup device equipped with. The image pickup apparatus according to claim 6 and
A control unit that controls a moving body based on the data of the predetermined object extracted by the extraction unit, and a control unit.
Equipped with
A device control system in which the plurality of image pickup units are mounted on the moving body and image the front of the moving body. The device control system according to claim 7 is provided.
A moving body controlled by the control unit. The computer
A step to acquire information in which the vertical position, the horizontal position, and the depth position of an object are associated with each other at multiple time points.
A step of predicting the position of the predetermined object in the information acquired this time based on the position of the predetermined object in the previously acquired information.
From the information this time, a plurality of objects satisfying a predetermined condition according to the position of the predetermined object are extracted, and based on the degree of similarity between each image of the plurality of objects and the image of the predetermined object, Of the plurality of objects in the information this time, the step of extracting the same object as the predetermined object in the previous information is executed.
Satisfaction of the predetermined condition according to the position of the predetermined object means that the plurality of objects satisfy the predetermined distance from the position of the predetermined object calculated by the prediction unit according to the type of the predetermined object. The plurality of objects are located within the range.
Information processing method. On the computer
A step to acquire information in which the vertical position, the horizontal position, and the depth position of an object are associated with each other at multiple time points.
A step of predicting the position of the predetermined object in the information acquired this time based on the position of the predetermined object in the previously acquired information.
From the information this time, a plurality of objects satisfying a predetermined condition according to the position of the predetermined object are extracted, and based on the degree of similarity between each image of the plurality of objects and the image of the predetermined object, Of the plurality of objects in the information this time, the step of extracting the same object as the predetermined object in the previous information is executed.
Satisfaction of the predetermined condition according to the position of the predetermined object means that the plurality of objects satisfy the predetermined distance from the position of the predetermined object calculated by the prediction unit according to the type of the predetermined object. The plurality of objects are located within the range.
program.

Images