KR102614323B1 - Create a 3D map of a scene using passive and active measurements - Google Patents

Abstract

A method and device for generating a 3D map to obtain a three-dimensional (3D) map are presented. The method includes analyzing at least one image acquired by a passive sensor to classify a plurality of objects therein; Classifying the plurality of objects; determining whether to manually measure the distance to each object of the plurality of objects based on the classification; passively measuring the distance to at least one object among the plurality of objects based on the determination; Actively measuring the distance to some of the plurality of objects - when the distance to the object cannot be measured passively, measuring the distance to one of the same part of the plurality of objects Actively measuring the distance -; and generating a 3D map of the scene based on the distance measured for each of the plurality of objects.

Description

Create a 3D map of a scene using passive and active measurements

[Cross-reference with related applications]

This application claims priority to U.S. Provisional Application No. 62/141,296, filed on April 1, 2015, and U.S. Provisional Application No. 62/160,988, filed on May 13, 2015, the contents of which are incorporated herein by reference. It is included as

Autonomous vehicles (also known as driverless cars, self-driving cars, or robot cars) are vehicles that drive without human intervention. Autonomous vehicles use radar, Lidar, GPS, odometer, or computer vision to sense their environment to detect their surroundings. Advanced control systems interpret sensory information to identify obstacles and associated signs, as well as appropriate navigation paths. Self-driving cars are equipped with control systems to analyze sensory data to distinguish between other cars and obstacles on the road. Currently, driverless car technology is being developed by Google®, Tesla®, and some vehicle manufacturers such as Audi®, BMW®, Nissan®, etc.

For example, other companies such as Mobileye® are trying to provide solutions for hands-free driving technology in the market. The use of this technology is typically limited to specific driving infrastructure, such as highways or national highways. The core of these hands-free driving and autonomous vehicle technologies lies in rendering and creating a three-dimensional map of the scene at any moment during or immediately before movement. These maps attempt to mimic the scene as a driver would see it.

Rendering of these 3D maps is typically accomplished by measuring distances to many points in 3D space to determine the presence of objects and their respective distances to vehicles. The rendered 3D maps can be combined and processed by the vehicle to make driving decisions. Existing solutions for rendering detailed 3D maps are based on LiDAR (or LADAR) systems. LiDAR systems measure the distance to an object by illuminating multiple targets (points in space) using a single laser beam or multiple laser beams. These existing solutions configure the LIDAR system to scan the entire background (scene). Rendering a single 3D map requires a large number of laser measurements.

For example, Figure 1 shows an image 100 of a scene from which a 3D map is created. Some existing solutions enabled by hands-free and autonomous driving technologies measure the distance to each point 110 within the image 100. Therefore, a laser beam illuminates each point to render a 3D map. In many examples, the LiDAR system does not have prior knowledge of the scene, for example a picture of the scene. To achieve this, the technology must be based on very complex and expensive equipment. For example, a robot car made by Google® includes equipment equipped with a LiDAR system worth about $70,000. The LiDAR system includes 64 beam lasers. Because the hardware for rendering 3D maps is expensive, mass production of autonomous vehicles is not realistically possible. It should be noted that only a few points 110 are specifically shown in Figure 1 for simplicity.

Additionally, widespread production of autonomous vehicles using existing LiDAR systems will create hazardous conditions for pedestrians, drivers, and/or passengers due to the large number of laser beams transmitted from each vehicle that are likely to hit anyone within their line of sight. can do. Additionally, existing LiDAR solutions are configured to transmit laser beams at the highest energy level available. This is done to measure one point at the maximum range of the LIDAR system.

Additionally, when a 3D map is created by scanning the entire scene, crosstalk increases. This is because a large number of laser beams are transmitted by autonomous vehicles equipped with these scanning systems. As a result, the final resolution of the 3D map generated by scanning the entire scene may be limited.

Therefore, it is desirable to provide a solution for generating 3D maps that can overcome the shortcomings of the prior art.

A summary of several example embodiments of the disclosure follows. This summary is provided for convenience only to provide the reader with a basic understanding of the embodiments and does not fully define the scope of the disclosure. This summary is not an extensive overview of all contemplated embodiments, nor is it intended to identify key or critical elements of all embodiments or to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more embodiments in a simplified form as a prelude to the more detailed description that is presented later. For convenience, the term “some embodiments” may be used to refer to a single embodiment or multiple embodiments of the present disclosure.

Certain embodiments disclosed herein include methods for obtaining three-dimensional (3D) maps. The method includes analyzing at least one image acquired by a passive sensor to identify a plurality of objects in the image; Classifying the plurality of objects; Based on the classification, manually measuring each distance to at least one object among the plurality of objects classified as stationary objects; Actively measuring distances to some of the plurality of objects classified as non-stationary objects; Obtaining fused distance measurements for a plurality of objects by fusing each of the passively measured distances with each of the actively measured distances; and generating a 3D map of the scene based on the fused distance measurements.

Certain embodiments described herein include devices for obtaining three-dimensional (3D) maps. This device is a device for acquiring a three-dimensional (3D) map, comprising: a passive sensor for acquiring an image of a scene; A processing system, comprising: a processing system configured to analyze at least one image acquired by a passive sensor to identify a plurality of objects in the image and classify the plurality of objects; and an active sensor configured to actively measure respective distances to some of the plurality of objects classified as non-stationary objects, wherein the processing system is configured to measure each of the passively measured distances and the actively and further configured to fuse each measured distance to obtain fused distance measurements for a plurality of objects, and generate a 3D map of the scene based on the fused distance measurements.

Certain embodiments described herein also include methods for obtaining three-dimensional (3D) maps. The method includes acquiring at least one image by a passive sensor; segmenting the at least one image to render a segmentation map; actively measuring a distance value for each pixel of a first number of pixels within each segment identified in the segmentation map, the first number of pixels being less than the number of pixels within each segment; calculating a distance value for each pixel in the segment using the distance of pixels in the first number of pixels; and generating a 3D map of the scene based on the measured and calculated distance values.

The subject matter disclosed herein is particularly pointed out and expressly claimed in the concluding claims of the specification. These and other objects, features and advantages of the disclosed embodiments will become apparent from the following detailed description taken in conjunction with the accompanying drawings.
Figure 1 is a photograph of a scene from which a 3D map was created using a conventional approach.
2 is a block diagram of an apparatus configured to generate a 3D map from a vehicle according to one embodiment.
3A and 3B are diagrams illustrating identification of objects in an image according to one embodiment.
4 is a flowchart illustrating a method for generating a 3D map of a scene according to one embodiment.
5 is a flowchart illustrating a method for performing manual measurement according to one embodiment.
Figure 6 is an image showing a manual measurement based on known object size.
7 is a flowchart showing a method of performing manual measurement according to another embodiment.
Figure 8 is a screen shot used to explain determining whether an object is stationary based on changes in distance between frames.

It is important to note that the embodiments disclosed herein are merely examples of many advantageous uses of the innovative technology herein. In general, the description made in the detailed description herein is not necessarily limiting to the claimed embodiments. Additionally, some statements may apply to features of the invention and not to other features. In general, unless otherwise specified, singular elements may be plural and vice versa without loss of generality. In the drawings, like reference numbers refer to like parts across several views.

According to disclosed embodiments, a 3D map is rendered or otherwise generated based on active measurements, passive measurements, and/or analysis of information associated with such measurements. In some embodiments, the 3D map may be based on measurements captured from the vehicle. In one embodiment, the active measurement is a laser measurement, while the passive measurement is achieved by image processing. Accordingly, the disclosed embodiments enable providing a complete 3D map of the scene while using a small number of low energy laser beams. The scene may be a view captured by a device that allows it to create a 3D map. In certain configurations, the disclosed embodiments can be used to generate high-resolution 3D maps of specific sections of the environment without performing active measurements over the entire scene. The use of fewer active measurements reduces the risk of human exposure to laser radiation.

2 shows an example block diagram of an apparatus 200 configured to generate a 3D map from a vehicle, according to one embodiment. Device 200 includes memory 215, active sensor 220, and passive sensor 230 coupled to processing system 210. In certain configurations, device 200 further includes a storage device 240 coupled to processing system 210, one or more motion sensors 250, and a network interface 260.

Device 200 may be mounted on or integrated into a vehicle, for example. These vehicles may include, for example, cars, trucks, buses, drones, robots, etc. Device 200 may be used to generate a 3D map of the scene as observed by an operator (e.g., driver) of the vehicle. In one embodiment, device 200 is configured to control some or all of the vehicle's operations based on analysis of the generated 3D map. Accordingly, device 200 may be used in applications related to autonomous vehicles, hands-free driving systems, driver assistance systems, etc.

Passive sensor 230 may be an image sensor (eg, camera) configured to passively acquire images. The acquired images may be processed by processing system 210 or by passive sensor 230 using image processing techniques.

Active sensor 220 is configured to perform active measurements in a specific direction to determine the distance from an object in the scene. In one embodiment, the active measurement is a laser measurement. In another embodiment, active sensor 220 is configured to illuminate light rays in a particular direction (e.g., determined by processing system 210) and detect light rays reflected from objects in the scene. The distance from the object can be calculated based on the energy and reflection angle of each reflected ray. The active sensor 220 may be implemented with active stereovision, structured light, time-of-flight (TOF), etc., but is not limited thereto.

Processing system 210 is configured to process data (e.g., energy level, direction, angle of reflection of emitted and reflected beams) to generate a 3D map of the entire environment or a specific object within the environment. As described below, the number of active measurements is limited to a specific point and/or surface in space when initial scanning is performed using the image or images acquired by passive sensor 230. In embodiments, previously determined active measurements and/or sensory information collected from motion sensor 250 may be used to generate 3D map additional information, such as previously acquired images or previously created 3D maps. . The motion sensor 250 may include, but is not limited to, a global positioning system (GPS), an accelerometer, a proximity sensor, an odometer, etc. The side information along with the active/passive measurements can be used by the processing system 210 to determine the size of each object, its speed, and its orientation relative to the vehicle.

According to certain embodiments, processing system 210 is configured to generate 3D maps. To this end, processing system 210 fuses active and passive measurements; determine an object that is stationary or nearly stationary; Actively measures only the distance from a moving object; configured to estimate the distance from the object using the object's highest possible speed and/or manual measurements.

The fusion of active and passive measurements may include using the active sensor 220 to measure an object in areas where the passive sensor 230 does not provide reliable measurements. Whether the passive sensor 230 can provide reliable measurement of an object is based on, but is not limited to, classification of the object, visibility of the object in the image, etc. In one embodiment, active sensor 220 is not used if reliable passive measurements can be achieved. Accordingly, the disclosed embodiments can significantly reduce the number of active measurements performed by active sensor 220.

As such, fewer laser beams or other sources can be used, thereby achieving higher resolution as well as reducing energy consumption, crosstalk, and hazardous conditions. In one embodiment, device 200 may be configured to actively measure only suspect status objects using active sensor 220 . A suspect status object can be any object that requires active measurements to accurately generate a map based on it. The classification of objects is described below. The object may be in a suspicious state, for example, when the object is in a non-stationary state (i.e., moving) and/or in close proximity to the device 200. For example, when the device 200 is mounted on a car, pedestrians are recognized as suspect, while trees in the background are not.

In one embodiment, processing system 210 is configured to determine whether an object is at rest or near still based on the image provided by passive sensor 230 . For example, using image recognition, processing system 210 may be configured to determine a stationary object, such as a house, tree, pole, etc. In another embodiment, when passive sensor 230 is stationary, processing system 210 compares two consecutive images and, based on this comparison, determines which object has moved relative to the stationary object, thereby It may be configured to determine whether a stationary state is present. As an example, if a passive sensor is anchored to the side of a building in an environment, successive images can be compared to determine whether any object in the scene has moved relative to the anchored object.

In another embodiment, determining whether an object is stationary may be based on a comparison of frames (e.g., images) captured by passive sensor 230. Comparison may include, but is not limited to, determining distance changes between sets of points within a frame. In an example embodiment, each change in distance between two points may be equal to the difference between the 3D distance between the points shown in the first frame and the 3D distance between the points shown in the second frame. An object determined to have moved may be associated with an object that is non-stationary. In one embodiment, for example, if the sum of all distance changes associated with the object is greater than or equal to a predefined threshold, the object may be determined to have moved. In another embodiment, an object may be determined to have moved if the sum of distance changes associated with a point is greater than the sum of distance changes associated with each of the other objects that exceeds a predefined threshold. In another embodiment, the object may be determined to have moved if the distance change associated with the object in the subsequent frame is greater than the distance change associated with the object in the previous frame that exceeds a predefined threshold.

8 is an example diagram 800 used to describe using distance changes between frames to determine whether an object is stationary. In the example screen shot 800, each frame 810-830 represents the positions of points 1, 2, 3, and 4 in the image (which may represent an object) and each pair of points 1, 2. , 3 and 4). The overlapped frame 830 indicates that the frame 820 is overlapped on top of the frame 810. As shown in frame 830, points 1, 2, and 4 are at approximately the same locations as in frames 810 and 820, respectively. However, point 3 is in a different position in frame 810 than in frame 820. The sum of the distance changes associated with point 3 may be the sum of the distance changes between frames 810 and 820 for pairs of points (1 and 3), (2 and 3), and (4 and 3). As a result, the sum of the distance changes associated with point 3 will be greater than zero between frames 810 and 820, and may be greater than a predefined threshold. Additionally, the sum of the distance changes associated with point 3 becomes greater than the sum of the distance changes associated with each of points 1, 2, and 4. Accordingly, it may be determined that the object associated with point 3 has moved between frames 810 and 820.

In another embodiment, determining whether an object in an image is stationary may be based on a predicted location of the object in the image. The predicted image may be generated based on a frame of the currently acquired image or a generated 3D map. The predicted image may be further based on the movement of the sensor, which provides information used to acquire the current image or generate the current 3D map. Based on the current frame and/or any new position or orientation of the sensor, a prediction frame is generated that represents the predicted position of the object at a subsequent time (assuming the object does not move). The predicted frame may be compared to subsequent frames based on sensor readings at subsequent times to determine if there are any differences between the positions of the object. In one embodiment, if the difference in the position of the object between the prediction frame and the corresponding subsequent frame exceeds a predetermined threshold, it may be determined that the object has moved.

In one embodiment, when the passive sensor 230 moves, the distance from stationary objects may be measured using GPS and the speed of the car. In other embodiments, distances from stationary objects may be measured without using GPS by comparing frames (e.g., acquired images) of passive sensor information and/or 3D mapping information. In a further embodiment, finding the distance and/or angle change between any two frames may be determined for each pair of corresponding points within the frames based on a weighted score. To this end, in one embodiment, discovering a distance or angle change between two frames may further include determining coincident or otherwise corresponding points within the frames. In another embodiment, the weighted score is based on the degree of error associated with the distance of points within each frame, such as, but not limited to, error based on noise.

In another embodiment, the prediction frame may be generated based on the location and orientation of the sensor as well as 3D or 2D mapping information. A prediction frame may be a 3D image representing the predicted positions of stationary (i.e., non-moving) objects included in 3D mapping information. Such stationary objects may include, but are not limited to, trees, signs, signs, buildings, and other permanent or semi-permanent items or fixtures. Prediction frames can be used, for example, to fill exposed gaps in previous frames (e.g., when an object moves, prediction frames can be used to account for predictions about items behind the previous position of the moved object). can be).

In other embodiments, manual measurements may be performed based on the movement of objects that are classified as non-stationary and non-suspicious. To this end, passively measuring the distance may include, but is not limited to, determining the boundary speed of the object and the direction of movement of the object. Boundary speed is the speed at which the boundary (i.e., external surface) of an object moves. Based on the determined boundary speed and direction, the distance to the object may be estimated. In one embodiment, the estimated distance may be further based on a previously known distance to the object and/or a previously generated 3D map containing the object.

In another embodiment, processing system 210 is configured to estimate distances from non-stationary objects using the maximum and/or minimum possible speeds of such objects. The maximum or minimum possible speed of the object may be utilized as the boundary speed. For example, if the distance from a pedestrian is first measured, a distance range representing the possible distance of that pedestrian at a given time (e.g., between 0 and 2 meters) can be determined based on the person's predetermined maximum speed.

If the pedestrian is located within a range that does not require any action (e.g., walking on the sidewalk across the street from a vehicle), the processing system 210 does not trigger any other active measurements for the pedestrian. One action may include, for example, adjusting the vehicle. Whether action is needed can be based on predefined safe distance thresholds. The safe distance threshold may include, but is not limited to, the speed of the vehicle and/or object, the distance of the object from the device 200, the type of object, a combination thereof, etc. In another embodiment, active measurements may be performed when the estimated distance to a moving object (e.g., based on boundary speed and direction of the object) does not meet a safe distance threshold. One action may include, for example, controlling a hands-free or autonomous vehicle.

In one embodiment, processing system 210 is configured to estimate the distance of processing system 210 from non-stationary and stationary objects using manual measurements. In particular, the distance may be estimated based on the approximate size of the object and the amount of pixels occupied by the object in the image acquired by the passive sensor 230. This estimation may further include determining whether the subject is close to the vehicle. For some objects that are not close, active measurement by the active sensor 220 is not required. Close proximity may be established based on a predefined threshold (e.g., a distance of less than 100 meters may be determined as proximity).

By considering specific (e.g., suspect) objects, the number of active measurements by the active sensor 220 can be reduced. FIGS. 3A and 3B are diagrams showing example images of a scene in which a 3D map was created according to an embodiment of the present invention. First, an image 300 of the surrounding environment is acquired by a passive sensor 230. Image 300 is segmented to identify object 310. Segmentation and identification of objects can be performed using conventional image processing techniques. For example, roads, pedestrians, cars, vans, trucks, crash barriers, and noise barriers are identified as objects 310. Then, object classification and manual measurements are performed. Based on the object classification, it is determined which active measurements should be made, i.e. where to direct the laser beam emitted by the active sensor 220. Classification of an object can determine which object is in a stationary state, a non-stationary state, or a suspect state. For example, pedestrians, cars, vans, and trucks are classified as non-stationary and suspicious objects, and as such, active measurements in many different directions are triggered at locations 320 based in part on the object classification. Roads, crash barriers, sky, roads and acoustic barriers are classified as stationary and non-suspicious objects, so fewer active measurements are performed.

As evidenced by FIG. 3B, only a portion of the scene acquired in image 300 is scanned with active sensor 220. Position active measurements are labeled 320. Performing a larger number of active measurements on a limited number of objects provides higher resolution with respect to these objects. This can provide higher resolution in specific areas where detailed recognition is required.

Additionally, FIGS. 3A and 3B are illustrative only and do not limit the various disclosed embodiments. In particular, more, fewer or different objects 310 and/or 320 may be identified in the image without departing from the scope of the present invention.

Once all measurements are complete, a 3D map of the scene can be created. Specifically, the 3D map is created by calculating the device's distance from any pixel in the acquired image. That is, each pixel must be associated with a distance value to create a 3D map. Distance values can be derived from passive and active measurements. In one embodiment, only a series of distance measurements are performed (eg, measurements less than the number of pixels). Distance values can be calculated or extrapolated using plane equations or other equivalent techniques.

Returning to FIG. 2 , processing system 210 may include or be a component of a processor (not shown) or an array of processors coupled to memory 215 . Memory 215 contains instructions that can be executed by processing system 210. When executed by processing system 210, these instructions cause processing system 215 to perform various functions described herein. One or more processors may include general-purpose microprocessors, multi-core processors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic, and discrete. It may be implemented as any combination of hardware components, a dedicated hardware finite state machine, or any other suitable entity capable of performing computations or other manipulations of information.

Additionally, processing system 210 may include a machine-readable medium for storing software. Software is broadly interpreted to mean any type of instruction, whether referred to as software, firmware, middleware, microcode, hardware description language, etc. Instructions may include code (e.g., code in source code format, binary code format, executable code format, or any other suitable format). When executed by one or more processors, the instructions enable the processing system to perform various functions described herein.

Storage device 240 may be a magnetic storage device, optical storage device, etc., such as flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVDs), or other optical storage that may be used to store the desired information. It may be realized as a device, a magnetic cassette, a magnetic tape, a magnetic disk storage device or other magnetic storage device, or any other medium.

In one configuration, computer readable instructions for implementing any of the embodiments disclosed herein may be stored in storage device 240. Storage device 240 may also store other computer-readable instructions for implementing operating systems, application programs, etc. Computer-readable instructions may be stored in memory 215 for execution by processing system 210. Storage device 240 may also be configured to store, for example, images, generated 3D maps, 3D or 2D maps provided by a mapping service (e.g., street level maps).

The network interface 260 allows the device 200 to communicate with other devices, such as, but not limited to, a vehicle controller (not shown), a central controller (not shown), and a cloud storage device (not shown). make it possible For example, network interface 260 may be configured to allow device 200 to communicate with a vehicle's controller to provide operating commands (e.g., stop, turn right, etc.) over a network (not shown). In one embodiment, network interface 340 allows remote access to device 200 for purposes such as configuration, loading new maps, etc. Network interface 260 may include a wired connection or a wireless connection. Network interface 260 may transmit and/or receive communication media. For example, network interface 260 may include a modem, a network interface card (NIC), an integrated network interface, a radio frequency transceiver, an infrared port, a USB connection, etc.

4 is an example flow diagram 400 illustrating a method for generating a 3D map of a scene according to one embodiment. In one embodiment, this method may be performed by a device operable in a vehicle (e.g., device 200), with the generated 3D map of the scene being viewed by an operator of the vehicle. 3D maps can be created based on the fusion of passive and active measurements.

In step S410, one or more images are acquired by a passive sensor. In one embodiment, acquired images are stored in memory according to acquisition order. In other embodiments, the acquired images may include frames of video that are recorded continuously.

In optional step S420, the image may be segmented to generate a segmentation map. A segmentation map includes a plurality of segments, each segment characterized by homogeneous pixels having the same color or a substantially similar color.

In step S430, objects in the image or segmentation map are identified. In one embodiment, the object may be identified based on image recognition, for example. An object may include multiple segments. For example, a truck painted in two different colors can be segmented into two segments but identified as one object.

In step S440, the identified objects are classified. In one embodiment, a subject may be classified into one or more of the following categories: non-stationary (i.e., moving or possibly moving), stationary (i.e., not moving), suspicious, or non-suspicious. In one embodiment, whether an object is non-stationary or stationary may be determined by identifying the object (eg, using image recognition) and based on its type. For example, an object representing a building may be classified as non-stationary. In another embodiment, the non-stationary or stationary state category of an object is determined by comparing two consecutive images taken during a predefined time interval to determine whether the object's position has changed. An object is classified as stationary if its position remains the same; Otherwise, the object may be classified as non-stationary. In another embodiment, object classification for some of the identified objects may be based on the size of the identified objects. As a non-limiting example, objects with a size above a predefined threshold (e.g., large enough for a vehicle to need to navigate around the object) may be classified as suspect, while objects with a size below a predefined threshold may be classified as suspect. Objects that have (e.g., small enough for a vehicle to safely drive over them) can be classified as non-suspicious.

In one embodiment, classification based on image comparison may be further based on movement (or lack of movement) and/or positioning of passive sensors. Specifically, when the passive sensor is stationary (for example, a vehicle is stopped at a traffic signal) and two consecutive images are taken from the same angle, the comparison indicates changes in the position of the moving object relative to the other object. You can. When the passive sensor itself moves, two consecutive images are taken from different angles. To enable accurate comparison, in one embodiment, the second image (e.g., a later captured image) may be transformed to match the angle of the first image (e.g., a previously captured image). So, the two images can be placed at the same angle. Image transformation can be performed using virtual camera technology. It should be noted that in one scene, an object may be classified as stationary, whereas in the next scene, an object may be classified as non-stationary (e.g., a parked car). In one embodiment, classification of stationary objects may be performed based in part on geographic maps, such as 3D, 2D, street view, and/or satellite maps, if a mapping service is provided. This allows buildings, etc. to be identified based on their known location.

In one embodiment, the classification of suspect status subjects is based on a predetermined list of suspect status subjects. For example, an object predetermined to be in a suspicious state may be, but is not limited to, another vehicle on the road, a pedestrian, an animal, or debris on the road. The list can be updated dynamically. In general, objects that are or may be dangerous in close proximity to a vehicle are classified as suspect. It should be noted that subjects may be classified as “suspicious and quiescent” or “suspicious and non-quiescent.” Various embodiments of object classification have been described above.

In step S445, for each object classified as stationary, it is checked whether the distance to the object classified as stationary is known. If so, execution continues with S447; Otherwise, execution continues with S450. At S447, the distance to the stationary object is retrieved from previous calculations. This distance information can be derived from a previously created 3D map. A stationary object may be determined to be stationary based on a previously generated 3D map. As an example, if the passive sensor is not moving, the object is stationary, and a previously determined passive measurement and/or active measurement distance is available for the object from the passive sensor, then the current passive measurement for the object is the previously determined The distance can be determined to be

In step S450, manual measurements are performed based on the acquired images. Manual measurement involves measuring at least a distance from at least one, some or all of the classified objects. As mentioned above, these measurements are performed when information cannot be derived from previously created 3D maps. In one embodiment, the distance from each object classified as “stationary” or “non-suspicious and non-stationary” is determined through manual measurements. As described above, when the distance to a stationary object is not available, the distance is manually measured for the stationary object. In one embodiment, manual measurements may be made using the known size of the object (e.g., number of pixels), effective reference points, distance from the vanishing point and vanishing line, movement of the device or vehicle, etc. to the object in the acquired image. It may include the step of determining the distance. Various embodiments for performing manual measurements are described in more detail below with reference to FIGS. 5 and 6. Distance measurements may be stored in memory (e.g., memory 215). It should be noted that the distance measurement may be for all or any pixels that make up the object.

In other embodiments, manual measurements may be performed based on the movement of objects that are classified as non-stationary and non-suspicious. To this end, passively measuring the distance may include, but is not limited to, determining the boundary speed of the object and the direction of movement of the object. Boundary speed is the speed at which the boundary (i.e., external surface) of an object is moving. Based on the determined boundary speed and direction, the distance to the object may be estimated. In one embodiment, the estimated distance may be further based on a previously known distance to the object and/or a previously generated 3D map containing the object.

In step S460, active measurement is performed. In one embodiment, active measurements may be performed using an active sensor (e.g., active sensor 220). Performing active measurements may include measuring distances from some or all of the classified objects. In one embodiment, distances are actively measured from objects classified as “suspicious,” “non-stationary,” and/or “suspicious and non-stationary.” In other embodiments, distances are measured actively when reliable manual measurements cannot be achieved. Reliable manual measurements are performed when one or more objects and/or details appear multiple times in the acquired image; If the image contains plain areas (e.g. walls and sky) without obvious texture; When details in the image (such as the steering wheel inside a car) are hidden from other scenes; When only one image from a unique angle is available for passive measurement (for example, when the passive sensor includes only one camera); It cannot run if the image contains one or more noisy areas. By re-measuring these objects, accuracy increases and the number of false alarms is significantly reduced.

In an embodiment, S460 includes using a laser diode to emit a laser beam (or light pulse). The laser reaches the target and some of the laser energy is reflected back towards the active sensor. A return signal is detected, and the elapsed time between emission of the light pulse from the laser and detection of the returned signal is determined. The distance measurement of the distance to the object may be determined based on the determined elapsed time. It should be noted that the distance measurement may be for some or all of the pixels that make up the image. In one embodiment, multiple active measurements may be performed per object, and in each such measurement the laser pulse is directed at a different angle. The resolution of the measurements may be based on the number of measurements performed on each subject.

The timing, direction and energy level of each emitted laser beam can be controlled to achieve accurate distance measurements. In one embodiment, at least the direction of the laser beam is adjusted based on time and position differences between the time the image was acquired by the passive sensor and the time when the active measurement is triggered. This is done to compensate for movement of the active sensor and/or target object during that time. The direction pointing the active sensor can be estimated using accelerometer information by determining the current location of the active sensor based on its location when the image was acquired.

It should be noted that the active measurements performed in step S460 may be based on other active transmissions or emissions, such as, but not limited to, radar, acoustics, laser triangulation, etc. It should be noted that S450 and S460 can be performed in parallel. For example, manual measurements may be performed in parallel for objects classified as non-stationary and/or for objects that cannot be measured passively.

In certain embodiments, the energy level of the emitted laser is controlled based on the object's proximity to the vehicle (or active sensor). Proximity can be determined using previous active and/or passive measurements in the area being scanned. By controlling the energy level, the energy consumption of the active sensor can be reduced compared to always using a laser beam with the highest amount of energy possible. Additionally, controlling the energy level can reduce the risk of injuring people by the laser beam when a nearby object is scanned at a relatively low energy level.

In step S470, a 3D map is generated based on passive and active measurements. To create a 3D map of a scene, the distance values of pixels must be available in the acquired image. In one embodiment, one or more distance values may be included in the 3D map for each segment of the segmentation map. Each distance value can be either a passive or active measurement. In a preferred embodiment, at least three distance values for at least three pixels of each segment should be included in the 3D map.

As described above, a segment may be a portion of an object or the entire object. The distance value of every pixel within a segment may be calculated or extrapolated based on at least three distance measurements. To this end, in one embodiment, the plane equation is calculated based on at least three distance measurements for at least three pixels in each segment. The calculated plane equation can be used to determine all distance values of pixels on the same segment and/or on the same surface. It should be noted that other geometric techniques for calculating the distance values of all pixels within a segment may be used without departing from the scope of the present invention. It should be further noted that for some segments, the distance values for every pixel are available through active or passive measurements, so there is no need to solve the plane equations for these segments.

In other embodiments, if the distance value of every pixel cannot be estimated based on the plane equation, more active measurements will be triggered.

In another embodiment, if all distance values for all pixels are available from the previous frame, then a limited number of active measurements (e.g., three measurements) and the remainder of the distance values are adjusted from previously performed measurements.

Rendering of the 3D surface of each segment results in rendering of a 3D representation of the identified object, resulting in a 3D map of the scene. The resulting 3D map can later be used, for example, to provide driving directions to a controller controlling the operation of the vehicle.

5 is an example flowchart (S450) illustrating a method for performing manual measurements based on acquired images according to one embodiment. In one embodiment, the method may be performed by a passive sensor (e.g., passive sensor 230). In S510, the currently acquired image (eg, Image _n+1 ) and the previously acquired image (eg, Image _n ) are searched along with a list of objects classified as stationary. At S520, the current and previously acquired images are compared to identify differences for each object in the image. In one embodiment, S520 may further include changing one of the images so that the angles of the images are the same. This change is performed based on a change in the position of the sensor's orientation.

At S530, it is checked whether there are any changes between the objects appearing in each image, and if so, execution continues with S540; Otherwise, execution terminates. A change can occur, for example, in a new stationary object identified in the current image (i.e., an object that does not appear in a previously identified image) or in a new location of a previously identified object (i.e., an object that appears in a previously identified image). It can be based on If no change is detected, a previously determined distance may be suitable for use as the current distance, thereby eliminating the need to calculate or recalculate the distances of any objects in the current image.

At S540, an object associated with a change between images is selected. At S550, it is checked whether the passive sensor has moved or was moved when the current and previously identified images were acquired. If so, execution continues with S560; Otherwise, execution continues at S570. Determining whether the passive sensor is moving may be based, for example, on information from one or more motion sensors (e.g., accelerometer, global positioning system, gyroscope, combinations thereof, etc.) coupled to the passive sensor.

At S560, the distance to the selected object is determined. The distance may be determined based on measurements from one or more motion sensors, for example. In one embodiment, the motion sensor is a GPS and the distance is calculated based on the difference in GPS coordinates captured when each image was taken. In another embodiment, distances from stationary objects may be measured without using GPS by comparing frames (e.g., acquired images) of passive sensor information and/or 3D mapping information. These embodiments are described in more detail above.

At S570, the distance is calculated based on pixels within the image. In one embodiment, the distance is calculated by recognizing the number of pixels for an object and comparing the number of pixels to a known reference measurement. If the width and height of the object are known, the distance can be calculated in relation to the known width and height.

6 is an example image 600 used to illustrate calculating distance based on the number of pixels for an object. For example, if the width 620 of the truck 610 in image 600 is known to consist of 100 pixels and the width of image 600 represents a length of 1 meter, then the truck in image 600 ( If the distance 630 to 610) includes 1000 pixels, it can be determined that the distance to the truck is 10 meters. The reference measurement may be based on a predetermined measurement, a distance from a known object, or a known size (e.g., height, width, etc.) of the object appearing in the image. In one embodiment, distance can be estimated by calculating vanishing points and vanishing lines in the image. For example, the roof line of a building can be used as a reference measurement.

It should be noted that in one embodiment, distances may be measured passively using different techniques to increase the accuracy of the measurements. As mentioned above, if certain passive measurements are unreliable, active measurements can be used to increase the accuracy of the distances determined.

In S580, the distance is stored (e.g. in memory). At S590, it is checked whether all necessary distances for changed objects have been determined (i.e., all objects have been processed). If so, execution terminates; Otherwise, execution returns to S540.

7 shows an example flow diagram S450 illustrating a method for performing manual measurements according to another embodiment. In S710, the current image (image _n+1 ), the previously acquired image (image _n ), and a list of objects classified as non-stationary are searched. In one embodiment, for each non-stationary object, a previously measured distance (e.g., with respect to Figure 1) and a predetermined known speed may also be retrieved. Known speeds include either maximum, minimum, or average speeds.

In S720, the current and previously acquired images are compared to detect the direction of movement of the non-stationary object. The direction may be towards or away from the vehicle, device, and/or passive sensor, but is not limited to this. In other embodiments, the direction may be a specific angle relative to a passive sensor, for example.

At S725, an object in a non-stationary state is selected. At S730, the current distance to the selected non-stationary object is estimated. The current distance can be estimated based on the time elapsed between captures of the current image and a previously acquired image, known speed, and previously measured distance. The current distance can be further estimated based on the determined direction. That is, if the determined direction is towards the vehicle, the current distance can be estimated using the maximum known speed. On the other hand, if the determined direction is away from the vehicle, the distance can be estimated using the minimum known speed. This is done to cover the worst-case scenario regarding the subject's distance from the vehicle (i.e., the distance may be an under-estimate). Such worst-case scenario estimates can, for example, help prevent self-driving cars, drones, etc. from crashing into moving objects.

At S740 based on the estimated distance, it is determined whether active measurement is necessary. If so, execution continues with S750; Otherwise, execution continues with S760. In one embodiment, the estimated distance is compared to a predetermined threshold, and if the threshold is not met, active measurements are required. Active measurements can be initiated if the object is not within a safe distance. For example, if a person is walking toward a vehicle and the estimated distance is 100 meters, the person is within a safe distance. However, if the expected distance of a person walking towards a vehicle is 1 meter, the person will be outside a safe distance and an active measurement should be triggered.

At S750, if it is determined that active measurement is needed, the subject is flagged for active measurement. At S760, the estimated distance is stored in memory. At S770, it is checked whether all objects have been processed. If so, execution ends. Otherwise, execution continues with S740.

It should be noted that the embodiments discussed herein have been described in the context of active measurement by lasers for the purpose of simplicity only and not to limit the disclosure. Other active measurements (e.g., by radar, sound, etc.) could equally be used without departing from the scope of the disclosed embodiments.

Various embodiments disclosed herein may be implemented as hardware, firmware, software, or any combination thereof. Additionally, the software is preferably implemented as an application program tangibly embodied in a computer-readable medium or a program storage unit comprised of parts or specific devices and/or combinations of devices. The application can be uploaded and run on a machine with any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPUs”), memory, and input/output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be part of microinjection code or part of an application, or a combination thereof, and may be executed by a CPU, whether or not such computer or processor is explicitly indicated. You can. Additionally, various other peripheral units may be connected to the computer platform, such as additional data storage units and printing units. Additionally, non-transitory computer-readable media is any computer-readable media other than transient radio signals.

All examples and conditional language cited herein are intended for educational purposes to help the reader understand the principles of the disclosed embodiments and the concepts the inventors contribute to promote the art, and are limited to the specifically cited examples and conditions. It should be interpreted that it does not work. Additionally, all statements herein reciting principles, aspects, and examples of the disclosed embodiments, as well as specific examples thereof, are intended to encompass all structural and functional equivalents thereof. Additionally, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., any developed elements that perform the same function regardless of structure.

Claims (20)

In a method for obtaining a three-dimensional (3D) map,
Analyzing at least one image acquired by a passive sensor to identify a plurality of objects in the image;
Classifying the plurality of objects;
Based on the classification, manually measuring each distance to at least one object among the plurality of objects classified as stationary objects;
Actively measuring distances to some of the plurality of objects classified as non-stationary objects;
Obtaining fused distance measurements for a plurality of objects by fusing each of the passively measured distances with each of the actively measured distances; and
generating a 3D map of the scene based on the fused distance measurements.
How to include . The method of claim 1, wherein analyzing the at least one image comprises:
segmenting the at least one image to render a segmentation map; and
Identifying the plurality of objects in the segmentation map
How to further include . The method of claim 1, wherein classifying the plurality of objects comprises:
Identifying the type of each of the plurality of objects; and
Classifying each of the plurality of objects based on the type of the identified object.
How to further include . According to paragraph 3,
comparing the at least one image to a previously acquired image; and
Identifying a change in the position of each of the plurality of objects based on the comparison
It further includes,
A method wherein each of the plurality of objects is classified based on the change in the position of the object. According to paragraph 1,
For each object in the plurality of objects, determining whether the object is identified in at least one geographic map; and
Classifying each object identified in the at least one geographic map as a stationary object.
How to further include . According to paragraph 1,
In response to the previous distance to at least one object among the plurality of objects classified as the stationary object being unavailable, passively measuring the respective distance to at least one object among the plurality of objects. steps to do
How to further include . The method of claim 1, wherein the step of manually measuring each distance to the at least one object:
determining whether the passive sensor is moving;
When the passive sensor moves, determining each distance to the at least one object using a motion sensor; and
When the passive sensor is not moving, determining a respective distance to the at least one object based on a known reference measurement and the number of pixels for the at least one object.
How to further include . 8. The method of claim 7, wherein the known reference measurement is at least one of the number of pixels from a valid reference object and the size of the object. The method of claim 1, wherein
A step of manually measuring the respective distances to some of the plurality of objects classified as objects in a non-stationary state and objects in a non-suspicious state,
For each of some of the plurality of objects, determining a boundary velocity of the object;
determining the direction of movement of the object,
comprising estimating the distance to the object based on the determined direction and the determined boundary velocity.
How to further include . The method of claim 1, wherein the step of actively measuring the respective distances to some of the plurality of objects classified as objects in a suspicious state.
How to further include . The method of claim 1, wherein when reliable manual measurements cannot be determined for some of the plurality of objects, the respective distances to some of the plurality of objects classified as non-stationary objects are determined. Steps to actively measure
How to further include . The method of claim 1, wherein the step of actively measuring distances to some of the plurality of objects includes:
The method further includes performing a plurality of laser measurements on each of some of the plurality of objects, wherein the plurality of laser measurements are determined based on a required resolution. According to clause 12,
For each of some of the plurality of objects, calculating the distance to each point of the object based on the laser measurement.
How to further include . The method of claim 1, wherein generating the 3D map comprises:
The method further comprising rendering a 3D surface of each object in the plurality of objects using the fused distance measurement. The method of claim 1, wherein the scene is a field of view captured by a device that enables creation of the 3D map. A non-transitory computer-readable medium storing instructions for executing the method according to claim 1. In a device for obtaining a three-dimensional (3D) map,
Passive sensor for acquiring images of the scene;
As a processing system,
Analyze at least one image acquired by a passive sensor to identify a plurality of objects in the image,
a processing system configured to classify the plurality of objects; and
An active sensor configured to actively measure the distance to some of the plurality of objects classified as non-stationary objects,
The processing system is
Obtaining fused distance measurements for a plurality of objects by fusing each passively measured distance with each actively measured distance,
further configured to generate a 3D map of the scene based on the fused distance measurements,
Device.


delete delete delete

Images