JP5227065B2 - 3D machine map, 3D machine map generation device, navigation device and automatic driving device - Google Patents

Description

The present invention evaluates a three-dimensional position of a moving body such as an automobile, a heavy machine, and an aircraft, and automatically operates or assists the three-dimensional map, its three-dimensional map generation device, and automatic using the three-dimensional map. It relates to a driving device.
In particular, the present invention searches for the current position or future position of a moving body that travels and sails in a three-dimensional space and does not determine the traveling direction of the moving body, the vehicle posture, etc. By determining, an automatic driving device that can determine the present situation and future situation of a moving body with high accuracy, a navigation device (International Publication WO2005 / 038402) and an old and new video coordinate integration device ( The present invention relates to a suitable three-dimensional map and its three-dimensional map generation apparatus, including JP-A-2007-114916).

As a navigation device for navigating the movement of a vehicle or the like, a car navigation system using a GPS geodetic satellite is known (see Patent Documents 1-3).
In general, as a map of a navigation device for navigating the movement of a vehicle or the like, a two-dimensional map is prevalent, and even if it is called a three-dimensional map, it can only display a townscape expressed in CG at most. .
In addition, when performing automatic driving of a vehicle or the like, first, a method for accurately evaluating the self position must be obtained. Currently known as self-location assessment is GPS.
Here, as the position accuracy obtained by the GPS navigation system, conventionally, there is an influence of radio wave reflection and refraction in the ionosphere, and the error is 50 to 300 meters.

In recent years, a method has been added to measure the error of radio wave arrival time using known points of latitude, longitude, and altitude, and to transmit it as a correction signal to correct the error at the receiving point. Has been reduced to about a dozen meters.
However, this level of accuracy is insufficient for vehicle control, and the current GPS accuracy, which can be understood only for the approximate position, cannot guide the vehicle in the road lane.
Therefore, even if the accuracy of the map is slightly better than the GPS accuracy, there has been no problem in use.

JP-A-11-304513 JP 2001-255157 A JP 2002-357430 A

However, in the conventional navigation system in which the positional accuracy error is in the range of several tens of meters as described above, the error is too large to be applied to automatic driving.
Needless to say, the self-position evaluation and self-position evaluation of the automatic driving device are common to navigation, and the automatic driving device can be applied to navigation and driving assistance as it is. Accordingly, the following description may be made without clarifying the distinction, but the self-position assessment and the self-posture assessment are explained as being common to both automatic driving and navigation.

For example, in order to identify the lane on the road and realize automatic driving of the vehicle, it is necessary to increase the vehicle position accuracy on the road to an error of several centimeters. This accuracy is close to the accuracy of surveying. As long as this navigation system is used, it is impossible to measure the position continuously in real time with a precision of several centimeters and output it in any way.
In addition, it is not limited to automatic driving, for example, in the case of using a garage for a car, taking off and landing of an aircraft, robot work, navigation, etc., it is necessary to obtain a positional accuracy of about several centimeters in real time, Such a navigation system has not been realized so far.

Therefore, as proposed in the present application and Japanese Patent Application Laid-Open No. 2005-295495, the present inventor prepared a three-dimensional map in advance as a result of earnest research and compared it with a moving image captured by a camera mounted on a moving object. As a result, the camera position and the attitude angle can be obtained with high accuracy, and the three-dimensional position coordinates of the moving object can be displayed with high accuracy by the camera position information.
That is, the present invention has been proposed in order to solve the problems of the above-described conventional techniques, and has been proposed in the above-mentioned International Publication WO2005 / 038402 and Japanese Patent Application Laid-Open No. 2007-114916 related to the present application and the present applicant. By making the 3D map for navigation into a 3D machine map for the machine to determine its position and orientation, instead of being seen by humans, the amount of map data is reduced and further processed by the computer. It is an object of the present invention to provide a three-dimensional map generation device that can improve the calculation efficiency and make a more practical one as a three-dimensional machine map.
Note that the principle is common to the navigation devices disclosed in International Publication WO2005 / 038402 and Japanese Patent Application Laid-Open No. 2007-114916. Therefore, preferably, the three-dimensional mechanical map of the present invention is also premised on being used for the present application and the navigation device of International Publication WO2005 / 038402 and Japanese Patent Application Laid-Open No. 2007-114916.

A first invention according to the present invention is a three-dimensional machine map and has the following characteristics.
The three-dimensional machine map is seen by machines, not humans, and is a completely new concept that does not exist in the past. Because it is a map for the machine to see, it is not the image that the human sees, but it is conveniently processed for the machine to see, and by using this 3D machine map, the position and orientation, and the relative The relationship, the destination, and the direction to the target can be confirmed three-dimensionally.
Since the three-dimensional machine map is for the machine to see, the feature is that the amount of data can be reduced as much as possible than the image. By reducing the amount of data, 3D machine map data can be easily distributed in a narrow band. As a result, the manufacturing cost can be drastically reduced. Therefore, the practicality of the invention according to the above-mentioned International Publication WO2005 / 038402 and Japanese Patent Application Laid-Open No. 2007-114916 related to the proposal of the present application and the applicant of the present application is further increased.
In addition, the relationship between the real-time video obtained from the camera mounted on the vehicle and the 3D machine map can be updated as soon as possible by using a common algorithm as much as possible and using the same format for the comparison data. It can be.

A second invention according to the present invention is a three-dimensional machine map generation device, and the creation of a three-dimensional machine map can be manufactured as inexpensively as possible (by adding a designated object manually) as much as possible. In addition, the three-dimensional machine map can be updated at low cost, easily, with light work, and automatically.
In addition, it is possible to add coordinates and attributes such as road markings, road signs, and guide plates to the three-dimensional machine map.

In order to achieve the above object, the present invention first proposes a new concept map as the first invention.
The concept of a map is something that humans see in the first place, but the 3D machine map referred to in the present invention is not something that humans see on paper or a display, but the machine reads it as data and judges the position and direction. Is.
Therefore, it must be in a form that is easy for the machine to handle. If the machine evolves, it will eventually be possible to see and judge what humans see like humans, but unlike human visual information, machines are still undeveloped, A three-dimensional machine map is generated mainly with a portion having features suitable for viewing. In the 3D machine map, the target space is decomposed into geometrically simple figures that are easy to recognize mechanically, and only the most convenient ones are selected and reconstructed in the 3D space. Construct a 3D machine map.

For example, a 3D machine map is a reconstructed 3D space by analyzing a 3D analysis of straight lines, straight line endpoints, L-shaped corners, crosses, right angles, circles, parallel lines, planes, point distributions, etc. is there.
Also, by classifying these geometrically simplified figures for each attribute, the function and range of use can be expanded.
In addition, the feature of the 3D machine map is that it can be directly coupled to the mechanical system, and it has the feature that it can directly coordinate with other position evaluation means, for example, position data by GPS, and can coordinate coordinates of a plurality of data. .

As a third aspect of the invention, feature points are automatically extracted from an all-round video from an all-around camera mounted on a vehicle, and image processing is performed using a CV video obtained by CV calculation to obtain relative coordinates. The simple geometric figure is extracted and arranged at the position coordinates or recorded. Alternatively, a three-dimensional machine map can be generated by arranging or recording simple geometric figures themselves at the position coordinates. Furthermore, by installing guide markers in the vicinity of the vehicle travel path and recording the guide marker figures on the three-dimensional machine map, the tertiary that can be used in sections where simple geometric figures cannot be obtained, nighttime driving, snowfall driving, etc. An original machine map can be generated.
A navigation device and an automatic driving device can be realized in which this three-dimensional machine map is mounted on the vehicle or received on the vehicle side after receiving the distribution and compared with a real-time image captured by the mounted camera.

A navigation device is a tertiary structure consisting of a 3D machine map and a real geometrical image obtained by reproducing a recording medium from a real image obtained by a camera provided on a moving object to be navigated. Compared to the 3D coordinates of the original machine map, find the point and direction on the 3D coordinates that match the actual image, the position on the 3D coordinates of the camera provided on the moving body, speed, acceleration, viewpoint direction, Among the predetermined items including the three-axis rotation posture, the three-axis rotation speed, and the three-axis rotation acceleration, there is provided a point search navigation device that outputs a plurality of items that are any one or a combination thereof.
In addition, the automatic driving device is a simple geometric figure with visual features that is obtained by reproducing a recording medium from a real image obtained by a camera equipped with a three-dimensional machine map and a moving object to be automatically operated. Compared with the 3D coordinates of the 3D machine map consisting of 3D coordinates, the point and direction on the 3D coordinates that match the actual image are obtained, and the position, speed, Among the predetermined items including acceleration, viewpoint direction, three-axis rotation attitude, three-axis rotation speed, and three-axis rotation acceleration, a plurality of items obtained by combining any of these are output, and the vehicle traveling direction, speed, An automatic driving device that realizes automatic driving by controlling stopping, starting, etc. is configured.
Furthermore, by independently controlling the plurality of suspensions and the four wheels, it is possible to realize automatic driving that enables more delicate vehicle control including the position of the center of gravity and vehicle inclination.

Specifically, the three-dimensional machine map of the present invention is a simplified three-dimensional structure in which a part or most of a target three-dimensional space is replaced with a plurality of geometric simple figures and simplified and reconstructed. A configuration having spatial data, and the simplified three-dimensional spatial data being read by a computer, so that the geometric simple figure outputs related data such as coordinates, postures, and / or attributes corresponding to the three-dimensional space. It is as.

Here, the geometric simple figure refers to a figure having a relatively simple shape such as a point, a line segment, an end point of the line segment, a right quadrangle, a cross, a quadrangle, a circle, an ellipse, or the like, and a combination thereof.
These figures may be two-dimensional or three-dimensional, but they are arranged in a three-dimensional space, so in a three-dimensional machine map, they are always arranged three-dimensionally in three-dimensional coordinates.
These are easier to detect, generate, analyze, compare, etc. by computer image analysis technology than general complex shapes, so the amount of data is lighter than having an actual three-dimensional space as video or CG as it is. It has the excellent feature of becoming.
Further, as will be described later, it is also advantageous in generation, update, comparison, recording, reproduction, and other image processing by a computer.
Further, as will be described later, by installing a graphic that is easily detected by the computer on and near the road, automatic driving can be ensured even under various conditions.
Further, as described later, a method of registering with coordinates turned into CG the geometric simple shapes, cut left image without CG reduction, can also this to it with a three-dimensional machine Map Register with coordinates It is.
Also, as described above, the CV image itself is not cut out as a 3D machine map without being converted into a CG image, but if there is no problem in increasing the data amount, the CV image itself is not cut out and the 3D image is cut out. Needless to say, it can be used as a map. If it is necessary not only for the machine but also for the purpose of human beings to see, it is possible to make the CV video itself a three-dimensional map if it is necessary to confirm the site with CV video.

When replacing and reconstructing an actual three-dimensional space with the plurality of geometric simple figures, a geometric simple figure is used to facilitate reading and writing by a computer. The process up to the stage where a registered geometric simple figure is read by a computer and output is called a three-dimensional machine map.
In addition, a part of an image regarded as a geometric simple figure is cut out from an actual three-dimensional space, and a plurality of image fragments are rearranged to form a three-dimensional machine map, which is then read and written by a computer. In order to make it easier, a plurality of video fragments are registered, the registered video fragments are read by a computer, and the output is output as a 3D machine map.
Geometric simple figures and video fragments are elements that make up a three-dimensional machine map, and can generally be interchanged with each other. However, in this specification, explanation is made without particular distinction, unless the replacement is not possible. To do.

A simple geometric figure registered in a three-dimensional machine map is always arranged in a three-dimensional space. This reflects a part of the target three-dimensional space, and not all of the real space.
This is a map for the computer to read and is not primarily seen by humans. The read geometric simple figure (simple geometric figure) may be read as a figure for reproduction and display as a figure, and since it is not uniquely seen by humans, it indicates the shape and type of the figure. Along with the attribute, it may be read only as its coordinates and orientation. Here, the coordinates and orientation are three-dimensional, have a total of six degrees of freedom, and correspond to a real three-dimensional space.
This can also be said to be the real space seen by the computer. The 3D machine map does not reflect all of the real space, but only what is convenient for computer recognition, registration, etc. is captured and reflected in the 3D machine map.

The three-dimensional mechanical map of the present invention is not only a simple geometric figure extracted from the real space , but also the simplified three-dimensional data reconstructed using the object that matches the purpose of use as the geometric simple figure. It can be read or written by a computer by reconstructing the target real 3D space according to the purpose of use, and all the coordinates, postures, and / or attributes of the figure are related The data can be output together.

According to such a configuration, for example, in addition to the arbitrarily generated three-dimensional machine map according to the present invention, the object for each purpose is replaced with a geometric simplified figure and registered as the simplified three-dimensional data. To do.
Depending on the purpose, the object to be selectively made into a graphic changes, but in any case, it is common to read by a computer and output together with attributes.

Further, the 3D machine map of the present invention is classified as a geometric simple figure for objects necessary for traffic such as traffic signs, road indications, road widths, road guide boards, etc. , particularly for traffic purposes. In addition, the attribute information can be added and arranged in the three-dimensional space.
Furthermore, the three-dimensional machine map of the present invention classifies objects such as traffic signs, road displays (features), road guide boards, etc. as geometric simple figures , especially for the purpose of automatic driving. In addition to the attribute information, it is arranged in addition to the coordinates of the three-dimensional space where the object exists, and in addition, automatic driving such as running center line, road width, number of lanes, road gradient, turning radius etc. that are not directly numerically displayed in reality Alternatively, numerical attribute information necessary for driving assistance can be arranged in correspondence with a three-dimensional space for vehicle travel.

According to such a configuration, in particular, traffic signs, road signs, road guide slopes, etc. that were originally installed for the purpose of being read by the driver are selectively figured out, and in addition, the driving center that is not actually numerically displayed in reality It is possible to provide a three-dimensional machine map that uses numerical attribute information necessary for automatic driving or driving assistance such as lines, road width, number of lanes, road gradient, turning radius, etc. as a three-dimensional map for automatic driving. .
Here, traffic signs, road signs, road guide slopes, etc., may be registered as images as their 3D position or 3D position and direction information. It is also possible to register as In that case, since the classification of the shape is made as the attribute of the target object, the countermeasure object has information on the standardized shape.
Therefore, the shape of the object and its direction in the three-dimensional space can be expressed in the three-dimensional machine map not only by the shape itself but by the attribute and its center coordinates (representative coordinates). Since traffic signs, road signs, road guide boards, and the like have standardized shapes, registering attributes classified into shapes is more advantageous for data processing.
On the other hand, since the attribute can also have a partial video of the object, a part of the actual video of the object is also added, and the target three-dimensional space is reconstructed so that it can be read or written by a computer. 3D machine maps can be provided.

The three-dimensional mechanical map generation device of the present invention includes an in-vehicle all-around video imaging device that acquires an all-around video from a running vehicle, and a CV that acquires an all-around CV image (see Japanese Patent Application Laid-Open No. 2005-295495). An arithmetic device, a geometric simple figure detection device for extracting, selecting, separating, and three-dimensionally determining a position coordinate and a posture from the all-round CV image, and detecting the detected A three-dimensional coordinate adding device for giving a three-dimensional coordinate to a geometric simple figure, an attribute registration device for giving an attribute to the detected geometric simple figure, and a three-dimensional space from the detected geometric simple figure And a three-dimensional machine map recording device for recording a three-dimensional machine map according to the present invention generated by adding other information according to the purpose.

According to the three-dimensional mechanical map generation device of the present invention having such a configuration, the vehicle-mounted all-around video imaging device can load the all-around camera on the vehicle and acquire the all-around video from the running vehicle. . Although the present invention is established even if it is not an all-round image, it is advantageous that it is an all-round image because information can be obtained evenly regarding the omnidirectional elevation angle.
Details of the CV arithmetic unit are described in detail in Japanese Patent Application Laid-Open No. 2005-295495 related to the present applicant. A feature point is automatically extracted from the all-around video, tracked to an adjacent frame, and the tracking data is analyzed to obtain the all-round CV video. The CV calculation and CV calculation device will be described later (see FIGS. 13 to 25).

Geometric simple figure detection device extracts, selects and separates geometric simple figures from all-round CV images using three-dimensional CV values, relies on CV values, and determines the position coordinates and postures in three dimensions. Can be obtained. It is also possible to automatically recognize and track geometric simple figures.
The three-dimensional coordinate adding device gives the three-dimensional coordinates of the geometric simple figure by tracking the detected figure at the same time as the geometric simple figure detection operation.
The attribute registration device gives an attribute to the detected geometric simple figure. If the geometric simple figure is classified in advance, the shape classification becomes the attribute information. If color is required, color information becomes attribute information. If there is a meaning related to traveling, the meaning becomes an attribute. It is also possible to use the representative coordinates of the geometric simple figure as data and the three-dimensional shape as attribute information. If the shape is known, the center coordinate of the shape can be used as an attribute. Here, attributes have a wide range of meanings related to objects.
The three-dimensional machine map recording apparatus reconstructs in the computer a three-dimensional space corresponding to the three-dimensional space to which the original image belongs from the detected geometric simple figure.
The 3D machine map recording device records the reconstructed 3D machine map and integrates other necessary information according to the purpose to complete the 3D machine map according to the purpose.
What has these features is a three-dimensional machine map generator.

Further, the navigation device of the present invention is a navigation device provided with the three-dimensional mechanical map according to the present invention, and analyzes from the real-time video and / or using the CV value of the vehicle acquired by the navigation device, It is configured to be able to take a risk avoidance action by obtaining the mechanical values such as the position and posture of a stationary object in the image and the position, posture, rotation, speed, acceleration and the like of a vehicle other than the moving own vehicle.

Normally, the CV calculation calculates the relationship of movement between the own vehicle and the object of the stationary coordinate system with six degrees of freedom, but mathematically and physically in the relationship between the own vehicle and the moving other vehicle. Is exactly the same, and there is no essential difference mathematically and physically.
Therefore, by extending the definition of the CV value so far to the relationship between the host vehicle and the moving other vehicle, it is possible to define the relative motion between the two persons and between the two persons in the same manner. Using this, the position, rotation, speed, acceleration, and the like of the moving body can be obtained.

The navigation system of the present invention, the real-time video recording, and / or from the recording of the resulting CV values from the navigation device, allows for the dynamic analysis of related objects including the own vehicle at the time of the accident and Can be configured.
According to the three-dimensional mechanical map generating apparatus of the present invention having such a configuration, although there is difficulty in accuracy due to the need for real-time processing, by using the recorded image as a drive recorder, detailed data This is useful for accident analysis.

The navigation system of the present invention, the real space, and or real-time video, as shown in FIG. 12, to synthesize a CG for guiding such as an arrow, or / and a voice, can be configured to navigation Can do.
With such a configuration, navigation can be executed by CG or voice, and a navigation device that is easier to use can be provided.

The navigation device of the present invention can be configured to analyze a real-time video with the same algorithm as that of the three-dimensional map, analyze it, extract a simple geometric figure, and update the three-dimensional machine map.
With such a configuration, the three-dimensional machine map can be updated with a real-time video.
The 3D machine map is generated from the recorded video and needs to be updated. When there is a difference from real-time video while driving, it is possible to apply partial updates to the 3D machine map from real-time video by placing it as the same algorithm.

The navigation system of the present invention, by delivering the information of the own vehicle acquired in another vehicle, may be configured to perform safer navigation by exchanging information with each other.
Since the navigation device according to the present invention is a navigation device that performs advanced determination, communication traveling is possible by transmitting the determination result and the command such as the update information and the route change to other vehicles.

Furthermore, the navigation device of the present invention is capable of displaying the image from the camera installed at the fixed point of the road on the three-dimensional mechanical map by integrating the coordinates and synthesizing it with the relationship between the consistent position and orientation. Can be configured.
With this configuration, the fixed-point video can be understood more concretely by displaying the real-time video from the camera installed at the fixed point on the road with the viewpoint aligned on the 3D machine map. become. Only at this time, the three-dimensional machine map is also an image for viewing.

Further, the automatic driving device of the present invention is an automatic driving device provided with a three-dimensional mechanical map according to the present invention, and is installed fixedly on a vehicle, and an in-vehicle camera device that acquires an image around the vehicle, A real-time video generation device that acquires a real-time video output from a vehicle-mounted camera, a comparison device that compares the real-time video with a three-dimensional mechanical map mounted on the vehicle, and a region where the vehicle is expected to travel Range, a plurality of regions in the real-time video, a part of the geometric component of the three-dimensional machine map obtaining a corresponding position with a part of the geometric component of the three-dimensional machine map, A reproduction search device that adjusts the viewpoint direction to approach and searches within a limited range of the three-dimensional machine map, and a plurality of parts of the real-time video are duplicated of the three-dimensional machine map. The position coordinates of the 3D machine map when it matches the part, the coincidence output device that outputs the rotation coordinates when they match, and the position coordinates and rotation coordinates of each frame of the real-time video when the vehicle is running CV value acquisition device for acquiring the CV value of the real-time video by calculating the vehicle position and posture, speed, acceleration, direction, angular velocity, angular acceleration, etc., and the travel information acquisition device for acquiring in real time And an object information acquisition device for calculating in real time by calculating the position and posture of a vehicle other than the host vehicle and obstacles in the real-time video, obstacles, etc., speed, acceleration, direction, angular velocity, angular velocity, etc. from the CV value The vehicle includes a travel lane, a travel speed, and an automatic travel device that travels by determining a travel direction.

Further, the automatic driving device of the present invention provides a guide marker for selecting a figure or image selected from geometric simple figures constituting the three-dimensional mechanical map in order to use as a coordinate reference when the vehicle is driven by automatic driving. A designated guide marker figure corresponding to a three-dimensional space of a three-dimensional machine map of a figure or a guide marker image and a guide marker having a known shape are set in advance at an appropriate density on the road or in the vicinity of the road, and the tertiary A unified shape guide marker graphic corresponding to the three-dimensional space of the three-dimensional machine map by using the original machine map generator to reflect the guide marker graphic corresponding to the three-dimensional machine map, and the image that is the origin of the guide marker graphic A guide marker image corresponding to the three-dimensional space of the three-dimensional machine map as a guide marker image as it is, the specified guide marker graphic and the unified shape guide marker With attributes registering by figure as a guide marker graphic in, it is constituted by combining these with attributes registered as a guide marker image by image.

The automatic operation device of the present invention, the a guide marker detecting device that detects a guide marker image that is reflected in the real-time video of the road and road near-vehicle camera, the detected said guide marker image and the three-dimensional machine Map The guidance marker map corresponding device that correlates with the upper guide marker graphic and the guidance marker map corresponding device calculate the coordinates and orientation of the vehicle from the coordinates of the corresponding guide marker graphics. A vehicle CV value acquisition device, a marker coordinate reading device that reads the coordinates of a new guide marker graphic on the three-dimensional machine map, and the coordinates of a guide marker that appears one after another as the vehicle progresses, and the coordinates as an expected position The vicinity of the coordinates is searched for in the real-time video, and a new guidance marker video and a new The operation of calculating the coordinates and posture of the vehicle that has traveled ahead of the previous position by the vehicle CV value acquisition device based on the coordinates of the plurality of corresponding guide marker graphics as attributes. It is equipped with an iterative computing device that continues continuously and an output device that continuously outputs the vehicle position and orientation by repeated operations to constantly detect the vehicle position, and acquires the coordinates and orientation of the vehicle. However, it is configured to proceed.

The automatic operation device of the present invention, recording of real-time video that is loaded on the vehicle, and / or based on the recording of the resulting CV value from the automatic operation device, in particular the time of accident, the vehicle It is set as the structure which performs the dynamic analysis about each of the related target object containing and those relations.
Furthermore, the automatic driving device of the present invention is configured to analyze and analyze a real-time image with an algorithm similar to that of the three-dimensional machine map, extract a simple geometric figure, and update the three-dimensional machine map.

In the automatic driving apparatus according to the present invention having the above-described configuration, first, a CV value obtained by analyzing a real-time video by a camera installed in a vehicle is acquired. With this alone, the accuracy is low with respect to the absolute value, but a highly accurate CV value can be obtained with respect to the relative value.
Furthermore, by adding the data of the measurement equipment installed in the vehicle, the accuracy can be further improved by using both of them, and the CV value with the absolute accuracy added can be acquired.
By analyzing the CV value acquired in this way, the three-dimensional position, speed, acceleration, rotational attitude, rotational speed, rotational angular speed, and the like of the vehicle can be acquired.
Further, by analyzing the CV value, it is possible to obtain the position and schematic shape of a stationary object that is around the vehicle and that is involved in traveling.
Further, by analyzing the CV value, it is possible to obtain the approximate shape, three-dimensional position, posture, rotational speed, speed, rotational speed, acceleration, rotational angular speed, etc. of the moving body other than the vehicle.
Based on the CV value acquired from these real-time images, an automatic driving device capable of optimizing and controlling the vehicle and maintaining safe operation can be realized.

Thus, according to the automatic driving device of the present invention, the GPS, the vertical meter installed in the vehicle, the CV value obtained by directly analyzing the real-time video by the camera installed in the vehicle, etc. The accuracy can be improved by adding data of measuring instruments such as IMU and gyro. Further, by analyzing the CV value and the added measuring device data, a more accurate CV value can be obtained.
Then, the highly accurate three-dimensional position, speed, acceleration, rotational attitude, rotational speed, rotational speed, rotational angular velocity, etc. of the vehicle are obtained from the highly accurate CV value, and the CV value and the added measurement device data are analyzed. Then, a position and a schematic shape of a stationary object that is around the vehicle and related to traveling are acquired, and further, a CV value and the added measurement device data are analyzed, and a schematic shape of a moving body other than the vehicle 3D position, posture, rotational speed, speed, rotational speed, acceleration, rotational angular speed, etc. are acquired.
And based on these various physical mechanical data acquired, the said vehicle can be optimally controlled and a safe driving | operation can be hold | maintained.
Note that a method (automatic driving method) for actually controlling the vehicle by the generated control signal can be realized by a known technique, and thus detailed description thereof is omitted.

Here, the CV value is defined for all images output from the moving camera, and is defined for the all-round image acquired when the three-dimensional machine map is generated. Is a CV value defined in a real-time video installed in the CV value, and is a CV value calculated in real time.
In addition, the CV value is not limited to the flow of feature point extraction, feature point tracking, and CV calculation, which are defined first, as a broad CV value. Even when calculated from matching, the camera position (vehicle position) and posture are defined for all cases.

As described above, the CV calculation obtains the motion relationship between the subject vehicle and the object in the stationary coordinate system with six degrees of freedom, but mathematically and physically between the subject vehicle and the moving other vehicle. This is exactly the same, and there is no essential difference mathematically and physically.
Therefore, by extending the definition of the CV value so far to the relationship between the host vehicle and the moving other vehicle, it is possible to define the relative motion between the two persons and between the two persons in the same manner. By using this, the position, rotation, speed, acceleration, etc. of the moving body can be obtained, and the vehicle can be controlled safely using these signals.

In addition, the automatic driving device of the present invention is capable of analyzing the positional relationship of related objects including the own vehicle at the time of an accident by leaving a record of real-time video and a record of the CV value obtained from the automatic driving device. It can be set as the structure which performs a mechanical analysis.
Although real-time processing has difficulty in accuracy, using video recorded like a drive recorder can acquire detailed data with high accuracy, which is useful for accident analysis.

In addition, the navigation device of the present invention can be configured to synthesize guidance CG such as arrows or the like in real space and / or real-time video (see FIG. 12) or / and to navigate by voice. .
With such a configuration, navigation can be executed by CG or voice, and a navigation device that is easier to use can be provided.

In addition, the automatic driving device of the present invention analyzes a real-time image with the same algorithm as that of the three-dimensional map, extracts a simple geometric figure, detects a difference from the three-dimensional mechanical map, If it exists, it can be set as the structure which updates the said three-dimensional map.
With such a configuration, the three-dimensional machine map can be updated with a real-time video.
The 3D machine map is generated from the recorded video and needs to be updated. While traveling, when there is a difference between real-time video, in your Kukoto as the same algorithm, it is possible to make a partial update to the three-dimensional machine map from the real-time video.
A three-dimensional machine map can be composed of simple geometric figures. In this case, an image fragment that is a source of a simple geometric figure must exist in the original video. However, there are cases in which there is no video that is the basis of a simple geometric figure in an image fragment.
Therefore, in the present invention, a real-time image is analyzed by an algorithm similar to that of a three-dimensional map, analyzed, and a simple geometric figure is extracted.

Moreover, the automatic driving apparatus of this invention can be set as the structure which performs safer automatic driving | operation by exchanging information mutually by distributing the acquired information of the own vehicle to other vehicles.
Since the automatic driving apparatus according to the present invention is also a navigation apparatus that performs advanced determination and control, communication traveling is possible by transmitting a determination result, a command such as update information or a course change to another vehicle.

Furthermore, the automatic driving apparatus of the present invention can capture the image from the camera installed at the fixed point on the road on the three-dimensional mechanical map by integrating the coordinates and synthesizing them in the relationship between the consistent position and orientation. It can be set as the structure which can be performed.
With this configuration, the real-time video from the camera installed at the fixed point on the road is decomposed into simple geometric figures, and the fixed point video is captured by matching the viewpoint on the 3D machine map. Can be understood more specifically.

Note that the present invention as described above utilizes the CV image technology (Japanese Patent Laid-Open No. 2005-295495) based on an all-round image proposed by the present inventor.
The CV calculation and the CV value (Japanese Patent Laid-Open No. 2005-295495), which are the basis of the present invention, are born from a spherical image of a moving image obtained by an all-round camera. By treating the moving image as a spherical image, the concept of focal length and optical axis is released, and the moving image is converted into a three-dimensional model by reverting to the principle of triangulation while maintaining a two-dimensional surface in a form suitable for vision. It became easy to handle. Furthermore, by introducing a simplified concept of CV value criteria in moving images, the moving image processing is simplified and made three-dimensional with the surface, making it easy to divide and track moving images and to achieve high accuracy. That made a lot of progress.

According to the three-dimensional machine map, the three-dimensional machine map generation device, the navigation device, and the automatic driving device of the present invention, the three-dimensional position of a moving body such as an automobile, a heavy machine, and an aircraft can be evaluated, the traveling direction of the moving body, A three-dimensional map for guiding the situation, etc., and its three-dimensional map generator, searching the three-dimensional space for the current position or future position of the moving body that travels and navigates, and the traveling direction and vehicle posture of the moving body etc., human rather than determined look, by the machine to determine, it is possible to determine the current status and future status of the mobile body moving with high precision,

Hereinafter, preferred embodiments of a three-dimensional machine map, a three-dimensional machine map generation device, and a navigation device and an automatic driving device including the three-dimensional machine map according to the present invention will be described with reference to the drawings.
Here, the following three-dimensional machine map generation device, navigation device, automatic driving device, and other devices of the present invention are all realized by processing, means, and functions executed by a computer in accordance with instructions of a program (software). The The program sends commands to each component of the computer, and performs the following predetermined processing and functions such as automatic extraction of reference points and feature points from the video, automatic tracking of the extracted reference points, Calculation of three-dimensional coordinates, calculation of CV (camera vector) value, etc. are performed.
Thus, each process and means in the present invention are realized by specific means in which the program and the computer cooperate.
Note that all or part of the program is provided by, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and the program read from the recording medium is installed in the computer and executed. The The program can also be loaded and executed directly on a computer through a communication line without using a recording medium.

  Hereinafter, as an embodiment of the present invention, image processing is performed using CV computation from a full-circle video, a three-dimensional machine map is generated, and a normal wide-angle video (not a full-circumference video) by an on-vehicle normal video camera is used. The case where the navigation apparatus and automatic driving apparatus which concern on the said international publication WO2005 / 038402 and Unexamined-Japanese-Patent No. 2007-114916 is implement | achieved using the three-dimensional mechanical map produced | generated by the apparatus is shown.

[3D machine map]
First, an embodiment for generating a three-dimensional machine map from an all-round image will be described.
The goals on the machine map and real-time video are as follows.
○ Targets on the 3D machine map • The 3D machine map is viewed by machines, not humans. → The machine confirms the relative relationship between the position and posture, surroundings, and the direction to the target in three dimensions.
・ Reduce the data volume of 3D machine maps as much as possible. (Enable map data distribution)
・ Creation of 3D mechanical maps is automated as much as possible by generating them from the CV video database (+ adding specified objects manually).
・ Updating 3D machine maps is cheap, easy, light work, and even automatic.
-Add coordinates and attributes such as road markings, road signs, and guides to 3D machine maps.
・ The real-time video side and the 3D machine map are made as common as possible, and changes and update parts can be updated immediately. (Goal)

○ Using a real-time image obtained by a normal wide-angle camera, which is not a real image from the target / vehicle on the real-time image side or an all-round image, a guidance instruction CG is synthesized and displayed in the image.
・ Put tags on the 3D machine map and real-time video to add notes.
・ Addition of danger avoidance action function by 3D machine map and real-time video analysis. (Goal)
・ Vehicle travel record and record analysis. (Vehicle CV value recording and recording analysis)
・ Accident analysis based on vehicle travel record analysis data.
・ Update 3D machine maps with vehicle travel record analysis data.
・ Data sharing with other vehicles by automatic distribution of update data.
・ Information acquisition from fixed point camera.

[3D map generator]
Hereinafter, an embodiment of a three-dimensional map generation apparatus for generating a three-dimensional machine map according to the present invention will be described with reference to FIG.
FIG. 1 is a block diagram showing a configuration of a 3D map generation apparatus 110 according to an embodiment of the present invention.
In order to generate a three-dimensional machine map, the all-around video (see FIG. 5) can be acquired from a running vehicle by mounting an all-around camera on the vehicle by the in-vehicle all-around video imaging device 111.
The acquired all-round image is Mercator (see FIG. 11), and FIG. 5 shows a perspective view of a portion of the image that is projected onto a plane. FIG. 5 as an example for explaining the shooting position is later taken to be taken from a vehicle on the right lane of two lanes on one side.

Although the present invention can be established even if it is not an all-round image, it is advantageous that it is an all-round image because information can be obtained equally with respect to the omnidirectional full depression angle and elevation angle.
In addition, by loading GPS, IMU, gyroscope, etc. to acquire position information, the approximate position and approximate attitude become known, which facilitates subsequent processing. However, if there is a highly accurate map to be calibrated, in principle, GPS, IMU, gyro, etc. may be omitted.
In addition, since it is advantageous in calculation to know the approximate position in the initial manufacturing process, it is desirable to obtain data by loading these GPS, IMU, gyro, and the like simultaneously.
Also, since the vertical is important information in the subsequent processing, it is desirable to load a vertical meter.

The details of the CV arithmetic unit 112 are described in detail in the above-mentioned Japanese Patent Application Laid-Open No. 2005-295495 related to the applicant of the present application. By forming a triangle to be tracked and calibrating with the camera movement base and tracking point, and analyzing the tracking data, it is possible to obtain a full-circle CV image having the three-dimensional coordinates of the feature points and the camera position and orientation. it can. The CV value represents the position and orientation of the camera with 6 degrees of freedom.
That is, the respective coefficients of 6 degrees of freedom of x, y, z, θx, θy, and θz are determined by calculation for all frames.
The CV calculation and the CV calculation device will be described in detail later (see FIGS. 13 to 25).

The geometric simple figure detection device 113 extracts, selects, separates and isolates a geometric simple figure from the all-round CV image based on the CV value and performs a three-dimensional analysis, and the position coordinates thereof. And the posture can be obtained three-dimensionally.
Here, the geometric simple figure is a point, a line, an end point, a surface, a right-angled quadrature, a circle, an arc, or the like. In this embodiment, a point, a line, or an end point is detected.
For example, from daytime images, straight lines and straight line endpoints are extracted mainly from road markings, white edges of buildings, window frames, etc., but from night images, points are mainly extracted from street lights and lights in buildings. Will do.
Therefore, by simultaneously reconstructing points, lines, and end points as the geometric simple figure, it can be used as a three-dimensional machine map day and night.

The three-dimensional coordinate adding device 114 detects the three-dimensional position and orientation from the tracking data of the graphic detected at the same time as the geometric simple graphic detection operation (see FIG. 6) by the same principle as the CV calculation, and the third order. The original coordinates (see FIG. 7) are given.
FIG. 6 is an image obtained by projecting the entire circumference image on a plane, and the straight line and the end point of the straight line are automatically detected there as a geometric simple figure. FIG. 7 shows three-dimensional coordinates obtained by tracking the automatically detected geometric simple figure on a plurality of adjacent frames. This is already part of the 3D machine map.
Further, since the coordinates obtained by the CV calculation are relative coordinates, absolute coordinates are added by post-processing from known data on the map or high-accuracy GPS data.

The attribute registration device 115 gives an attribute to the detected geometric simple figure.
In the present embodiment, the PRM recognition (US Pat. No. 6,614,932, International Publication WO02 / 01505) technology according to the present applicant is recognized by comparing with a pattern prepared in advance, and the geometric simple figure can be classified into shapes. For example, shape classification is attribute information. If color is required, color information becomes attribute information.
In addition, an object necessary for road traffic, such as a traffic sign, can be recognized by the PRM technique and its attributes can be registered.
Here, lines and end points are registered as attributes (see FIG. 7). It is also possible to use the representative coordinates of a geometric simple figure as data and the three-dimensional shape as attribute information. Here, attributes have a broad meaning.

PRM is an abbreviation for Parts Reconstruction Method (3D object recognition method), and is a technique for recognizing an object developed by the inventors of the present application (see International Publication WO02 / 01505).
Specifically, the PRM technology prepares all the shapes and attributes of the object to be predicted in advance as parts (operator parts), compares these parts with actual live-action images, and selects matching parts. Technology that recognizes objects. For example, “parts” of objects required for traveling vehicles and the like are lanes as road markings, white lines, yellow lines, crossing roads, speed signs as road signs, guide signs, etc. Therefore, the recognition can be easily performed by the PRM technology. In addition, even when searching for an object in a CV video, it is possible to limit the expected three-dimensional space in which the object exists to a narrow range, thereby improving recognition efficiency.

For example, if it is a traffic light, it is “detection” just by acquiring its three-dimensional shape, and it is “specific” if the coordinates and orientation are known. It can be said that the attributes were understood (recognized) as they changed and understood the meaning of each traffic.
Note that not all monitored objects (predicted objects, obstacles, suspicious objects, etc.) are recognized by PRM recognition. That is, an object whose information does not exist in the database to be stored cannot be recognized. However, this is not a problem. Even if it is a human being, it is the same as not being able to recognize what is not in memory. This is because recognition is obtained by comparing itself with memory of something close to itself.

The three-dimensional machine map recording device 115 reconstructs a three-dimensional space corresponding to the original three-dimensional space from the detected geometric simple figure (see FIGS. 3 and 4).
This is already a three-dimensional machine map, but the three-dimensional machine map recording device 115 records it according to a format that uses the reconstructed three-dimensional machine map, and integrates other necessary information according to the purpose. This completes the 3D machine map according to the purpose.
What has these features is a three-dimensional machine map generator.
FIG. 3 is a bird's-eye view of a three-dimensional machine map, and FIG. 4 is a vehicle view at the time of image acquisition. When a 3D machine map is visually represented, it can be viewed from only one direction. Originally, a 3D machine map is determined by a machine (computer) and can be converted into all directions and all viewpoints. have.

  In addition, by providing the three-dimensional map as described above, the navigation device (see FIG. 2) according to the present invention described later enables the image geometry in the range where the information recorded on the recording medium is observed from the moving body. Types of geometric simple features and their three-dimensional coordinates, three-dimensional arrangement of two-dimensional images of small areas including visual geometric simple features, their three-dimensional coordinates, and simple geometric features The shape of the object including the figure and its three-dimensional coordinates, the peripheral image necessary for the movement of the moving body other than the visual feature points, the shape and three-dimensional coordinates of the CG, the road on which the moving body moves, the vehicle traveling Including images of roads, planned routes, etc., CG and its three-dimensional shape and its three-dimensional coordinates, each of which includes any one or a combination thereof or all of them or their attribute information, Recorded with the machine map .

The three-dimensional machine map generated by the three-dimensional map generation apparatus 110 according to the present embodiment as described above is, for example, as shown in FIGS.
However, since the 3D machine map is not for human viewing, the ones shown here are expressed in a visible format. Not what you want.

[Navigation device]
Next, according to an embodiment of the navigation device of the present invention using the above-described three-dimensional machine map, according to the method proposed in the above-mentioned International Publication WO2005 / 038402 and Japanese Patent Application Laid-Open No. 2007-114916 related to the present applicant, FIG. Will be described with reference to FIG.
As shown in the figure, the navigation device 120 according to the present embodiment includes the above-described three-dimensional machine map 121 (see FIGS. 3 and 4) and a real-time video 122 (FIG. 8) from a camera loaded on a moving vehicle. Are compared in the comparison operation device 123.
As a result, the machine automatically determines the position and orientation of the vehicle (see FIG. 9), and the position / direction rating output device 124 identifies the current position on the three-dimensional machine map (see FIG. 10). The device is realized.

FIG. 8 is a real-time video from a camera mounted on a moving vehicle, and this real-time video is compared with a three-dimensional machine map.
Here, the video for creating the three-dimensional mechanical map shown in FIG. 8 is the right lane, but it is important that the real-time video shown as an example is the left lane.
Regardless of the lane at the time of 3D machine map acquisition, it is important that the shooting position of the real-time video is freely selected.

FIG. 9 shows a situation in which the machine automatically determines the position and orientation of the vehicle and evaluates the position and direction.
In FIG. 9, first, a three-dimensional machine map is projected onto a real-time image.
Next, the horizontal line of the 3D machine map and the edge component of the image are matched.
On the adjusted 3D machine map and the real-time image, the vertical line of the 3D machine map and the edge component of the image are matched.
By repeating this operation and obtaining the coincidence point between the 3D machine map and the real-time video, the current position and orientation are acquired from the coordinates of the coincident 3D machine map.
FIG. 10 visually shows the result of actually evaluating the position and orientation of successive frames in a three-dimensional machine map. However, in actuality, it is done as data processing inside the computer and is not visible.

In the vehicle control device 125, the moving path of the moving body is registered in advance in the three-dimensional machine map using image processing technology, and the three-dimensional machine map and the real-time video by the camera loaded on the actual moving body are displayed. Is compared with the three-dimensional machine map, the three-dimensional coordinates indicating the camera position of the moving object can be guided with higher accuracy than the GPS system.
In addition to the own vehicle, other vehicles appearing in the video can also be measured by extracting the feature points and tracking the vehicle's position, speed, acceleration, and posture. The rotational posture of the body can also be acquired, and using this, control such as vehicle collision avoidance and danger avoidance can be performed.

Further, the vehicle control device 125 analyzes the real-time video, obtains a CV value by a three-dimensional mechanical map and CV calculation, analyzes the CV value, and analyzes the position, rotation posture, acceleration, speed, and The position, rotation, acceleration, and speed of other vehicles in the video can be obtained and an automatic crisis avoidance operation can be performed.
Furthermore, when an accident occurs unfortunately, it is possible to obtain not only the own vehicle but also all the positions, rotations, accelerations, and speeds of the vehicle related to the accident in the image from the recorded image of the in-vehicle camera. It will be possible.

  As described above, the navigation device 120 according to the present embodiment compares the real-time video captured from the on-vehicle camera with the on-vehicle three-dimensional mechanical map by using the computer, obtains the correspondence with the real space, and is mounted on the vehicle from the correspondence relationship. A navigation device having a three-dimensional mechanical map that is obtained by calculating the three-dimensional position and posture of a camera and determining the vehicle position and posture is realized.

Here, the real-time video is not an all-around video, but a real-time video is obtained by using one normal wide-angle camera. Judging from the performance and accuracy, it is clear that the omnidirectional video image from the omnidirectional camera is superior. However, in order to spread it as an industry, this embodiment adopts an inexpensive normal camera and a normal wide-angle image except for special applications.
In addition, since a real-time video is used, it is also used as a video for viewing this, and a guidance CG is displayed on this real-time video (guidance display 126), or an information tag is pasted to put information on the information tag. You can type or record and recall.
FIG. 12 shows an example in which a guidance CG is displayed on the video.
A 3D machine map is for the machine to see, but this is clearly for humans to see.
As mentioned above, although an example of the navigation apparatus of this embodiment was shown, the navigation apparatus which concerns on this invention is not limited only to the said embodiment.

[CV calculation]
Next, the CV calculation which is the premise of the three-dimensional machine map generation process in the three-dimensional machine map generation apparatus of the present invention as described above will be described with reference to FIGS.
The CV calculation is one of the methods for obtaining the CV value, and the result obtained by the CV calculation is referred to as CV value and CV data. The notation CV is an abbreviation for camera vector: CameraVector, and the camera vector (CV) is a value indicating a three-dimensional position and a three-axis rotation posture of a camera such as a video camera that acquires an image for measurement or the like.
The CV calculation acquires a moving image (video image), detects a feature point in the image, tracks it in a plurality of adjacent frames, and creates a triangle formed by the camera position and the tracking locus of the feature point in the image. The three-dimensional position of the camera and the three-axis rotational posture of the camera are obtained by generating a large number of images and analyzing the triangle.

In the CV calculation, it is an important characteristic that, in the process of obtaining the CV value, three-dimensional coordinates are simultaneously obtained for feature points (reference points) in the video at the same time.
Further, the CV value obtained by calculation from the moving image is obtained simultaneously with the three-dimensional camera position and the three-dimensional camera posture corresponding to each frame of the moving image. Moreover, in principle, the characteristic that a CV value is obtained corresponding to an image with a single camera is an excellent feature that can be realized only by CV calculation.
However, the CV value can also be obtained from, for example, measurement means (GPS and gyro, IMU, or the like) using another method. Therefore, acquisition of the CV value is not limited to the case of CV calculation.
In addition, when acquiring a CV value by GPS and a gyro, IMU, etc., in order to acquire each frame of a moving image and its 3D camera position and 3D camera posture at the same time, an image frame and measurement It is necessary to synchronize the sampling time with high accuracy and completely.

CV data obtained by calculation from a moving image is a relative value when not processed, but if it is a short section, three-dimensional position information and three-axis rotation angle information can be acquired with high accuracy.
Also, when CV data is acquired from an image, the acquired data is a relative value, and calibration at a known point is necessary to convert it to an absolute value. It is difficult to realize by other methods that can measure the positional relationship with an arbitrary object in the image by the point three-dimensional position coordinates and orientation, or by the feature point newly specified and acquired in the image. It has excellent characteristics.
In addition, since the CV value corresponding to the image is obtained, the compatibility with the image is good, and as long as the CV value is acquired from the image, there is no contradiction in the image. The CV calculation that can directly determine the camera position and its three-axis rotation posture from is suitable for in-image measurement and in-image survey.
In the present invention, a three-dimensional machine map is generated based on the CV value data obtained by the CV calculation.

[CV calculation unit]
The CV calculation is performed by the CV calculation unit 2 shown in FIG. 13 that functions as the CV calculation device 112 (see FIG. 1) of the three-dimensional machine map generation device 110 described above.
As shown in FIG. 13, the CV calculation unit 2 obtains a CV value by performing a predetermined CV calculation process on the video image input from the all-around video image unit 1. Feature point extraction unit 2-1, feature point correspondence processing unit 2-2, camera vector calculation unit 2-3, error minimization unit 2-4, 3D information tracking unit 2-5, high-precision camera And a vector calculation unit 2-6.

First, in principle, any video can be used for CV calculation. However, in the case of a video with a limited angle of view, the video is interrupted when the viewpoint direction is moved. It is desirable to treat a peripheral image (see FIGS. 14 to 16) or a wide-angle image as a part of the entire image. Note that a moving image is similar to a continuous still image and can be handled as a plurality of still images.
In general, a moving image recorded in advance is used as the video, but it is also possible to use a video captured in real time in accordance with the movement of a moving body such as an automobile.

Therefore, in the present embodiment, as an image used for the CV calculation, an all-around image (see FIGS. 14 to 16) obtained by capturing the entire 360 ° circumference of a moving body such as a vehicle, or a wide-angle image close to the all-around image. Can be handled as a part of the all-round video by developing the whole-round video in the direction of the viewpoint.
The planar development of the all-around video is to express the all-around video as a normal image in perspective. Here, the term “perspective” refers to the fact that the entire perimeter image itself is displayed by a method different from the perspective method, such as Mercator projection and spherical projection projection (see FIG. 16). This is because it can be converted and displayed into a normal perspective image by performing planar development display like the equidistant projection.

In order to generate an all-round video in the all-round video image section 1, first, as shown in FIGS. 14 and 15, a travel is performed for the purpose of obtaining CV value data using the all-round video camera 1-1. With the all-around video camera 1-1 fixed to the moving body 1-1a such as a vehicle, the periphery of the moving body is photographed along with the movement of the moving body 1-1a.
Note that the moving body 1-1a can be provided with, for example, a position measuring device composed of a GPS device alone or an IMU device that acquires absolute coordinates for the purpose of acquiring the position coordinates.
The all-round video camera 1-1 mounted on the moving body 1-1a may have any configuration as long as it captures and acquires a wide range of images. For example, a camera with a wide-angle lens or a fisheye lens There are a moving camera, a fixed camera, a camera in which a plurality of cameras are fixed, a camera that can rotate around 360 degrees, and the like. In this embodiment, as shown in FIGS. 14 and 15, a plurality of cameras are integrally fixed to a vehicle, and an all-round video camera 1-1 that captures a wide range of images as the moving body 1-1a moves. I use it.

Then, according to the all-round video camera 1-1 as described above, as shown in FIG. 15, by being installed on the ceiling of the moving body 1-1a, a 360-degree all-around video of the camera is displayed. Images can be taken simultaneously with the camera, and a wide range of video can be acquired as moving image data by moving the moving object 1-1a.
Here, the omnidirectional video camera 1-1 is a video camera that can directly acquire the omnidirectional video of the camera, but can be used as the omnidirectional video if more than half of the entire circumference of the camera can be acquired as video.
Even in the case of a normal camera with a limited angle of view, the accuracy of the CV calculation is reduced, but it can be handled as a part of the entire peripheral video.

Note that the wide-range video imaged by the all-around video camera 1-1 can be pasted as a single image on a virtual spherical surface that matches the angle of view at the time of imaging.
The spherical image data pasted on the virtual spherical surface is stored and output as spherical image (360 degree image) data pasted on the virtual spherical surface. The virtual spherical surface can be set to an arbitrary spherical shape centered on the camera unit that acquires a wide range image.
FIG. 16A is an appearance image of a virtual spherical surface to which a spherical image is pasted, and FIG. 16B is an example of a spherical image pasted to the virtual spherical surface. FIG. 4C shows an example of an image obtained by developing the spherical image of FIG.

Then, the all-round video image generated and acquired as described above is input to the CV calculation unit 2 to obtain CV value data (see FIG. 13).
In the CV calculation unit 2, first, the feature point extraction unit 2-1 has a sufficient number of moving image data captured and temporarily recorded by the all-around video camera 1-1 of the all-around video image unit 1. Feature points (reference points) are automatically extracted.
The feature point correspondence processing unit 2-2 automatically obtains the correspondence relationship by automatically tracking the automatically extracted feature points in each frame image between the frames.
The camera vector calculation unit 2-3 automatically calculates a camera vector corresponding to each frame image from the three-dimensional position coordinates of the feature points for which the correspondence relationship has been determined.
The error minimizing unit 2-4 performs statistical processing so that the distribution of the solution of each camera vector is minimized by overlapping calculation of a plurality of camera positions, and automatically calculates the camera position direction subjected to the error minimizing process. decide.

The 3D information tracking unit 2-5 positions the camera vector obtained by the camera vector computing unit 2-3 as an approximate camera vector, and based on the 3D information sequentially obtained as part of the image in the subsequent process, Partial 3D information contained in a plurality of frame images is automatically tracked along the images of adjacent frames. Here, three-dimensional information (three-dimensional shape) is mainly three-dimensional distribution information of feature points, that is, a collection of three-dimensional points, and this collection of three-dimensional points constitutes a three-dimensional shape. To do.
Based on the tracking data obtained by the three-dimensional information tracking unit 2-5, the high-precision camera vector computing unit 2-6 obtains a camera vector with higher accuracy than the camera vector obtained by the camera vector computing unit 2-3. Generate and output.
Then, the camera vector obtained as described above is input to a three-dimensional machine map generation device 200 described later, and used for CV tag generation and pasting processing.

There are several methods for detecting a camera vector from feature points of a plurality of images (moving images or continuous still images). In the CV calculation unit 2 of the present embodiment shown in FIG. By automatically extracting a number of feature points and automatically tracking them, a three-dimensional vector and a three-axis rotation vector of the camera are obtained by triangulation geometry and epipolar geometry.
By taking a large number of feature points, camera vector information is duplicated, so that a unit triangle composed of the coordinates of three points can be obtained from many feature points more than ten times, and duplicate information Therefore, it is possible to obtain a more accurate camera vector by minimizing the error.

A camera vector is a vector of six degrees of freedom possessed by a moving camera.
In general, a stationary three-dimensional object has six degrees of freedom of position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) of the respective coordinate axes. Therefore, the camera vector refers to a vector of six degrees of freedom of the camera position coordinates (X, Y, Z) and the rotation angles (Φx, Φy, Φz) of the respective coordinate axes. When the camera moves, the direction of movement also enters the degree of freedom, which can be derived by differentiation from the above six degrees of freedom.
Thus, the detection of the camera vector in the present embodiment means that the camera takes six degrees of freedom values for each frame and determines six different degrees of freedom for each frame.

Hereinafter, a specific camera vector detection method in the CV calculation unit 2 will be described with reference to FIG.
First, the image data acquired by the all-around video camera 1-1 of the above-described all-around camera video unit 10 is input to the feature point extracting unit 2-1 of the CV calculating unit 2 indirectly or directly, and the feature points are obtained. A point or a small area image that should be a feature point is automatically extracted from the appropriately sampled frame image by the extraction unit 2-1, and a feature point between a plurality of frame images is extracted by the feature point correspondence processing unit 2-2. Is automatically determined.
Specifically, more than a sufficient number of feature points that are used as a reference for detecting a camera vector are obtained. An example of feature points between images and their corresponding relationships are shown in FIGS. In the figure, “+” is a feature point that is automatically extracted, and the correspondence is automatically tracked between a plurality of frame images (see correspondence points 1 to 4 shown in FIG. 19).
Here, for feature point extraction, as shown in FIG. 20, it is desirable to specify and extract a sufficiently large number of feature points in each image (see circles in FIG. 20). For example, about 100 feature points are extracted. Extract points.

Subsequently, the camera vector calculation unit 2-3 determines the three-dimensional coordinates of the extracted feature points by calculation, and calculates the camera vector based on the three-dimensional coordinates. Specifically, the camera vector calculation unit 2-3 has a sufficient number of feature positions that exist between consecutive frames, a position vector between moving cameras, a camera three-axis rotation vector, and each camera position. Relative values of various three-dimensional vectors such as vectors connecting the feature points with each other are continuously calculated.
In this embodiment, for example, camera motion (camera position and camera rotation) is calculated by solving an epipolar equation from the epipolar geometry of a 360-degree all-round image. In addition, if this is described as triangulation, a device that measures the azimuth and depression angle elevation angle is mounted on the vehicle, and a plurality of target survey points are obtained from data obtained by measuring the same object with the moving surveying device, This is the same as the case of obtaining the coordinates of each target point and the trajectory of the surveying instrument.

Images 1 and 2 shown in FIG. 19 are images obtained by developing a 360-degree all-round image with Mercator development. θ2, φ2). The spatial coordinates of each camera are z1 = (cos φ1 cos θ1, cos φ1 sin θ1, sin φ1), z2 = (cos φ2 cos θ2, cos φ2 sin θ2, sin φ2). If the camera movement vector is t and the camera rotation matrix is R, z1T [t] × Rz2 = 0 is the epipolar equation.
By providing a sufficient number of feature points, t and R can be calculated as a solution by the method of least squares by linear algebra calculation. This calculation is applied to a plurality of corresponding frames.

Here, it is preferable to use a 360-degree all-round image as an image used for the calculation of the camera vector.
The image used for the camera vector calculation may be any image in principle, but the all-around video is extremely advantageous because the direction of the object is expressed as it is in latitude and longitude. An image using a wide-angle lens such as a 360-degree all-round image shown in FIG. 19 selects many feature points and can be easily traced over a plurality of frames. Further, even when an image is obtained by a narrow-angle lens, it is effective because the feature point does not escape from the field of view when shooting from various directions while moving around the same object.
Therefore, in this embodiment, a 360-degree all-round image is used for the CV calculation, whereby the tracking distance of the feature points can be increased, and a sufficiently large number of feature points can be selected. It becomes possible to select a feature point convenient for each short distance. In addition, when correcting the rotation vector, the calculation process can be easily performed by adding the polar rotation conversion process. As a result, a calculation result with higher accuracy can be obtained.
In FIG. 19, in order to facilitate understanding of the processing in the CV calculation unit 2, a 360-degree all-round spherical image obtained by combining images taken by one or a plurality of cameras is developed using a Mercator projection called a map projection. However, in an actual CV calculation, it is not necessarily a developed image by the Mercator projection.

Next, in the error minimizing unit 2-4, a plurality of vectors based on each feature point are obtained by a plurality of calculation equations based on a plurality of camera positions corresponding to each frame and the number of feature points. Then, statistical processing is performed so that the distribution of the position of each feature point and the camera position is minimized to obtain a final vector. For example, the optimal solution of the least square method is estimated by the Levenberg-Marquardt method for multiple frame camera positions, camera rotations, and multiple feature points, and errors are converged to determine the camera position, camera rotation matrix, and feature point coordinates. .
Further, feature points having a large error distribution are deleted, and recalculation is performed based on other feature points, thereby improving the accuracy of computation at each feature point and camera position.
In this way, the position of the feature point and the camera vector can be obtained with high accuracy.

21 to 23 show examples of the three-dimensional coordinates of feature points and camera vectors obtained by CV calculation. FIG. 21 to FIG. 23 are explanatory diagrams showing the vector detection method by the CV calculation of the present embodiment, and showing the relative positional relationship between the camera and the object obtained from a plurality of frame images acquired by the moving camera. FIG.
FIG. 21 shows the three-dimensional coordinates of the feature points 1 to 4 shown in the images 1 and 2 in FIG. 19 and the camera vector (X, Y, Z) that moves between the images 1 and 2.
22 and 23 show the positions of feature points obtained from a sufficiently large number of feature points and frame images and the position of the moving camera. In the figure, a circle mark that continues in a straight line at the center of the graph is the camera position, and a circle mark located around the circle indicates the position and height of the feature point.

Here, in the CV calculation in the CV calculation unit 2, in order to obtain more accurate three-dimensional information of the feature point and the camera position at a high speed, as shown in FIG. Set feature points and repeat multiple operations.
Specifically, the CV calculation unit 2 automatically detects feature points having image characteristics in the image, and obtains corresponding points of the feature points in each frame image. And the n + m-th two frame images Fn and Fn + m are used as unit calculations, and unit calculations with n and m appropriately set can be repeated.
m is the frame interval, and the feature points are classified into a plurality of stages according to the distance from the camera to the feature point in the image. The distance from the camera to the feature point is set so that m becomes larger. It is set so that m is smaller as the distance to is shorter. This is because the change in position between images is less as the distance from the camera to the feature point is longer.

Then, while sufficiently overlapping the classification of the feature points by the m value, a plurality of stages of m are set, and as n progresses continuously with the progress of the image, the calculation proceeds continuously. Then, the overlap calculation is performed a plurality of times for the same feature point in each step of n and m.
In this way, by performing unit calculation focusing on the frame images Fn and Fn + m, a precise camera vector is calculated over a long time between each frame sampled every m frames (frames are dropped). However, in m frames (minimum unit frames) between the frame images Fn and Fn + m, a simple calculation that can be performed in a short time can be performed.

If there is no error in the precision camera vector calculation for every m frames, both ends of the camera vector of the m frames overlap with the Fn and Fn + m camera vectors that have been subjected to the high precision calculation. Accordingly, m minimum unit frames between Fn and Fn + m are obtained by a simple calculation, and both ends of the camera vector of the m minimum unit frames obtained by the simple calculation are Fn and Fn + m obtained by high precision calculation. The scale adjustment of m consecutive camera vectors can be made to match the camera vectors.
In this way, as n progresses continuously with the progress of the image, the scale adjustment is performed so that the error of each camera vector obtained by calculating a plurality of times for the same feature point is minimized, and integration is performed. A camera vector can be determined. Accordingly, it is possible to speed up the arithmetic processing by combining simple arithmetic operations while obtaining a highly accurate camera vector having no error.

Here, there are various simple calculation methods depending on the accuracy. For example, when (1) many feature points of 100 or more are used in high-precision calculation, the minimum number of simple calculation is about ten. (2) Even if the number of the same feature points is the same as the feature points and camera positions, innumerable triangles are established there, and equations for that number are established. By reducing the number of equations, it can be simplified.
In this way, integration is performed by adjusting the scale so that the error of each feature point and camera position is minimized, distance calculation is performed, and feature points with large error distribution are deleted, and other features are added as necessary. By recalculating the points, the calculation accuracy at each feature point and camera position can be improved.

  In addition, by performing high-speed simple calculation in this way, camera vector real-time processing becomes possible. In real-time processing of camera vectors, calculation is performed with the minimum number of frames that can achieve the target accuracy and the minimum number of feature points that are automatically extracted, the approximate value of the camera vector is obtained and displayed in real time, and then the image is accumulated. Accordingly, the number of frames can be increased, the number of feature points can be increased, camera vector calculation with higher accuracy can be performed, and approximate values can be replaced with camera vector values with higher accuracy for display.

Furthermore, in this embodiment, it is possible to track three-dimensional information (three-dimensional shape) in order to obtain a more accurate camera vector.
Specifically, first, in the three-dimensional information tracking unit 2-5, the camera vector obtained through the camera vector calculation unit 2-3 and the error minimization unit 2-4 is positioned as an approximate camera vector, and the subsequent process Based on the three-dimensional information (three-dimensional shape) obtained as part of the image generated in step 3, the partial three-dimensional information contained in multiple frame images is continuously tracked between adjacent frames to obtain a three-dimensional shape. Auto-tracking.
Then, from the tracking result of the three-dimensional information obtained by the three-dimensional information tracking unit 2-5, a higher-precision camera vector is obtained by the high-precision camera vector calculation unit 2-6.

The feature point extraction unit 2-1 and feature point correspondence processing unit 2-2 described above automatically track feature points in a plurality of inter-frame images. There may be restrictions. In addition, since the image is two-dimensional and the shape changes during tracking, there is a certain limit in tracking accuracy.
Therefore, the camera vector obtained by the feature point tracking is regarded as an approximate value, and the three-dimensional information (three-dimensional shape) obtained in the subsequent process is traced on each frame image, and a high-precision camera vector is obtained from the trajectory. Can do.
The tracking of 3D shapes is easy to obtain matching and correlation accuracy, and the 3D shape does not change in size and size depending on the frame image, so it can be tracked over many frames. The accuracy of the camera vector calculation can be improved. This is possible because the approximate camera vector is known by the camera vector computing unit 2-3 and the three-dimensional shape is already known.

  When the camera vector is an approximate value, the error of 3D coordinates over a very large number of frames is small because there are few frames related to each frame by feature point tracking. The error of the three-dimensional shape when a part of the image is cut is relatively small, and the influence on the change and size of the shape is considerably small. For this reason, the comparison and tracking in the three-dimensional shape is extremely advantageous over the two-dimensional shape tracking. In tracking, when tracking with 2D shape, tracking changes in shape and size in multiple frames are unavoidable, so there are problems such as large errors and missing corresponding points. However, in tracking with a three-dimensional shape, there is very little change in shape, and in principle there is no change in size, so accurate tracking is possible.

Here, as the three-dimensional shape data to be tracked, there are, for example, a three-dimensional distribution shape of feature points, a polygon surface obtained from the three-dimensional distribution shape of feature points, and the like.
It is also possible to convert the obtained three-dimensional shape from a camera position into a two-dimensional image and track it as a two-dimensional image. Since the approximate value of the camera vector is known, projection conversion can be performed on a two-dimensional image from the camera viewpoint, and it is also possible to follow a change in the shape of the object due to movement of the camera viewpoint.

The camera vector obtained as described above can be displayed superimposed on the video image shot by the all-round video camera 1-1.
For example, as shown in FIG. 25, the image from the in-vehicle camera is developed in a plane, the corresponding points on the target plane in each frame image are automatically searched, and the corresponding points are combined to match the target plane. A combined image is generated and displayed in the same coordinate system.
Furthermore, the camera position and the camera direction can be detected one after another in the common coordinate system, and the position, direction, and locus can be plotted. The CV data indicates the three-dimensional position and the three-axis rotation, and the CV value can be observed simultaneously in each frame of the video image by displaying it over the video image. FIG. 25 shows an example of an image in which CV data is displayed superimposed on a video image.
If the camera position is correctly displayed in the video image, the position in the video image indicated by the CV value is the center of the image, and if the camera movement is close to a straight line, the CV values of all frames are displayed overlapping. Therefore, for example, as shown in FIG. 25, it is appropriate to display the position of 1 meter right below the camera position. Alternatively, it is more appropriate to display the CV value at the height of the road surface based on the distance to the road surface.

[Automatic driving device]
Next, an embodiment of the automatic driving device of the present invention using the above-described three-dimensional machine map and CV calculation will be described with reference to FIG.
As shown in the figure, the automatic driving device 130 according to the present embodiment is similar to the navigation device 120 described above, from a three-dimensional mechanical map 121 (see FIGS. 3 and 4) and a camera loaded on a moving vehicle. The real-time video 122 (see FIG. 8) is compared and calculated.
Specifically, the automatic driving device 130 includes an in-vehicle camera device 131, an altime video generation device 132, a comparison device 133, a three-dimensional machine map 121, a reproduction search device 134, a coincidence output device 135, and a CV. A value acquisition device 136, a travel information acquisition device 137, an object information acquisition device 138, and an automatic travel device 139 are provided.

The in-vehicle camera 131 is fixed to the vehicle and acquires an image around the vehicle. For example, as shown in FIG. 14, the in-vehicle camera 131 installs the all-round video camera 1-1 at the center of the roof of the moving body (vehicle) 1-1a, and captures an image around the vehicle as the vehicle moves. To do. Further, as shown in FIG. 31 to be described later, the in-vehicle camera 131 can install a plurality of cameras on the roof of the vehicle with a predetermined layout.
The real-time video generation device 132 acquires a real-time video output from the in-vehicle camera 131.
The comparison device 133 holds and stores the three-dimensional mechanical map 121 mounted on the vehicle and the real-time video captured by the vehicle-mounted camera 131 by an information processing device mounted on the vehicle such as a PC (personal computer). Compare in PC memory.

The reproduction search device 134 is a region in which the vehicle is expected to travel in a plurality of regions in the real-time video, and a part of the geometric component is a part of the geometric component of the three-dimensional machine map. While obtaining the corresponding position, the viewpoint position, viewpoint orientation, and viewpoint direction of the 3D machine map are adjusted so as to approach each other, and the limited range of the 3D machine map is searched.
The coincidence output device 135 outputs the position coordinates of the three-dimensional machine map when the plurality of parts of the real-time video coincide with the plurality of parts of the three-dimensional machine map, and the rotation coordinates when the parts coincide with each other.
The CV value acquisition device 136 acquires the CV value of the real-time video by obtaining the position coordinates and the rotation coordinates of each frame of the real-time video when the vehicle is traveling.
The CV value is acquired by the above-described CV calculation (see FIGS. 13 to 25).

The travel information acquisition device 137 calculates the vehicle position and posture, speed, acceleration, direction, angular velocity, angular acceleration, and the like and acquires them in real time.
The object information acquisition device 138 obtains the position and posture, speed, acceleration, direction, angular velocity, angular velocity, etc. of vehicles and obstacles other than the own vehicle in the real-time video from the CV value obtained by the CV value acquisition device 136. Calculate and obtain in real time.
Then, the automatic travel device 139 determines the travel lane, travel speed, and travel direction of the vehicle based on the control signal generated based on the data obtained by each of the devices described above.
The vehicle control (automatic driving) by the automatic traveling device 139 can be realized by a known technique.
As described above, the automatic driving device 130 based on the three-dimensional machine map and the real-time video is configured.

Next, in the above-described automatic driving apparatus according to the present invention, a case where a guide marker is used will be described with reference to FIGS.
In the above-described automatic driving apparatus equipped with a three-dimensional mechanical map, the CV value of the camera that acquired the real-time video is obtained by the correspondence between the simple geometric figures constituting the three-dimensional mechanical map and the real-time video, and the vehicle position is obtained. As the vehicle will be controlled.
For this reason, it is good if the simple geometric figures that make up the 3D machine map can always correspond to the real-time video, but the video may not be predictable, and the weather and time zone, especially at night, during snowfall, etc. It is also assumed that a video corresponding to an appropriate simple geometric figure cannot be seen (a real-time video cannot be handled).
Therefore, by using the following guide marker, it is possible to always take correspondence between the three-dimensional machine map and the real-time video from the traveling vehicle.

FIG. 27 is a block diagram showing a configuration of a guide marker used in the three-dimensional machine map generation device 110 according to the embodiment of the present invention shown in FIG.
As shown in the figure, a guide marker 140 according to an embodiment of the present invention includes a designated guide marker 141 and a unified shape guide marker 142. The guide marker 140 generates a guide marker figure 143 and a guide marker image 144. Then, the data is input to the three-dimensional machine map generation device 110.
Specifically, the guidance marker 140 is used in the machine map generation device 110 to serve as a coordinate reference when the vehicle travels by automatic driving.

The designated guide marker graphic 141 is a graphic corresponding to the three-dimensional space of the three-dimensional machine map as a guide marker graphic or a guide marker image selected from the geometric simple figures constituting the three-dimensional machine map. It is.
The unified shape guide marker graphic 142 corresponds to a three-dimensional machine map by setting a guide marker having a known shape in advance at an appropriate density, for example, on a road or in the vicinity of the road. The guide marker figure reflected is made to correspond to the three-dimensional space of the three-dimensional machine map.
The guide marker image 144 is obtained by associating a partial image of the video that is the basis of the guide marker graphic 143 as a guide marker image with the three-dimensional space of the three-dimensional machine map.

A guide marker graphic 143 and a guide marker image 144 corresponding to the three-dimensional space of the three-dimensional machine map are generated by the above-described designated guide marker graphic 141 and the unified shape guide marker graphic 142, and are generated via the three-dimensional machine map generation device 110. A three-dimensional machine map 121a with a guide marker is generated.
Then, in the designated guide marker graphic 141 and the unified shape guide marker graphic 142, an attribute of a graphic can be registered as a guide marker graphic.
Also, an attribute of an image can be registered as a guide marker image.
Even if the combination of the guide marker graphic 143 and the guide marker image 144 does not correspond to the simple geometric figure constituting the three-dimensional machine map and the real-time video, the real-time from the traveling vehicle and the three-dimensional machine map is real-time. It will be possible to always take correspondence of the video.

Hereinafter, a preferable aspect of the guidance marker 140 (guidance marker figure 143 and guidance marker image 144) will be specifically described.
First, attention is paid to a figure that is particularly characteristic of the simple geometric figure constituting the three-dimensional machine map, and it is preferentially associated as a guide marker figure (designated guide marker figure 141).
In an area where the simple geometric figure is not characterized, a guidance marker is positively installed on an actual road or in the vicinity thereof (unified shape guidance marker figure 142). In this case, by using infrared rays or polarized light, it is possible to use a CCD or infrared filter having sensitivity to infrared rays on the camera side so that it is difficult for humans to visually recognize.

Further, since the shape, figure, and position of the marker can be predicted since it is registered in the three-dimensional machine map, the guide marker is used only for confirmation work, so that the calculation becomes easy.
In this case, it is possible to select a shape and a figure that are easily recognized by an information processing apparatus such as a PC that performs arithmetic processing and that are not easily affected by sunlight or the climate. For example, a figure combining ◯ and + can be used as a figure resistant to scale and rotation.
Further, when detecting and recognizing these figures on the PC side, not the so-called advanced recognition work but the confirmation work of the object whose shape and attribute are already known.

Furthermore, since the guidance marker may be a confirmation work and has a small shape and a small image, the marker can be simply extracted from the CV image used when generating the three-dimensional machine map without replacing it with a geometric figure. The image of the portion may be cut out as it is and used as a guide marker image as it is, and may be pasted on a three-dimensional machine map.
In addition, in order to measure the vehicle position from only the marker, a distant marker is also required. Therefore, the distant guide marker is used as a reflector, and the position is determined by directing laser light directly on the guide marker instead of the camera. It is also possible to measure and confirm.

It is desirable that the 3D mechanical map with the guide marker is composed of a geometric simple figure or a combination thereof that is easily recognized or detected by the PC in comparison with the 3D map generation after being acquired as a real-time image by the vehicle-mounted camera. .
Also, in the 3D machine map generation, the unified shape guidance marker is actually installed in the field, so that the simple geometry can be captured correctly and reliably in the subsequent map generation process. A geometric figure is suitable.
The three-dimensional machine map is composed of these guide marker graphics.
It should be noted that the installation density or the specified density of the guide markers needs to be a density at which a plurality of guide markers can be simultaneously detected in the field of view of the real-time video acquired by the in-vehicle camera. For example, it is necessary that at least three guide markers are simultaneously shown in the real-time video.

Next, an embodiment of an automatic driving apparatus that associates a simple geometric figure constituting a three-dimensional machine map with a real-time image using the guide marker 140 as described above will be described with reference to FIG. .
FIG. 28 is a functional block diagram illustrating a configuration of an embodiment of an automatic driving device using a guide marker.
As shown in the figure, an automatic driving device 150 according to the present embodiment is an automatic driving device including a three-dimensional mechanical map registered as the above-described guide marker graphic, and includes a guide marker detection device 151, a guide marker, and the like. The map corresponding | compatible apparatus 152, the vehicle CV value acquisition apparatus 153, the marker coordinate reader 154, the iterative calculation apparatus 155, and the output device 156 are provided.

The guidance marker detection device 151 detects a plurality of guidance marker images captured in a real-time image of the road and the vicinity of the road by the in-vehicle camera.
The guidance marker map corresponding | compatible apparatus 152 matches the detected guidance marker image | video with the guidance marker figure on a three-dimensional machine map.
The vehicle CV value acquisition device 153 obtains the coordinates and orientation of the vehicle from the coordinates of the corresponding guide marker graphics by the guidance marker map correspondence device.
The marker coordinate reading device 154 reads the coordinates of the new guide marker figure on the three-dimensional machine map for the coordinates of the guide marker that appears one after another as the vehicle progresses.

The iterative calculation device 155 searches the real-time video for the vicinity of the coordinates using the coordinates read by the marker coordinate reading device 154 as the predicted position, and uses the guide marker map corresponding device 152 to generate a new guide marker image and a new guide marker graphic. Make it correspond. And the operation | work which calculates | requires the coordinate and attitude | position of the vehicle which advanced ahead from the previous position by calculation with the vehicle CV value acquisition device from the coordinates which the plurality of corresponding guide marker figures have as attributes is continuously continued.
The output device 156 continuously detects the vehicle position constantly by continuously outputting the vehicle position and posture by the above repetitive work.
As described above, the automatic driving apparatus 150 according to the present embodiment can be made to advance while always acquiring the coordinates and posture of the vehicle.

In the automatic driving apparatus according to the present invention described above, similar to the navigation apparatus described above, based on the recording of the real-time video loaded on the vehicle and / or the recording of the CV value obtained from the automatic driving apparatus. At a special point in time, such as an accident, it is possible to perform a mechanical analysis of each of the related objects including the vehicle and their relationship.
The real-time video can be analyzed and analyzed with the same algorithm as the three-dimensional machine map, simple geometric figures can be extracted, and the three-dimensional machine map can be updated.

Next, a more specific embodiment of the above-described automatic driving apparatus according to the present invention will be described with reference to FIGS.
FIG. 29 is a block diagram showing a schematic configuration of a preferred embodiment of an automatic driving apparatus having a three-dimensional machine map according to the present invention.
The automatic driving device 160 shown in the figure stores a field recognition device 161 for recognizing a place where a vehicle that performs automatic driving is placed, and a three-dimensional machine map for a machine (computer) to judge, not a human. 3D machine map database 162 to be stored, obstacle recognition device 163 around the vehicle, self-position evaluation device 164 for evaluating the self-position and posture, and signals necessary for controlling the vehicle from the information of each device And an automatic travel device 165 for controlling the vehicle.

The field recognition device 161 acquires information on the environment itself, which is a “field” for automatic driving. Obtain information such as the three-dimensional space, climate, time, and location of the environment where the vehicle is located. The field recognition device 161 sets the conditions for automatic driving only after the vehicle itself is positioned in the field information.
The three-dimensional machine map database 162 stores a three-dimensional machine map of all spaces used by the vehicle.
Here, the space where the vehicle exists, the space to go from now on and its attributes are registered.
In addition, the 3D machine map is linked to other devices and has a wide range of uses. There are devices that do not necessarily require a three-dimensional machine map (for example, a three-dimensional machine map database 162, an automatic travel device 165, etc.) for use in connection with other devices. It can be demonstrated.

The obstacle recognition device 163 exhibits a function of recognizing obstacles around the vehicle. Specifically, relations with oncoming vehicles, translational vehicles, pedestrians crossing roads or pedestrian crossings, road falling objects, obstacles to vehicle driving, or pedestrians that may pose a danger to pedestrians, etc. Recognize
The self-position assessment device 164 performs the most basic work for automatic driving, which accurately determines its position and posture in the three-dimensional machine map.
Specifically, the self-position evaluation by the self-position evaluation device 164 compares the real-time video from the camera installed on the vehicle with the three-dimensional machine map, and determines the position and posture of the own vehicle relative to the three-dimensional machine map. It will be demanded carefully. The number of physical parameters representing the position and orientation is 6, which is equivalent to determining the camera position and orientation having 6 degrees of freedom. Furthermore, it is possible to acquire a CV value as a relative value only from a real-time video, and perform highly accurate self-position evaluation from both.
The automatic traveling device 165 integrates information from the field recognition device 161, the three-dimensional machine map database 162, the obstacle recognition device 163, and the self-position evaluation device 164, and generates a control signal necessary for automatic traveling.

Next, a more specific embodiment of the automatic driving apparatus provided with the above three-dimensional machine map will be described.
First, the basic principle of the automatic driving apparatus according to an embodiment of the present invention will be described with reference to FIG.
As shown in the figure, the automatic driving device 170 according to the present embodiment includes a three-dimensional mechanical map device 171, a spatial analysis device 172, an approximate position rating device 173, a relative self-position / posture rating device 174, Absolute self-position / posture evaluation device 175, moving body separation / posture evaluation device 176, mobile body position / posture evaluation device 177, object recognition device 178, mobile body recognition device 179, determination device 180, control device 181 and display The apparatus includes a device 182, a CV video update device 183, and a CV video database 184.

The three-dimensional machine map device 171 arbitrarily reads a three-dimensional map for a machine (computer) to judge, not a human.
The space analysis device 172 performs a three-dimensional analysis on the environment in which the vehicle performing automatic driving is located, by using a correspondence between the three-dimensional mechanical map and a real-time image acquired by a camera mounted on the vehicle.
The approximate position rating device 173 evaluates and specifies the approximate position of the vehicle using a known technique such as GPS.
The relative self-position / posture evaluation device 174 calculates a relative coordinate and posture by CV-calculating the real-time video.

The absolute self-position / posture evaluation device 175 narrows down the vehicle position from the approximate position obtained by the approximate position-assessment device 173 and the relative coordinates and posture obtained by the relative self-position / posture evaluation device 174, and the three-dimensional machine within that range is narrowed down. A matching point is searched for by matching with a real-time image in the map, and the current position coordinates and posture with high accuracy are obtained as absolute values.
The moving body separation device 176 tracks many feature points related to the real-time video, extracts a tracking component of the moving body from among them, and separates the moving body video component from the stationary coordinate system.
The moving object position / posture evaluation device 177 extracts the position and posture of the moving object from the moving object tracking component related to the moving object image component obtained by the moving object separation device 176.

The object recognition device 178 recognizes an object by comparing other vehicles related to vehicle travel, obstacles, and the like with registered attributes.
When the mobile object recognition device 179 needs to know the attribute of the mobile object, the mobile object recognition device 179 determines the attribute type of the mobile object in comparison with the registered attribute.
The determination device 180 takes in information from all or a part of all of the above devices, or a part thereof, determines safety, selects an appropriate course, and generates a signal necessary for vehicle control.
The control device 181 controls the vehicle safely and appropriately by signals necessary for vehicle control.
The display device 182 displays information from all the above devices, or some signals, control contents, and the like.

The CV video update device 183 fetches from the spatial analysis device 172 a CV video obtained by performing a CV calculation while acquiring a new video in a region / site where a three-dimensional mechanical map does not exist or does not match the actual map.
The CV video database 184 is a storage unit that stores and stores the CV video captured from the CV video update device 183, and the contents of the database are updated as needed by the CV video updated or added by the CV video update device 183. .
Then, the contents of the three-dimensional machine map device 171 are updated by the CV video database 184.

Specifically, the automatic driving apparatus 170 of the present embodiment configured as described above operates as follows.
The space analysis device 172 acquires information on the environment itself, which is a “place” for automatic driving.
Specifically, information such as the three-dimensional space, climate, time, and location of the environment in which the vehicle exists is acquired.
This spatial analysis device 172 establishes the conditions for automatic driving only by positioning the vehicle itself in the field information.

In addition, the space analysis device 172 acquires a real-time video image, acquires a CV value from the real-time video image, and obtains the outline of the three-dimensional space and the relationship between the vehicle position and the posture based on the acquired CV value.
The calculation of the CV value can be performed in the same manner as in the CV value acquisition device 136 (see FIG. 26) described above (see FIGS. 13 to 25).
Here, in order to acquire the real-time video, for example, it can be acquired by the in-vehicle camera as shown in FIG. 14 described above. However, in this embodiment, as shown in FIG. Real-time video is acquired by setting with a predetermined layout.

Specifically, as shown in FIG. 31, two pairs of cameras 1 and 2 are used to secure a horizontal viewing angle of 135 degrees and a combined viewing angle of 270 degrees or more. Secure. If the combined viewing angle of 270 degrees cannot be secured with two cameras, the desired combined viewing angle may be secured by using three cameras.
Attach such a pair of cameras at four locations of the vehicle (two locations at the front of the vehicle and two locations at the rear) to secure a 360-degree view of the entire circumference in the horizontal direction. Parallax is also used. The overlapping visual field will be described later.
Thereby, in this embodiment, a total field of view in the horizontal direction is ensured by a total of eight cameras at four locations. All-round images are generated by combining and synthesizing images obtained from these eight cameras. This is treated as real-time video below.
In addition, as shown in FIG. 31, you may make it attach the perimeter camera which can image | photograph all horizontal directions in the position of the roof center part (A of FIG. 31) of a vehicle.

Also, the CV value can be obtained by performing CV calculation on the real-time video. Details of the CV calculation for obtaining the CV value are as described above (see FIGS. 13 to 25).
On the other hand, the three-dimensional machine map is recorded in the three-dimensional machine map device, and the approximate position evaluation device 173 can designate the approximate position and take out the nearby three-dimensional machine map.
In order to generate a three-dimensional machine map, a CV image is generated after processing such as CV calculation of the entire circumference image, vertical correction, and addition of absolute coordinates.
Then, a new feature point is three-dimensionally measured from the CV image and decomposed into a simple geometric figure to generate a three-dimensional machine map.
The method for calculating the camera position of the real-time video by comparing the three-dimensional machine map and the real-time video is as described above (see FIG. 26).

Here, if the vehicle does not pass the area where the 3D machine map does not exist or the site does not match the reality, for example, the site is under construction, the autonomous driving vehicle will acquire a new video. While performing the CV calculation, the CV video from the space analysis device 172 is taken into the CV video update device 183, and the content (local content) of the CV video database 184 is updated or newly added. The contents of the three-dimensional machine map apparatus 171 are updated by the updated or added CV video database 184.
As described above, in the present embodiment, when traveling in a region where the map and the reality do not match and when traveling in a region that does not exist on the map, the CV video that is the basis of the three-dimensional machine map is added or updated. You can go forward.

Since real-time processing is required for vehicle travel, it is not necessary to convert the CV video into a three-dimensional machine map, and the CV video can be used as it is as a three-dimensional map.
Even if it is a CV image itself, the purpose of the three-dimensional machine map can be achieved. Therefore, the CV image can be used as it is instead of the three-dimensional machine map. The three-dimensional machine map stored in the three-dimensional machine map device 171 has a main purpose of reducing the data amount as much as possible. For example, when the data amount does not cause a problem, the CV video is converted into a tertiary image. It is possible to replace the original machine map.

However, since the amount of data increases when the update area increases if the CV video is used as it is, it is preferable to appropriately use the CV video and the three-dimensional machine map.
In this way, in this embodiment, it is possible to travel where there is no map while making a map.
The CV video is recorded and converted into a three-dimensional machine map by post-processing. Update information can also be transmitted to other companies via communication.
In practice, it is common to drive a map-dedicated vehicle in an area without a map, and to update a map data, to drive an update-only vehicle.

Next, the approximate position evaluation device 173 obtains the approximate position of the vehicle where the GPS or the like is installed, outputs the approximate position of the site, and sends it to the three-dimensional machine map device 509 so that the approximate position of the three-dimensional machine Maps are output one after another.
On the other hand, the relative self-position and posture are determined by the relative self-position posture evaluation device 174 by performing CV calculation on the real-time video from the space analysis device 172.
Here, “relative” means having no absolute coordinates related to the latitude and longitude altitude.
In the automatic driving device 170 of the present embodiment, since the visual fields of the eight cameras of the space analysis device 172 overlap (see FIG. 31), it is possible to acquire a video with a partial parallax, and the distance length You can have a unit scale of length, not relative.

Next, by comparing the real-time image with the three-dimensional mechanical map in the vicinity of the approximate position output from the three-dimensional mechanical map device 171, the absolute position of each camera mounted on the vehicle is calculated. It is done.
In the present embodiment, it is assumed that an end point, a direct simple geometric figure, a designated marker figure, and a unified shape marker figure are used for comparison between the three-dimensional machine map and the real-time image (see FIG. 27).
As a result, in the 3D machine map, in addition to the end points and straight lines, guide markers are registered and the center coordinates are indicated, so that the absolute 3D coordinates can be acquired.
That is, by automatically searching for a point and posture where the simple geometric figure matches the marker figure, each position of the camera can be determined on the three-dimensional machine map. In this way, the absolute self position and posture of the vehicle are obtained.

Here, since it takes a certain amount of time to calculate the absolute self-position / posture accurately by the absolute self-position / posture evaluation device 175, in consideration of the efficiency of the system, in this embodiment, the absolute self-position / posture evaluation is performed. The calculation frequency is set to 5 frames per second by thinning out frames of real-time video output at 30 frames per second.
Furthermore, the absolute self-position / posture is obtained accurately from intermittent frames of 5 frames per second from the real-time video, and the middle is a CV value (output of the relative self-position / posture assessment device 174) obtained by direct CV calculation of the real-time video of 30 frames per second. Find by interpolation. Alternatively, the signal for interpolation may be a vehicle speed pulse of the own vehicle or a combination of an accelerometer and a gyro.

The object recognition device 178 can sufficiently process the CV video in this state having a unit scale of length, but in this embodiment, the object recognition can also be performed by the above-described PRM method. In the case of the PRM method, in order to search for parts registered in the object database, the object recognition is performed after the processing by the absolute self-position / posture evaluation device 175 is performed using a three-dimensional machine map in the later process. Is desirable.
By obtaining the absolute self-position / posture by calculation, a CV image can be used, and this makes it easy to cut out an object in a three-dimensional space.
Although the object overlaps in a plurality of CV video frames, since the absolute value of the three-dimensional coordinates is obtained, the object can be cut out from a plurality of related frames and processed. , Recognition accuracy is improved.

In the moving body separating apparatus 176, a large number of feature points can be set in the CV video, and these feature points can be separated into those belonging to the stationary coordinate system and those belonging to the moving body coordinate system.
The moving object position / posture evaluation device 177 obtains a scale from the tracking data of the feature points belonging to the moving object coordinate system and the parallax of the overlapping visual field, and the position, velocity, acceleration, posture, direction, angular velocity of the moving object with respect to the stationary coordinate system. Angular acceleration and the like can be obtained by calculation.
This means that not only the speed of the vehicle to be translated or the oncoming vehicle, but also the attitude, the rotational angular velocity, and the like are all obtained by calculation in the same manner as the own vehicle.
Therefore, not only as information for controlling the speed and orientation of the host vehicle, but also by recording the information, it is extremely useful for running analysis and self-analysis.

In the mobile object recognition device 179, when the mobile object position / posture evaluation device 177 knows the CV value of the mobile object, the mobile object recognition is further performed by searching the database for candidates for the mobile object using the PRM method. Make it possible.
Up to this point, the CV value of the real-time video, the absolute value of the position and orientation of the host vehicle, the speed, the acceleration, the rotation, the traveling direction, the rotational angular velocity, the attribute parts near the coordinate values in the 3D machine map, the object recognition result, the opposite Information such as absolute values of position and orientation, speed, acceleration, rotation, traveling direction, rotational angular velocity, moving object recognition result, and the like of a vehicle, a translation vehicle, and the like is acquired.
These pieces of information are input to the determination device 180, and the driving conditions of the host vehicle satisfying the safe driving conditions and the appropriate driving conditions are determined by artificial intelligence, and converted into control signals for the host vehicle.

Furthermore, it is also possible to transmit the own vehicle travel condition to the oncoming vehicle and the translation vehicle and add it to the travel condition of the opponent vehicle. Similarly, a signal from another vehicle can be acquired by the communication means and reflected on the determination device 512 of the host vehicle.
In the control device 181, by controlling the speed and direction of the own vehicle, and also a plurality of suspensions and wheels independently by the control signal of the own vehicle acquired in this way, or by controlling the headlight direction, The actual vehicle travel can be controlled.
Further, automatic driving / manual driving of the vehicle can be arbitrarily switched, and the driver may be in the vehicle, so that the control signal can be visualized and displayed on the display device 182. ing.

The display device 182 realizes a surround view and a super-wide-angle back monitor by combining and displaying images from the eight cameras (see FIG. 31) described above.
Display examples on the display device 182 are shown in FIGS.
FIG. 32 shows a display example of surround view, FIG. 33 shows a display example of an ultra-wide-angle back monitor, and FIG. 34 shows a display example in which a CG guide is added to a real-time CV video.

Although the automatic driving apparatus 170 according to the present embodiment has been described above, the automatic driving apparatus 170 according to the present embodiment is not limited to the configuration and operation described above, and various additions and changes can be made.
For example, the guidance CG can be displayed in the real-time video by setting the destination and passing place on the three-dimensional machine map. Further, when the guidance CG on the display screen is clicked with a fingertip or the like, the contents of the guidance are displayed, and communication with the guidance destination becomes possible.
In addition, a 3D machine map or CV video is displayed on the display device, a scheduled basic travel route is automatically or manually described on the 3D machine map, the basic travel route is automatically traveled, and obstacles and traffic conditions Depending on the route, you can change course, change route, change speed, avoid obstacles and reach your destination.
Furthermore, it is possible to receive an image from a camera on which a road fixed point is installed via a communication means, and to coordinate and synthesize it with a three-dimensional machine map or a V image, and there is no contradiction via the display device 182. It is also possible to display the relationship between position and posture.

As described above, the three-dimensional machine map and the three-dimensional machine map generation device, and the navigation device and the automatic driving device using the three-dimensional machine map of the present invention have been described with reference to the preferred embodiments. Needless to say, the present invention is not limited to the embodiments, and various modifications can be made within the scope of the present invention.
For example, since the navigation device and the automatic driving device of the present invention have a three-dimensional map, the moving body to be applied is not limited to a vehicle traveling on the ground, and may be one that navigates a three-dimensional space. Since it can also be used on airplanes, it is possible to perform highly accurate navigation and automatic driving when landing. The robot is the ultimate automatic operation and can be used as it is. In addition, navigation and automatic driving on a cosmic scale are possible by using stars and constellations that can be seen from a spacecraft.

  The present invention provides a navigation device suitable for, for example, a car navigation device mounted on an automobile, a navigation device mounted on an airplane, an unmanned automatic driving heavy machine in a dangerous place, a navigation device for automatic driving, a robot navigation device, and the like. Can be used as

It is a block diagram which shows the structure of the three-dimensional map production | generation apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the navigation apparatus which concerns on one Embodiment of this invention. It is the figure which showed the three-dimensional mechanical map production | generation which concerns on this invention in an overview. It is the figure which showed the three-dimensional mechanical map production | generation which concerns on this invention with the eyes of the vehicle at the time of image acquisition. It is the figure which projected a part of the perimeter image on the plane and expressed in perspective, and was taken from the vehicle on the right diagonal line of the two lanes on one side. It is a figure which shows the image which projected the perimeter video on the plane. It is a figure which shows the result of having acquired the three-dimensional coordinate about the perimeter image shown in FIG. It is a figure which shows the real-time image | video image | photographed from the camera mounted in the moving vehicle. It is a figure which shows the matching process of a real-time image | video and a machine map. It is the figure which showed the present position of the vehicle based on the position and attitude | position of the vehicle which the machine judged automatically on the three-dimensional machine map on the three-dimensional machine map. It is the figure which represented the perimeter video acquired with the all-around camera by the Mercator method. It is a figure which shows an example which synthesize | combined CG for guidance, such as an arrow, with the real-time image | video. It is a block diagram which shows the basic composition of one Embodiment of the CV calculating part which functions as a CV image acquisition part of the three-dimensional machine map generation apparatus of this invention. It is the schematic which shows the means to image | photograph the perimeter video image | video used by the CV calculating part shown in FIG. 13, and is a perspective view of the vehicle which mounts the perimeter camera on the roof part. It is the schematic which shows the means which image | photographs the perimeter video image | video used by the CV calculating part shown in FIG. 13, (a) is a front view of the vehicle which mounts a perimeter camera in a roof part, (b) is a top view similarly. It is. It is explanatory drawing which shows the conversion image obtained from the image | video image | photographed with a omnidirectional camera, (a) is a virtual spherical surface to which a spherical image is affixed, (b) is an example of a spherical image affixed to a virtual spherical surface , (C) shows an image obtained by developing the spherical image shown in (b) on a plane according to the Mercator projection. It is explanatory drawing which shows the detection method of the specific camera vector in the CV calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the detection method of the specific camera vector in the CV calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the detection method of the specific camera vector in the CV calculating part which concerns on one Embodiment of this invention. It is explanatory drawing which shows the designation | designated aspect of the desirable feature point in the detection method of the camera vector by the CV calculating part which concerns on one Embodiment of this invention. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the CV calculating part which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate of the feature point obtained by the CV calculating part which concerns on one Embodiment of this invention, and a camera vector. It is a graph which shows the example of the three-dimensional coordinate and camera vector of the feature point obtained by the CV data calculating part which concerns on one Embodiment of this invention. In the CV calculating part which concerns on one Embodiment of this invention, it is explanatory drawing which shows the case where a some feature point is set according to the distance of a feature point from a camera, and a some calculation is repeated. It is a figure at the time of displaying the locus | trajectory of the camera vector calculated | required by the CV data calculating part which concerns on one Embodiment of this invention in a video image | video. It is a block diagram which shows the structure of the automatic driving apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the guidance marker used with the automatic driving | operation apparatus of this invention. It is a block diagram which shows the structure of the automatic driving | operation apparatus using the guidance marker of this invention. It is a block diagram which shows the structure of the automatic driving apparatus which concerns on other embodiment of this invention. It is a block diagram which shows the structure of the automatic driving apparatus which concerns on other embodiment of this invention. It is the schematic diagram seen from the vehicle upper surface which shows an example of the camera structure for acquiring the real-time image | video in the automatic driving device of this invention. It is a figure which shows the example of a display at the time of combining and displaying the image | video acquired with the camera shown in FIG. 31, and is a display example of a surround view. It is a figure which shows the example of a display at the time of combining and displaying the image | video acquired with the camera shown in FIG. 31, and is a display example of a super-wide-angle back monitor. It is a figure which shows the example of a display at the time of combining and displaying the image | video acquired with the camera shown in FIG. 31, and is a display example which added the CG guide to the real-time CV image | video.

Explanation of symbols

2 CV operation unit 110 3D machine map generation device 111 All-round video imaging device 112 CV operation device 113 Geometric simple figure detection device 114 3D coordinate addition device 115 Attribute registration device 116 3D machine map recording device 120 Navigation device 121 Three-dimensional machine map 122 Real-time video 123 Comparison calculation device 124 Position / direction rating output device 125 Vehicle control device 126 Guide display

Claims (12)

An in-vehicle all-around video capturing device that acquires all-around video from a running vehicle;
A CV arithmetic unit that acquires all-round CV video;
A geometric simple figure detecting device for extracting, selecting and separating a geometric simple figure from the all-round CV image and obtaining its position coordinate and orientation in three dimensions;
A three-dimensional coordinate adding device for giving three-dimensional coordinates to the detected geometric simple figure;
An attribute registration device for giving an attribute to the detected geometric simple figure;
A three-dimensional mechanical map recording device for reconstructing a three-dimensional space from the detected geometric simple figure and generating and recording a predetermined three-dimensional mechanical map by adding other information according to the purpose; ,
Have
The predetermined three-dimensional machine map is
A part or most of the target three-dimensional space is replaced with a plurality of simple geometric figures, and simplified and reconstructed simplified three-dimensional spatial data is stored. Are read together with related data such as coordinates, postures, and / or attributes corresponding to the three-dimensional space, and
Along with the geometric simple figure extracted from the real space, the object that matches the purpose of use is used as the geometric simple figure, and has the reconstructed simplified three-dimensional data, and the intended real three-dimensional space is used. It can be read or written by a computer by being reconstructed in conformity with the map data output together with related data such as all coordinates, postures, and / or attributes of the figure,
This three-dimensional machine map
For the purpose of automatic driving of the vehicle, objects such as traffic signs, road indications (features), road guide boards, and the like originally installed for the purpose of reading by the driver are classified as geometric simple figures and their attribute information In addition, it is arranged in addition to the coordinates of the three-dimensional space where the object exists, and in addition, automatic driving such as driving center line, road width, number of lanes, road gradient, turning radius, etc. that are not directly numerically displayed in reality A three-dimensional machine map generation device characterized in that numerical attribute information necessary for assisting is arranged in correspondence with a three-dimensional space for vehicle travel. A navigation device comprising a three-dimensional machine map generated by the three-dimensional machine map generation device according to claim 1,
Analyzed from the real-time video and / or using the CV value of the vehicle acquired by the navigation device, the position and posture of a stationary object in the video, and the position, posture, rotation and speed of a vehicle other than the moving own vehicle A navigation device characterized by obtaining a mechanical numerical value such as acceleration and performing a risk avoidance action. A navigation device comprising a three-dimensional machine map generated by the three-dimensional machine map generation device according to claim 1,
A navigation apparatus characterized in that a mechanical analysis of a related object including an own vehicle at the time of an accident is performed from recording of a real-time video and / or recording of a CV value obtained from the navigation apparatus. A navigation device comprising a three-dimensional machine map generated by the three-dimensional machine map generation device according to claim 1,
A navigation apparatus characterized by synthesizing a guidance CG such as an arrow or the like in an actual space and / or real-time video, or performing navigation by voice. A navigation device comprising a three-dimensional machine map generated by the three-dimensional machine map generation device according to claim 1,
A navigation device, wherein real-time video is analyzed and analyzed by an algorithm similar to that of the three-dimensional machine map, simple geometric figures are extracted, and the three-dimensional machine map is updated. A navigation device comprising a three-dimensional machine map generated by the three-dimensional machine map generation device according to claim 1,
A navigation apparatus, wherein information is exchanged with another vehicle by distributing the acquired information of the own vehicle to the other vehicle. A navigation device comprising a three-dimensional machine map generated by the three-dimensional machine map generation device according to claim 1,
A navigation apparatus characterized in that an image from a camera installed at a fixed point on a road is coordinate-integrated and synthesized on the three-dimensional machine map, and is displayed with a consistent position and posture relationship. An automatic driving device comprising a three-dimensional mechanical map generated by the three-dimensional mechanical map generating device according to claim 1,
An in-vehicle camera device that is fixedly installed in a vehicle and acquires an image around the vehicle;
A real-time video generation device that acquires real-time video output from the in-vehicle camera;
A comparison device for comparing the real-time video with a three-dimensional mechanical map mounted on the vehicle;
In a range of an area where the vehicle is expected to travel, a part of a geometric component in a plurality of regions in the real-time video determines a corresponding position with a part of the geometric component of the three-dimensional machine map. However, a playback search device that adjusts the viewpoint position, viewpoint orientation, and viewpoint direction of the three-dimensional machine map to approach each other, and searches within a limited range of the three-dimensional machine map;
A coincidence output device that outputs position coordinates of a three-dimensional machine map when a plurality of parts of the real-time image coincide with a plurality of parts of the three-dimensional machine map, and a rotation coordinate when the parts coincide with each other;
A CV value acquisition device that acquires the CV value of the real-time video by obtaining the position coordinate and the rotation coordinate for each frame of the real-time video during travel of the vehicle;
A travel information acquisition device that calculates the vehicle position and posture, speed, acceleration, direction, angular velocity, angular acceleration, etc., and acquires in real time;
From the CV value, the object information acquisition device for calculating in real time by calculating the position and posture of the vehicle and obstacles other than the own vehicle and obstacles in the real-time video, speed, acceleration, direction, angular velocity, angular velocity, etc.
An automatic driving device comprising: an automatic travel device that travels by determining a travel lane, a travel speed, and a travel direction of the vehicle. In order to use as a coordinate reference when the vehicle is driven by automatic driving, a selected figure or image is selected as a guide marker figure or a guide marker image from a geometric simple figure constituting the three-dimensional machine map. A designated guide marker figure corresponding to the three-dimensional space of
A guide marker having a known shape is installed in advance at an appropriate density on the road or in the vicinity of the road, and the guide marker figure reflected in correspondence with the three-dimensional machine map by the three-dimensional machine map generator is three-dimensional. Unified shape guidance marker figure corresponding to the three-dimensional space of the machine map,
A partial image of the video that is the basis of the guide marker graphic, as it is as a guide marker image, and a guide marker image corresponding to the three-dimensional space of the three-dimensional machine map,
9. The automatic driving apparatus according to claim 8, wherein among the designated guide marker graphic and the unified shape guide marker, an attribute is registered as a guide marker graphic, and an attribute of an image is registered as a guide marker image and combined. A guidance marker detection device for detecting a guidance marker image reflected in a real-time image of the road and the vicinity of the road by the in-vehicle camera;
A guide marker map corresponding device for associating the detected guide marker image with a guide marker graphic on a three-dimensional machine map;
A vehicle CV value acquisition device for calculating the coordinates and posture of the vehicle from the coordinates of the corresponding guide marker graphics by the guidance marker map correspondence device;
A marker coordinate reading device that reads the coordinates of a new guide marker figure on the three-dimensional machine map, the coordinates of the guide marker newly appearing one after another according to the progress of the vehicle,
By searching for the vicinity of the coordinates in the real-time video with the coordinates as the predicted position, the guide marker map corresponding device associates a new guide marker video with a new guide marker graphic, and a plurality of corresponding guide marker graphics are obtained. A repetitive arithmetic device that continuously continues the operation of calculating the coordinates and posture of the vehicle that has traveled ahead of the previous position by the vehicle CV value acquisition device from the coordinates possessed as attributes;
An output device that continuously outputs the vehicle position and posture due to repetitive work and continuously detects the vehicle position, and
The automatic driving apparatus according to claim 9, wherein the automatic driving apparatus is advanced while acquiring the coordinates and posture of the vehicle.   Based on the record of real-time video loaded on the vehicle and / or the record of the CV value obtained from the automatic driving device, each of the related objects including the vehicle and their The automatic driving device according to any one of claims 8 to 10, wherein a mechanical analysis of the relationship is performed.   The automatic driving apparatus according to any one of claims 8 to 11, wherein a real-time image is analyzed and analyzed by an algorithm similar to that of the three-dimensional machine map, a simple geometric figure is extracted, and the three-dimensional machine map is updated. .