JP4210309B2 - Image data display system with map information and image data display program with map information - Google Patents

Description

  The present invention captures and collects image data for grasping a geographical environment, for example, by shooting on a road or flying over the sky, and extracting and processing according to a required geographical range, image resolution, and image data amount, The present invention relates to an image data display system associated with geographic information that is managed spatially and temporally in association with geographic coordinates and displayed in conjunction with a map.

 With the advancement of information technology and the development of large-capacity communication networks, large-capacity databases such as photographs and moving images can now be handled easily on display media such as the Internet, car navigation, and mobile phones.

 Along with this, there is a growing need to accurately convey the actual landscape to the user without deformation by using an image as a geographic attribute in a geographic information system (GIS) or a map search site on the Internet.

  A technique has been developed for taking a cityscape image taken from a specific road and using it for maintenance of roads and rivers (see, for example, Patent Document 1).

  In this technique, an image obtained from a feature on a road contour line is managed on a one-dimensional feature such as a road center line in association with a distance from a start point designated on the feature.

In addition, there is a technique for managing an artificially created CG image in association with a road having a graph structure in part, not an actually captured image (see, for example, Patent Document 2). Recently, a technique for displaying an actual image has been developed (see, for example, Patent Document 3).
Japanese Patent No. 3099103 Japanese Patent Laid-Open No. 6-83937 JP 2005-3752 A

An object of the present invention is to display data configured by associating image data obtained by photographing a surrounding environment of a road network having a graph structure and geographic information including a surrounding map.

  In the conventional technique, only one type of image can be managed in association with one piece of map information, and therefore, it has been impossible to select and display from a plurality of images having different seasons and time zones.

  In other words, it was impossible to simultaneously manage images taken in different seasons and time zones, and to select and display images of the seasons and time zones desired to be viewed.

  Therefore, it is necessary to display an optimal image that satisfies the condition desired to be viewed together with a map by specifying conditions or searching using data configured by associating a single map information with a plurality of images.

  Further, in searching for images, a large-scale database is searched from a large number of image files, so a high-speed search method is required. In order to be able to use it in combination with existing geographic information, for example, when an arbitrary point (for example, a building) is selected, a surrounding image having coordinates near the coordinates of the point as an attribute is searched, and the image It is necessary to be able to calculate the azimuth angle based on the traveling direction of and to display the target building on the front.

  Furthermore, as an advanced image search method, it is necessary to perform not only search by location but also semantic search of an image, that is, search based on the contents of an object being captured.

  For example, it is necessary to search for an image corresponding to geographical contents such as an image near a park or an image near a large store.

The present invention is obtained on the basis of the information of the position sensor, which is taken every fixed time when a moving body is equipped with a hybrid position sensor having an all-around photographing camera and a GPS and moves along a traveling path. An all-around image of the shooting point is extracted as a first all-around image from a plurality of all-around image information obtained together with the shooting position, the shooting center direction of the all-around shooting camera, the shooting date, and the shooting time zone. In the image data display system with associated geographic information, the first omnidirectional image is displayed on the screen in a computer graphic with a predetermined display range, and a map including the range of the shooting point is displayed in conjunction with the screen.
ID code and the traveling path start point of the both ends of the node ID number and該進path of said advancing path on the map, route network and having a link information including the vector information with coordinates and directions of the end point,該路-line network The first omnidirectional image at, and for each of the first omnidirectional images, the photographing position, the photographing center direction, the photographing order in the traveling path, and the ID number of the traveling path corresponding to the photographing position are included. A storage device that stores image associated information and stores the map; and
Means for reading image attachment information for each of the first omnidirectional images, extracting the shooting positions included in the image attachment information, and displaying them at corresponding positions on the map;
Means for reading, from the storage device, the image attachment information having the designated shooting position and the first omnidirectional image associated with the image attachment information when the shooting position is designated on the map; ,
Image display means for displaying the read first omnidirectional image as a second omnidirectional image in an image display area different from the map display area by the computer graphic in the predetermined display range;
Accompanying the instruction of continuous display, the imaging order of all the image auxiliary information having the ID number of the traveling path of the read image auxiliary information is read, and these images are arranged in ascending order or reverse order based on the imaging order. The first omnidirectional image associated with the attached information is sequentially read, and each time it is read, the first omnidirectional image is continuously displayed in the image display area as the third omnidirectional image by the computer graphic within the predetermined display range. Captured image display means;
When the photographing order is the number of either end of the both ends, the continuous display of the captured image display means is stopped, and the third omnidirectional image displayed when the stop is stopped is the fourth The ID code of the node at either end of the traveling path having the ID number of the image ancillary information associated with the fourth omnidirectional image is searched for as the omnidirectional image, and the ID code Means for retrieving the ID numbers of all the traveling paths connected to the related nodes, and reading the direction of the road center of the retrieved traveling paths;
For each direction of the road center of all the travel routes read, the direction of all the travel routes read from the photographing center direction on the fourth all-around image based on the direction and the photographing center direction. The azimuth for each direction of the road center is obtained, and the display position of the travel path selection polygon for selecting the travel path for each of these azimuths is displayed on the fourth all-around image at a predetermined height on the road. And obtaining a fifth omnidirectional image obtained by synthesizing the traveling path selection polygon on the fourth omnidirectional image at these display positions, and obtaining the fifth omnidirectional image within the predetermined display range in the computer. Means for graphic display;
And means for causing the captured image display means to read the image attached information having the ID number of the link information associated with the travel path selection polygon when the travel path selection polygon is selected. And

  In the present invention, it is obtained by photographing the surrounding environment of a road network having a graph structure (network information in which the intersection is a node and the continuous line segment indicating the road center line between the intersections is a link as shown in FIG. 17). Using data configured by associating a plurality of image data and geographic information including a surrounding map, an image corresponding to the corresponding point can be displayed while being linked with the map.

  When a road network is used as a graph structure, an intersection or the like as a node (a point at which a road end or road attribute is changed (a bridge end or a tunnel end)), a road center line connecting nodes as a link May be interpreted.

  In addition, by associating map information such as roads with an image and using the image as a geographic attribute representing a place, it is possible to visually convey the actual local situation to the user.

  Furthermore, since a plurality of images having different seasons and time zones can be managed in association with one piece of map information, it is possible to provide an image closest to the time and time conditions required by the user.

  In this embodiment, map information such as roads is associated with an image, for example, property information such as apartments and stores for real estate transactions, and route environment display from a town landmark or station to a target real estate property or store. By using video as geographical attributes to represent places, such as advertising activities in shopping streets, location and surrounding situation in emergency notifications such as firefighting and police, and local situation in road river management, the local real situation can be made available to users This is an image data display system with geographic information that can be conveyed visually.

 In addition, if a target town is acquired in a wide area and data is provided for a service, the user can freely select an arbitrary place to be examined. Further, for example, if an image taken at different seasons or at different times is prepared, it is an image data display system with geographic information related that can provide changes in the town and changes in the environment.

 As means for providing information to a large number of unspecified users in response to these needs, many uses can be made by using web distribution.

 In order to display and search the video database in association with the map, it is not the conventional management as two-dimensional geographic coordinates, but the point where the video was shot (shooting point), the direction taken, the time taken (season), etc. By managing multidimensional spatio-temporal information as metadata, it becomes possible to search and provide necessary information to the user.

 If a large amount of such video information is collected in advance, a business model can be constructed that can be quickly provided according to the range and position designated by the user. Also, when a new image is taken, if it is managed in the same way as already collected data, it can be easily reused upon request.

 Furthermore, if it is possible to use not only a single shooting of the same part but also a plurality of times, it is possible to provide an image in a state desired by the user, for example, an image specifying a season or time. Become. For example, if images with different shooting years can be used, it is possible to investigate secular changes in a specific district. In addition, it is possible to view the cityscape, sightseeing road, etc. in a state that the user wants to see, such as “I want to see the image at the time of cherry blossoms” or “I want to see the image at the time of autumn leaves”.

 In the case of wide roads, the buildings on the side close to the driving route appear well, but the buildings on the opposite lane may not appear well. In other words, it is necessary to perform reciprocal shooting in order to better express the image along the railway. Even in the case of reciprocal shooting, it may be managed in the same manner as in the case of the multi-period shooting as well as the traveling direction.

 In addition, a system that uses ancillary information (such as a shooting direction) attached to an image to search an image in which a specific feature is captured from a large amount of image data and displays the position of the image. It is.

  Next, embodiments of the present invention will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, it should be noted that the drawings are schematic.

  As shown in FIGS. 1 and 22, an image data display system 100 with a geographic information association according to an embodiment of the present invention includes a geographic display unit 101, a shooting position display unit 102, a shot image display unit 103, and a route search interlocking unit 104. , An image search means 105, and a storage device 106 for storing the map data 108 and the cut-out image data f4.

  The geographic display means 101 reads the map data 108 stored in the storage device 106, and displays the map while scrolling, switching scales, switching display items, switching display colors, and the like.

  The shooting position display unit 102 displays the image data f4 (image file, attached information of the image file (image acquisition coordinates / image acquisition direction / image acquisition link ID) stored in the storage device 106 on the map displayed by the geographic display unit 101. -Order within acquisition link-Image acquisition camera information (photographing height, etc.)-From the link graph structure (including a list of link IDs connected to each node), as shown in FIG. The user draws the upper image acquisition coordinates), and the user designates the drawing position with a mouse click or the like, and displays a photographed image at a desired photographing point.

  The photographed image display unit 103 includes an image display unit and an image continuous display unit.

  As shown in FIG. 23, when one of the shooting positions of the shooting position display means 102 displayed on the geographic display means 101 is designated by a mouse operation or the like, the image display unit cuts out image data of the image closest to the shooting position. A search is performed using the image acquisition coordinates of the image file attached information, and the closest image file is read and displayed.

  In addition to the simple display in which the frame image at the designated shooting position is displayed as it is, as shown in FIG. 25, the stored image is an all-round image (360 ° circumference is taken and equirectangular projection (vertical is vertical angle). In the case of an image photographed with a method in which the horizontal is projected linearly on a horizontal angle), an image photographed with arbitrary camera parameters (camera direction / focal length (angle of view)) from the photographing center is subjected to CG processing, etc. The display is synthesized by means of the camera parameter designation display.

  Furthermore, when the map data 108 includes annotation data (data consisting of a combination of the feature name / feature type on the map and the position coordinates on the map), as shown in FIG. From the image acquisition coordinates and image acquisition azimuth included in f4 and the annotation display position included in the map data 108), the azimuth of the object to which the annotation data is viewed from the image acquisition coordinates is calculated, and the corresponding in the displayed image Draw annotation data at the azimuth position.

  The continuous image display unit displays images having the same link ID sequentially in ascending or reverse order according to the shooting order in the acquired link, thereby continuously displaying the images as if they were moving, and creating a moving image. Continuous display can be controlled with a movement direction designation button and a pause button. The movement direction designation button is displayed as a symbol corresponding to the traveling direction of the image data associated with the same route. In addition, the moving speed can be designated by a moving speed designation box.

  If the order in the acquisition link is the first or last in the continuous image display, the continuous display is discontinued and the route is selected. In route selection, the node ID of the last displayed side is checked from the attached information of the image file included in the image data. Next, the graph structure included in the cut-out image data is examined, and a link connected to the node is searched. Further, by examining the image acquisition link ID included in the cut-out image data, if there is image data obtained by capturing the link, an arrow, a click button, or the like is drawn on each path displayed in the image, and the next To specify which route image to display.

  The route search interlocking means 104 uses the road network included in the map data 108 to search for a route from the designated start point coordinates to the end point coordinates, and displays the route on the map displayed by the geographic display means 101. For the portion where the cut-out image data f4 corresponding to the route exists, the shooting position of the image acquisition coordinates of the image is also drawn on the map displayed by the geographic display means 101.

  The route search interlocking unit 104 draws only the shooting positions of all image acquisition coordinates on the route during the route search, and does not draw the rest. At this time, when a shooting position on the route is selected, the other shooting positions are drawn in a color different from the others.

  The above-described road network includes, for example, a travel locus pattern, a road shape, a width, and the like.

  A network having a graph structure includes a road, a track, a sea route, a pipeline, an air route, a mine shaft, and the like, and may be collectively referred to as a route network.

  In addition, images from the start point to the end point are sequentially drawn.

  The image search means 105 searches for an image of the object being photographed using the annotation data including the facility name included in the map data 108. It is assumed that the note data includes a note character string to be drawn on the map, a note display position, and a note type.

  The image search is performed according to the procedure shown in the flowchart of FIG.

Step S200: Matching of note data and image For each piece of note data, a photographed image for which it is determined that the note is best reflected is searched. Typically, an image file having a shooting position coordinate closest to the note display position of the note data is allocated, and the image file attached information of the cut-out image data having this image file is searched and selected. However, if there is no image satisfying a predetermined criterion (for example, a distance threshold), there is no image corresponding to the annotation data.

Step S201: Keyword input A keyword including the name and type of an object to be searched is input. The types of features include public facilities, parks, and intersection names. The keyword specifies all or part of the feature name (such as “Shinjuku” if you want to check Shinjuku Station).

Step S202: Note data search The note data corresponding to the keyword is searched. At this time, only the note for which the corresponding image has been selected in step S200 is targeted. Matching with keywords is done for note strings or note types.

Step S203: Candidate candidate selection When a plurality of notes are matched with keywords, they are displayed on the screen and selected by the user.

Step S204: Image Display An image corresponding to the note selected by the user is cut out from the image data and displayed on the image display unit of the computer screen.

  Next, the map data 108 and the cutout image data f4 stored in the storage device 106 will be described.

  The map data 108 includes network data f0 such as roads, background maps, annotation data given to objects existing on the map, and map symbol data.

  The map data 108 may be general data created for use in a navigation system or the like, not data created exclusively.

  On the other hand, the cut-out image data f4 includes an image file, image file attached information, and graph structure information. The image file attached information includes image acquisition coordinates, image acquisition orientation, image acquisition link ID, shooting start side node ID, shooting end side node ID, and shooting count counted from the shooting start side for each image file. The order, graph structure information associated with the acquired image, and image acquisition camera information (such as shooting height) are included.

  The link graph structure indicates the connection relation of the link from which the image is acquired, and includes a node ID and a list of link IDs of links connected to the node for each node included in the data.

  The image acquisition direction is the direction of the central axis of the camera at the time of shooting.

  When the image is an all-around image, the center of the all-around image viewed from the shooting point is the corresponding orientation.

The image acquisition link ID corresponds to the ID of the center of the road where the image was taken when the road network is used as the graph structure.

  A node is a coordinate point for specifying, for example, an intersection of roads, a dead end, and the like. A link is a vector connecting each node, the coordinates of the start point of the link, the coordinates of the end point, the direction of the link, and the link direction. Includes data such as road width.

  The cut-out image data f4 is data created by the geographic data-related image data management system 10 described later.

  The image data display system 100 associated with geographic information may be a program that operates independently on a computer. Further, the cut-out image data f4 may be realized as a system on the Internet in which the data server and the map data 108 are arranged on a client or a data server.

  Next, the geographic data-related image data management system 10 that creates the cut-out image data f4 used in the geographic information-related image data display system 100 will be described.

  As shown in FIG. 1, the image data management system with geographic coordinates 10 includes an image capturing subsystem 20, an image data editing subsystem 30, an image data management subsystem 40, and a storage connected to the image data management subsystem 40. The apparatus 50 is comprised.

  First, the image capturing subsystem 20 is connected to the image sensor 22, and an image data collecting unit 24 that collects an image of the surrounding environment along the map link as the collected image data d1 together with the photographing time, and the hybrid position sensor 23. , And a moving means equipped with a position data collecting means 25 for collecting the photographing position coordinates of the image as the collected image attribute information d2 together with the coordinate measurement time.

  Here, the map link means a road between two points (from one intersection to the next intersection) having the graph structure shown in FIG. For convenience, one point is referred to as a start point and the other point is referred to as an end point, but this does not mean the directionality of the link from the start point to the end point.

  The moving means refers to an aircraft for flying along a map link and shooting from above, or a vehicle 21 traveling on a road of the map link.

  Subsequently, the image data editing subsystem 30 includes a map data display unit 31 that displays a map link, an image data display unit 32 that displays image data collected by the image data collection unit 24, a photographing time and a coordinate measurement time ( Link start and end point passage time registration means 33 for recording the start time and end point of the map link from GPS time), image thinning means 34 for thinning out the data from the image data at regular intervals, and actual photographing management information registration means 35; including.

  Further, the image data management subsystem 40 is a photographing plan means 41 for managing a photographing plan by the image photographing subsystem 20, and a photographing that captures the collected image data d1 and the collected image attribute information d2 edited by the image data editing subsystem 30. Information capturing means 42, data extracting means 44 for extracting information that meets the specified conditions from information recorded in the storage device 50, registration of information in the storage device 50, and deletion of information from the storage device 50 And management means 43 for printing information in the storage device 50.

  The storage device 50 includes a graph structure database 51 that records map link information, a camera information database 52 that records image sensor 22 information, a plan / actual shooting information database 53 that records shooting plan information, and shooting information. The collected image database 55 and the collected image attribute information database 56 for recording the collected image data d1 and the collected image attribute information d2 captured by the capturing unit 42 are associated with the information of each database for each map link as shown in FIG. And a link related information database 54 for recording management information.

  Data exchange between the image capturing subsystem 20, the image data editing subsystem 30, and the image data management subsystem 40 is performed via a portable storage medium such as a disk. The data communication means can be provided in the wireless communication using wired or wireless data communication.

(Image capture subsystem 20)
Next, each unit of the image capturing subsystem 20 will be described in detail. Here, the vehicle 21 will be described as an example of the moving means, but is not limited to the vehicle 21.

  As shown in FIG. 12, the image capturing subsystem 20 is a subsystem intended to capture a peripheral image along the map link used in the embodiment of the present invention.

  First, the image data collection means 24 uses one or a plurality of synchronized cameras attached to the vehicle 21 as the image sensor 22 to capture an image while traveling on a road that is a map link. At this time, in order to synchronize with the position information described below, the time transmitted from the global radio positioning system (GPS) detected by the hybrid position sensor 23 is assigned to each image frame obtained in time series. Then, it is recorded in the temporary memory 26 as the collected image data d1.

  The time is assigned to each image frame in units of 1/1000 second using the detected GPS time and the clock of the computer on which the image capturing subsystem 20 is mounted.

  Next, the position data collection means 25 uses the hybrid position sensor 23 combined with GPS and other sensors (gyro / vehicle speed sensor) attached to the same vehicle 21 to obtain position information (coordinates) obtained from the GPS, Vehicle speed information obtained from a vehicle speed sensor, acceleration information obtained from a gyro, and direction information are collected. Further, the GPS time is given to these pieces of information and recorded in the temporary memory 26 as the collected image attribute information d2.

  When the vehicle speed obtained from the hybrid type position sensor 23 is 0, the position data collecting unit 25 is provided with a process for instructing the image data collecting unit 24 not to collect image data to perform shooting during stoppage. You can also quit. As a result, when a stop state occurs due to a signal or traffic jam, the recording of image data can be temporarily stopped to save the data collection capacity.

  Since the collected image data d1 and the collected image attribute information d2 are given time information using a common GPS time as shown in FIG. 16, the position and the image are associated with each other using the given time as a medium. be able to.

  The camera orientation can be obtained from the camera relative orientation relative to the vehicle 21 and the vehicle 21 orientation. The relative orientation is measured in advance. Since the image data collection cycle is usually shorter, the position / velocity / orientation, etc. in each image frame can be obtained by interpolating the position / velocity / orientation at the previous and subsequent times.

  The collected image data d1 and the collected image attribute information d2 recorded in the temporary memory 26 are output to a portable storage medium such as a disk as a primary shooting information import file f1 and input to the next image data editing subsystem 30. The

  The position data collection unit 25 is a link included in the shooting plan information d0 created by the shooting plan unit 41 of the image data management subsystem 40 described later and output to the portable storage medium as a shooting information export file f3 described later. The position information obtained from the hybrid type position sensor 23 is superimposed and displayed on the computer screen. Thereby, it can be understood which map link of the collection target the operator has collected.

(Image data editing subsystem 30)
Next, each unit of the image data editing subsystem 30 will be described in detail.

  As shown in FIG. 3, the image data editing subsystem 30 uses the collected image data d2 collected by the image photographing subsystem 20 and output as the primary photographing information import file f1 in the embodiment of the present invention. The map information such as a road network is used for map information having a graph structure using the shape information of the link information that is edited and created by the shooting plan means 41 of the image data management subsystem 40 described later and output as the shooting information export file f3. It is a subsystem that aims to correlate and optimize.

  First, as shown in FIG. 4, the map data display means 31 has the same structure as a physical image database, which will be described later, as a primary image information import file f1 output from the image capturing subsystem 20 as input information. The link shape is displayed from the time-series position information included in the collected image attribute information d2 constituting the primary shooting information import file f1 and the map information included in the shooting information export file f3. The time series position information is displayed as a point sequence, and the link shape is displayed as a continuous line segment.

  Next, the image data display means 32 displays the collected image data d1 of the primary imaging information import file f1 as shown in FIG. When the image is displayed, the image collection position is displayed as a symbol in the map displayed by the map data display means 31. Further, the display of image data can be sequentially switched back and forth in time series.

  Next, the link start / end point passage time registration unit 33 sets the start point and end point of the link selected by the mouse on the map shown in FIG. Specify and register the corresponding transit time. The passing time is specified from the time of the time series position corresponding to the start point or the end point using the time series position information included in the collected image attribute information d2 of the primary shooting information import file f1, and the primary shooting information import file. There is a method of designating from the image capturing time given to the collected image data d1 of f1. The start time and end time of the link are in the same format as the shooting plan export file, together with information such as the shooting date and time, the camera information key ID used for shooting, and the physical storage location (such as the hard disk volume name) of the image to be stored. Is saved in the actual photographing result information d3.

  As a method of designating from the time series position, there are a manual designation method for designating a time series position corresponding to the start point or end point of the link with a mouse or the like, and an automatic designation method for automatically designating.

  The automatic designation method is achieved by searching for the time series position closest to the start point or end point of the link. A map matching technique used in an automobile navigation system, that is, a technique of automatically associating with a route along which the user is traveling in consideration of not only the position but also the traveling direction and speed may be used. In this case, the closest point is not necessarily selected.

  In the method of designating from the image shooting time, the collected image data d1 corresponding to the start time of the link is displayed on the image display screen. By looking at the roads and intersections captured in the image, if it is determined that the operator is misaligned at the intersection, etc., the previous or next image is displayed in time series by operating the time series screen. The photographing time of the image at the intersection position determined by the operator can be corrected and registered as the link start point or end point passage time.

  If the start point passage time of the link registered by the link start / end point passage time registration means 33 is earlier than the end point passage time, it is determined that the image was taken while traveling from the start point to the end point on the corresponding link. . Conversely, if the start point passage time is later than the end point passage time, it is determined that the image is taken while traveling from the end point toward the start point on the corresponding link.

  The link start and end passage times registered by the link start / end point passage time registration means 33 are output to the secondary imaging information import file f2.

  Finally, the image thinning unit 34 thins the collected image data d1 of the primary photographing information import file f1 according to the movement distance or the like for the purpose of reducing the data capacity. As an example of the thinning method, there is the following method as shown in FIG. Starting from one frame, the distance from the starting frame to the i-th frame is calculated for each frame by the following formula.

[Equation 1]
L [0] = 0.0;
L [i] = L [i-1] + (v [i-1] + v [i]) / 2 * (t [i] -t [i-1]);
Where v [i]: speed of i-th frame (meter / second)
t [i]: Time of the i-th frame (second)
Let d be the thinning interval, and let L [jn] be the distance closest to n times d (n is an integer). Frames other than jn are thinned out. If the time information is also attached to the image frame after thinning, the image can be thinned without affecting other parts of the database.

  For example, when shooting at 15 frames per second from a car traveling at 10 km / h, 54,000 images are acquired in a 10 km section. If this is thinned out every 1 m, it becomes 10,000 sheets, and the data amount can be suppressed to 1/5 or less.

  In this way, the amount of data can be reduced by thinning out the number of image frames to such an extent that there is no human visual problem, and the amount of memory necessary for storage can be reduced, so that the system can be made inexpensive. Furthermore, if the amount of data to be subjected to image display processing is small, the display processing speed can be improved and the burden on a display device such as a computer can be reduced.

  In the case where it is not necessary to reduce the amount of data due to the low cost of the memory or the improvement of the processing capacity of the computer, the image thinning means 34 is not particularly required.

  The actual shooting management information registration means 35 inputs attribute information (shooting area information / worker, etc.) at the time of actual shooting registered in a plan / real shooting information database, which will be described later, and inputs it into the actual shooting management information d4.

(Image data management subsystem 40)
Next, each unit of the image data management subsystem 40 will be described in detail.

  As shown in FIG. 7, the image data management subsystem 40 manages the collected image data d1 actually shot and edited from the plan / progress management of the place to be shot and the equipment used for shooting, and the like. This subsystem is intended to collectively manage a data group necessary for extracting and providing the collected image data d1 that meets the user's requirements.

  First, as shown in FIG. 8, the shooting plan means 41 includes a screen display unit 411, a link selection unit 412, a camera information input unit 414, a shooting plan information input unit 413, and a shooting information export file creation unit 415. .

  The screen display unit 411 displays the nodes and links of the graph structure database 51 recorded in the storage device 50 on the computer screen.

  The link selection unit 412 selects and determines a link for which shooting is planned from among the links displayed by the screen display unit 411. Link selection methods include rectangular area specification by mouse operation on the screen, selection by position such as polygon area specification, selection by attribute information included in graph structure information such as road width and type, and composite selection by position and attribute information is there.

  For example, when a prefectural road of a certain municipality is extracted, only the prefectural road is extracted in advance by selection based on attribute information, and then the municipality boundary is designated by polygon designation, thereby achieving the purpose.

  The camera information input unit 414 records the input camera information used for shooting in the storage device 50 as the camera information database 52. At this time, a camera information key ID that is automatically or arbitrarily designated is assigned so that individual camera information can be identified.

  The shooting plan information input unit 413 receives input shooting plan information such as a shooting area (a group of links selected by the link selection unit 412), a shooting start date, a shooting operator, and a camera information key ID of a camera to be used. The plan / actual shooting information database 53 is recorded in the storage device 50. At this time, an imaging plan key ID designated automatically or arbitrarily is assigned so that each imaging plan can be identified.

  When the shooting plan ID is automatically generated, a combination of the creation date and the camera information key ID is used so that the IDs do not overlap.

  The shooting information export file creation unit 415 combines the link selected by the link selection unit 412 and the information input by the shooting plan information input unit 413, and links in the storage device 50 as link related information in the format shown in FIG. Record in the related information database 54. At this stage, the link related information and the link shape recorded in the graph structure database are further output to the portable storage medium as a photographing information export file f3.

  Next, as shown in FIG. 9, the photographing information capturing unit 42 includes an actual photographing management information recording unit 421 and an image data recording unit 422.

  The actual shooting management information recording unit 421 assigns the actual shooting key ID to the actual shooting management information d3 such as the photographer at the time of shooting, which is input from the image data editing subsystem 30 as the secondary shooting information import file f2. Record in the plan / actual imaging information database 53.

  The actual shooting ID is a combination of the shooting date / time and the camera information key ID used so that the IDs do not overlap.

  Further, the shooting information related to the shot link, that is, the shooting date and time, the camera information key ID used for shooting, the passage time of the start and end points of the link, and the physical storage location (such as the volume name of the hard disk) of the stored image, etc. The actual photographing result information d3 storing the information is updated to the corresponding link key ID portion of the link related information database 54 created and stored for each link key ID by the photographing information export file creation unit 415.

  In addition, the image data recording unit 422 receives the collected image data d1 and the collected image attribute information d2 input from the image data editing subsystem 30 as the secondary shooting information import file f2, and the collected image database 55 and the collected image attribute information database, respectively. 56.

  As shown in FIG. 10, the management unit 43 includes a network registration unit 431, a printing unit 432, and a maintenance unit 433.

  The network registration unit 431 reads road network data created and distributed externally and registers it in the graph structure database 51.

  The printing unit 432 prints information of each database recorded in the storage device 50 with a printer.

  The maintenance unit 433 periodically backs up each database recorded in the storage device 50, deletes unnecessary data (such as shooting plan information that is no longer shot), and the like. In addition, by searching for the work progress code in the link related information database 54 in which the shooting plan information d0 and the actual shooting management information d3 are reflected, the degree of shooting progress is totaled.

  Finally, as shown in FIG. 11, the data extraction unit 44 includes an extraction condition input unit 441, a graph structure information extraction unit 442, a link related information extraction unit 443, a collected image data extraction unit 444, and a collected image attribute information extraction unit. 554.

  The extraction condition input unit 441 inputs conditions for extraction from the database. The conditions include a positional condition, a link attribute condition, and a shooting attribute condition. The positional conditions include a rectangular area, a polygon area, and the like obtained from a mouse operation on the screen and file reading.

  The link attribute condition includes designation of attribute information included in the graph structure information such as road width and type. The shooting attribute condition includes designation of a shooting date and a shooting time zone.

  The graph structure information extraction unit 442 extracts, from the graph structure database 51, a link that matches the positional condition and the link attribute condition input by the extraction condition input unit 441.

  Further, the extracted link ID and the node IDs at both ends thereof are checked, the link IDs connected to each node are listed, and stored as the graph structure information of the link of the cut-out image data.

  The link related information extraction unit 443 extracts the link corresponding to the link ID extracted above from the link related information database 54. Further, link information that matches the photographing attribute condition input by the extraction condition input unit 441 is extracted from the information.

  The collected image data extraction unit 444 extracts information that meets the conditions input by the extraction condition input unit 441 from the collected image database 55.

  That is, the collected image data extraction unit 444 searches for an image photographed with the camera information key ID on the photographing date stored in each link related information extracted from the collected image database. Further, an image photographed between the viewpoint passage time and the end point passage time is extracted from the image, cut out as an image file, and stored as an image file in the image data.

  Further, the image acquisition link ID, the node ID on the shooting start side, the node IDs on both sides of the shooting collection, and the shooting order are cut out and stored as image file attribute information in the image data.

  The collected image attribute information extraction unit 445 extracts information that meets the conditions input by the extraction condition input unit 441 from the collected image attribute information database 56.

That is, the image acquisition coordinate corresponding to the image files extracted above, extracts information such as an image acquisition orientation, and stores the image file attributes information in the cutout image data.

  The information cut out as described above is output as cut-out image data f4. This includes image file / image file attached information and link graph structure.

  Note that the link can be cut out by the following method.

  Specification of positional conditions (extraction by geographic mesh or administrative area): The link to be extracted can be selected by specifying a mouse on the computer screen interface or specifying an area with the mouse. Also, the inside of the outer polygon such as the administrative area may be registered in advance, and the link in the outer polygon corresponding to the area designated by the user may be extracted.

  For example, the Tokyo Metropolitan City / Municipal border is created separately as a polygon, saved as an external file on the magnetic disk, and when the polygon is read when necessary, the polygon is input with the mouse. It can be used as an alternative.

  -Specifying link attribute conditions (road type, road width, etc.): If you specify the type of link required for shooting and refer to the information stored in the graph structure database 51, it corresponds to the type of link Can be extracted.

  Specification of shooting attribute condition: By referring to shooting date / time information included in the link related information database 54, it is possible to specify and extract the shooting time of the image.

  On the other hand, an image corresponding to each link is extracted based on information described in the link related information database 54. That is, from the collected image database 55 recorded in the storage location of the storage device 50 described in the record corresponding to the link key ID of the link to be extracted, the camera described in the camera information key ID is photographed, and What is necessary is just to extract the time which is attached to the image frame between the start point passage time and the end point passage time.

  The cut out image data can include the following.

-Image file-Image file attribute information-Image acquisition coordinates (latitude and longitude of each frame)
-Image acquisition direction (image center direction)
-Image acquisition link ID (ID of the road centerline that captured the image)
・ Node ID on the shooting start side
・ Node ID at the end of shooting
・ Shooting order ・ Image acquisition camera information (shooting height etc.)
-Link graph structure-For each node ID, a list of link IDs connected to that node (each database in the storage device 50)
Next, each database recorded in the storage device 50 will be described.

  As shown in FIG. 1, the database used in the embodiment of the present invention is roughly classified into an image information database 50a for managing the relationship between feature information having a graph structure and a photographed image, and an actual photographing. Divided into two groups, a physical image database 50b for storing the captured images and their associated information.

  The image information database 50a stores a graph structure database 51 for storing a graph structure, a camera information database 52 for storing information attached to shooting such as a camera used for shooting, a shooting plan, and information at the time of actual shooting. The plan / actual shooting information database 53 corresponding to each link and the link related information database 54 for associating the above information and the shot image for each link belong.

  The graph structure database 51 stores feature information having a graph structure as shown in FIG. When a road network is used as geographic information, a road center line connecting the intersection as a node of the graph structure and the intersection as a link corresponds.

  In the graph structure database 51, as shown in FIG. 12, each node is defined by geographic coordinates and identified by a node key ID. Each link is defined by the geographical coordinates (or node key ID) at both ends and the link shape, and is identified by the link key ID. Note that both ends of the link (road center line) are connected to nodes (intersections), and a node key ID indicating the intersection of the start point and end point of the link is assigned to each link. As a result, the connection relationship between the node and the link, that is, the network structure becomes clear.

  In the graph structure database 51, attached information can be attached to nodes and links. For example, in the case of a road network, it is possible to give information such as an intersection name if it is an intersection, and information on road width and regulation information (such as one-way traffic) if it is a link.

  Next, as shown in FIG. 13, the camera information database 52 stores device information such as a camera to be photographed, a mounted vehicle, and a camera height, and is identified by a camera information key ID.

  In the plan / actual shooting information database 53, shooting plan information d0 and actual shooting management information corresponding to each link are recorded as shown in FIG. That is, the operator, the camera information key ID of the camera used for the work, and the date of image capture. Each shooting plan and actual shooting management information are identified by a shooting plan key ID and an actual shooting key ID, respectively.

  As shown in FIG. 15, the link related information database 54 includes a link key ID, a shooting plan key ID, an actual shooting key ID, a work progress code indicating a shooting status (planned, shot, edited, not shot, etc.), It has a table structure having a series of information such as a shooting date, a camera information key ID used for shooting, a passage start time of a link start point and an end point, and a physical storage location (such as a hard disk volume name) of an image to be stored.

  After the shooting plan, a record in which only items not related to actual shooting, such as a link key ID, a shooting plan key ID, and a camera information key ID scheduled to be used, are set is recorded for a link scheduled to be shot. The At this time, the work progress code is not photographed.

  After shooting, the actual shooting key ID, shooting date and time, and camera information key ID used for shooting are stored, and the work progress code is updated.

  The link related information database 54 is a table that can have a plurality of records having the same link key ID, as indicated by the hatched portion in FIG. For this reason, it is possible to store a plurality of photographing results for one link key ID.

  As shown in FIG. 16, the collected image database 55 and the collected image attribute information database 56 belong to the physical image database.

  The collected image database 55 records a series of images actually taken in time series, as well as shooting camera information (camera information key ID) used for collecting each image, shooting date, and shooting time. .

  The collected image attribute information database 56 stores information such as the shooting position, shooting speed, shooting direction, shooting date, shooting time (GPS time), and shooting camera.

  Since the information in the collected image database 55 and the information in the collected image attribute information database 56 each have time information, they are associated with each other according to the time information.

  In the embodiment of the present invention, only one storage device 50 for storing the database is illustrated, but the storage device 50 is not limited to one. The physical image database 50b may be stored in a storage medium physically different from the image information database 50a in order to store a large amount of image data. Actually, it is divided and stored in a large-capacity hard disk of 100 gigabytes or more.

  In the database structure according to the embodiment of the present invention, images of a plurality of periods can be simultaneously stored for one link.

  Next, a procedure in the image data management system with a geographic information association according to the embodiment of the present invention will be described with reference to FIGS.

(Data registration)
Step S100: The link information from the external road network data f0 is registered in the graph structure database 51 by the management means 43 of the image data management subsystem 40.

(Shooting plan)
Step S101: The graph structure of the link is displayed from the link information of the graph structure database 51 by the screen display unit 411 of the photographing plan means 41 of the image data management subsystem 40.

  Step S102: The link selection unit 412 of the photographing plan means 41 of the image data management subsystem 40 selects a link to be photographed from the displayed link graph structure.

  Step S103: Information relating to the camera used for shooting is registered in the camera information database 52 by the camera information input unit 414 of the shooting plan means 41 of the image data management subsystem 40.

  Step S104: The shooting plan information input unit 413 of the shooting plan means 41 of the image data management subsystem 40 draws a shooting plan for the extracted shooting target link and inputs it to the plan / actual shooting information database 53.

  Step S105: The shooting plan information d0 is output to the shooting information export file f3 by the shooting information export file creation unit 415 of the shooting plan means 41 of the image data management subsystem 40.

(Shooting)
Step S106: The link shape in the information of the shooting information export file f3 is read into the image shooting subsystem 20, and the link to be shot is displayed.

  Step S107: The image is taken while actually traveling on the road of the link by the moving means (vehicle 21), and the collected image data d1 to which the photographing time is given is collected by the image data collecting means 24 of the image photographing subsystem 20. .

  Step S108: Collected image attribute information d2 to which the GPS time is added is collected by the position data collecting means 25 of the image capturing subsystem 20.

  Step S109: The image capturing subsystem 20 outputs the collected image data d1 and the collected image attribute information d2 together with the actual shooting result information to the primary shooting information import file f1.

(Image data editing)
Step S110: The photographing information export file f3 and the primary photographing information import file f1 are input to the image data editing subsystem 30.

  Step S111: Register actual photographing management information.

  Step S112: Map data is displayed by the map data display means 31 of the image data editing subsystem 30.

  Step S113: The image data display means 32 of the image data editing subsystem 30 displays the images collected for the link specified on the map data.

  Step S114: Time information is obtained from the displayed collected image data d1 and the collected image attribute information d2, and the link start and end point passage time registration means 33 of the image data editing subsystem 30 passes the start and end points of the corresponding link. Record the time.

  Step S115: Further, the image thinning means 34 of the image data editing subsystem 30 thins out the image from the collected image data d1 as necessary to reduce the data amount.

  Step S116: The edited information is output to the secondary shooting information import file f2.

(Information recording / update)
Step S117: The information of the secondary shooting information import file f2 is input to the image data management subsystem 40.

  Step S118: The actual photographing information data is recorded in the plan / actual photographing information database 53 by the photographing information capturing means 42 of the image data management subsystem 40.

  Step S119: The information in the link related information database 54 is updated by the photographing information capturing means 42 of the image data management subsystem 40.

(Data extraction)
Step S120: The data extraction means 44 of the image data management subsystem 40 extracts data from the collected image database 55 based on the information in the link related information database 54 in accordance with the user's request conditions. Output as f4.

(display)
Step S121: The image data display system with geographic information association 100 of the present invention displays a map and a peripheral image of a link (road) designated by a cut-out image data f4 and commercially available map data.

  The following effects can be obtained by the image data management system with geographic information described above, and the created cut-out image data is read by the image data display system with geographic information 100 to meet the user's request. Images can be displayed.

  In other words, by cooperating with the image capturing subsystem 20, it is possible to manage information such as which district has not been captured and how much shooting has progressed.

  In addition, since a shooting plan in which a necessary part is cut out from graph structure data such as a road can be created, shooting work can be performed in parallel using a plurality of vehicles 21 based on the shooting plan. Can be collected in large quantities.

  Since the camera information (camera shooting height, camera identification number, camera calibration information of each identification number) used for shooting is centrally managed as the camera information database 52, it is later used using the shot images. Can be measured from.

  Based on the data structure of the link-related information database 54, an actually captured image is managed in association with map information having a graph structure such as a complex road network including a Y-junction, an intersection of five or more roads, and a three-dimensional intersection. be able to.

  In addition, since a plurality of types of images can be managed in association with the same link, images taken in different seasons and time zones can be provided according to the user's request.

  Furthermore, when actually providing image data to a user, necessary data can be extracted from the entire image data photographed over a wide area by specifying the following conditions.

・ Geospatial segmentation (segmentation by geographic mesh or administrative area)
・ Cut out by link type (road type, road width, etc.)
-Extraction of image according to photographing time The means of each subsystem and the arrangement of data held in the drawings for explaining the embodiment of the present invention are merely examples, and are for realizing the functions of the present invention. It is not limited to. Therefore, it goes without saying that various combinations of arrangements of processing means are conceivable within a range in which the functions of the present invention can be realized.

  Hereinafter, the image data display system 100 with geographic information will be described in detail, but as a first stage, the description will be supplemented with respect to the primary shooting information import file f1, the secondary shooting information import file f2, the cutout image data f4, and the like.

  The primary shooting information import file f1 in FIGS. 1, 2, 3, and 7 includes time-collected image collection data d1 and collected image attribute information d2. The collected image attribute information d2 includes position information (coordinates) obtained from GPS, vehicle speed information obtained from a vehicle speed sensor, acceleration information and direction information obtained from a gyro, GPS time, and the like.

  The shooting position display means 102 reads the collected image attribute information d2 of the cut-out data f4, and displays this time-series information (image acquisition coordinates) as a symbol (for example, a circle: a shooting position hereinafter) as shown in FIG. Is possible.

  This will be described with reference to FIG. FIG. 31A shows the shooting position when the vehicle is shot at regular time intervals while being accelerated or slowed down.

FIG. 31 (b) shows that collected image data d1 (with shooting time) captured at a fixed interval stored in the primary shooting information import file f1 at this time is arranged on the distance axis (coordinates). .

  The map data display means 31 in FIG. 1 reads the image acquisition coordinates of the collected image attribute information d2 having the time of the collected image data d1, for example, and the image acquisition coordinates are defined in the distance axis (XY coordinate system). Allocated memory).

  At this time, the image thinning means 34 is involved.

  As shown in FIG. 31 (c), the image thinning means 34 sequentially sets the thinning interval d (distance) that does not cause human visual problems to the distance axis in order from the shooting start point, and for each thinning interval d. When a perpendicular is drawn with respect to the distance axis in FIG. 31B, it is determined whether or not the image acquisition coordinates (shooting position) of the collected image data d1 exist. 31 (d) is assigned to the distance axis as a representative photographing point (also simply referred to as a photographing position) as shown in FIG. 31 (d), and the collected image data d1 related to the assigned image acquisition coordinates (photographing position) and Only the collected image attribute information d2 is extracted and stored in the secondary shooting information import file f2.

  In this extraction, for example, time is extracted as a key. That is, only the collected image data d1 and the collected image attribute information d2 at the time when the portion of the thick line (representative photographing point) in FIG.

  On the other hand, when the representative photographing points are defined on the distance axis, the link start / end point passage time registration unit 33 obtains the actual photographing result information d3 having the time of the collected image attribute data d2 including the position information of these representative photographing points. Extracted from the export file f3 and stored in the secondary shooting information import file f2.

  Further, when the representative photographing points are defined, the actual photographing management information registering means 35 performs actual photographing management information d4 (with the time of the collected image attribute data d2 having image acquisition coordinates (photographing positions) of these representative photographing points. Shooting area information, operator information, etc.) are extracted from the shooting information export file f3 and stored in the secondary shooting information import file f2.

  That is, the secondary shooting information import file f2 stores the collected image data d1, the collected image attribute information d2, the actual shooting result information d3, the actual shooting management information d4, and the like of each representative shooting point.

  Then, the actual shooting management information recording unit 421 of the shooting information capturing unit 42 of the image data management subsystem 40 assigns an actual shooting key ID to the actual shooting management information d3 of the secondary shooting information import file f2 to plan / actual shooting. Record in the information database 53.

  Further, the image data recording unit 422 records the collected image data d1 and the collected image attribute information d2 of the secondary shooting information import file f2 in the collected image database 55 and the collected image attribute information database 56, respectively. The actual photographing management information d4 is stored in the plan / actual photographing information database 53.

Further, as shown in FIG. 11, the data extraction unit 44 includes an extraction condition input unit 441, a graph structure information extraction unit 442, a link related information extraction unit 443, a collected image data extraction unit 444, and a collected image attribute information extraction unit 554. Consists of
The extraction condition input unit 441 inputs conditions for extraction from the database. The conditions include a positional condition, a link attribute condition, and a shooting attribute condition. Positional conditions are also referred to as spatial clipping conditions. There are rectangular areas and polygonal areas obtained from mouse operations on the screen and file reading, and various information (image files, attribute information, etc.) in these areas are clipped images. It is called data.

  The link attribute condition includes designation of attribute information included in the graph structure information such as road width and type. The shooting attribute condition includes designation of a shooting date and a shooting time zone.

  The graph structure information extraction unit 442 extracts, from the graph structure database 51, a link that matches the positional condition and the link attribute condition input by the extraction condition input unit 441.

  Further, the extracted link ID and the node IDs at both ends thereof are checked, the link IDs connected to each node are listed, and stored as the graph structure information of the link of the cut-out image data.

  The link related information extraction unit 443 extracts the link corresponding to the link ID extracted above from the link related information database 54. Further, link information that matches the photographing attribute condition input by the extraction condition input unit 441 is extracted from the information.

  The collected image data extraction unit 444 extracts information that meets the conditions input by the extraction condition input unit 441 from the collected image database 55.

  The collected image data extraction unit 444 searches for an image taken with the camera information key ID on the shooting date stored in each link related information extracted from the collected image database. Further, an image photographed between the viewpoint passage time and the end point passage time is extracted from the image, cut out as an image file, and stored as an image file in the image data.

  Further, the image acquisition link ID, the node ID on the shooting start side, the node IDs on both sides of the shooting collection, and the shooting order are cut out and stored as image file attribute information in the image data.

  The collected image attribute information extraction unit 445 extracts information that meets the conditions input by the extraction condition input unit 441 from the collected image attribute information database 56.

  That is, information such as image acquisition coordinates and image acquisition orientation corresponding to the image file extracted above is extracted and stored as image file attribute information image file attributes in the cut-out image data.

  That is, the data extraction unit 44 extracts all the cut-out image data f4 (graph structure information, link related information, image file, image file attribute information) corresponding to the representative shooting point shown in FIG. 31D of the cut-out condition (area: area). ..) Is stored in the storage device 106 (see FIG. 31F).

  As described below, the cut-out image data f4 is used to display a map and an image file by the image data display system with geographic information association 100.

  FIG. 27 is a more specific configuration diagram of the image data display system 100 with geographic information association.

  As shown in FIG. 27, an image data display system 100 with a geographic information association includes a traveling direction mark display means 107, a map, in addition to a storage device 106, a geographical display means 101, a photographing position display means 102, and a photographed image display means 103. An upper photographing direction display unit 110, an annotation data / registration display unit 109, and the like are provided.

  The storage device 106 stores note data related information f6 and the like in addition to the map data 108 and the cutout image data f4.

  The note data related information f6 is preferably registered in advance by the note data / registration display means 109.

  The aforementioned map data 108 preferably has a hierarchical structure. For example, a hierarchical structure including a background map field, network information field, annotation data field, map symbol data field, and the like. The annotation data is the name of the feature, and this name is associated with the coordinates of the annotation data field.

  Further, the storage device 106 stores a plurality of pieces of cut-out image data f4 within the cut-out condition area. It is preferable that an ID code is given to these cut-out image data f4. More specifically, the data structure shown in FIG.

  Furthermore, the physical image database has a data configuration shown in FIG. 30 more specifically.

  When the region (display range) of the clipping condition is input, the geographic display unit 101 displays the background map, network information, note data, and map symbol in this region in an overlapping manner. At this time, only the note data in the vicinity of the road of the network data may be extracted and displayed.

  The annotation data / registration display means 109 has a function of displaying the annotation data on an image of a computer screen as shown in FIG.

  In order to realize the display of the annotation data, the annotation data / registration display means 109 searches for the image acquisition coordinates existing within a certain distance from the coordinates of the annotation data of the map data (for example, the coordinates of the center character of the annotation data). Among these image acquisition coordinates, an image acquisition coordinate closest to the note data is searched. Then, the ID of the cut-out image data having the image acquisition coordinates is registered in advance in the storage device 106 as the annotation data related information f6 together with the annotation data, the coordinates of the annotation data, and the like.

  For example, as shown in FIG. 33A, when the network data f0 is a road, the image acquisition coordinates exist as Ga, Gb... And the note data exists as Ca, Cb. The cut-out image data having the image acquisition coordinates at the closest distance to the coordinates of the annotation data Ca, Gb. Then, the direction with respect to the coordinates of the note data is obtained from the image acquisition coordinates of the retrieved cut-out image data. This azimuth is indicated by how many times (this angle is θk) from the camera center.

  For each piece of note data, Gb..., The ID of the cut-out image data having the closest image acquisition coordinates is made to correspond, and the azimuth θk and the image acquisition coordinates are made to correspond (FIG. 33B).

  The image of the image file linked to this ID is a 360-degree panoramic image, and the center position of the image corresponding to the camera center is the camera center (0 degree). The coordinates of the annotation data on the image are obtained at the position corresponding to the height of the line of sight in the direction of θk from the camera center. That is, a memory defining a field (XY coordinate system: corresponding to the coordinate system of the panoramic image) for displaying the note data is provided, and the coordinates of the note data on this memory (referred to as the note data display position on the image). ) Is defined.

  That is, the note data related information f6 includes the ID of the adjacent clipped image data, the note data display position, the direction (θk), the characters of the note data, and the like.

  The geographic information unit 101 reads out map data corresponding to the input region of the extraction condition from the storage device 106 and displays it on the computer screen. This map data includes a background map, annotation data, map symbol data, and the like.

  The shooting position display means 102 extracts the image acquisition coordinates of all the clipped image data f4 stored in the storage device 106, and images are displayed at positions on the map (for example, shooting points on the road) corresponding to these coordinates. A symbol (referred to as an imaging position symbol) indicating the acquired coordinates (imaging position) is displayed (see FIGS. 23 and 24).

  That is, since all the shooting position symbols existing in the region for the cutout condition are displayed, the operator can determine at a glance where the image is on the map.

  When the operator designates any of these photographing position symbols with a mouse or the like, the photographing position display means 102 displays the designated photographing position symbol with another symbol (for example, cross, double circle, different color). ).

  When the shooting position symbol on the map on the screen is designated, the map traveling direction mark display means 107 sets the shooting order of images having coordinates corresponding to the shooting position symbol to the start number or end number in the same link. Determine whether or not.

  If it is the start number or end number, it is determined as an intersection. When it is determined as an intersection, the connection link ID associated with the intersection ID of the link is read.

  Then, the coordinates of the image in the vicinity of the intersection on the connection link ID searched from the coordinates of the intersection are read, and the orientations Hi (3 or 4 differences,...) Of all the links connected to the intersection are obtained. According to the direction, an arrow is displayed on the link (see FIG. 28).

  On the other hand, when a shooting position symbol on the computer screen is selected, the shot image display means 103 extracts an image file of cut-out image data having image acquisition coordinates of the shooting position symbol, and displays the operator input from this image file. Display the range image on the screen.

  The image file of the cutout image data described above is a 360-degree panoramic image. For this reason, the operator designates a display range of about 50 degrees on the left and right, for example, with the photographing direction as the center.

  The image of the image file in the display range is displayed on the computer screen (see FIG. 28), and the display range and the shooting direction are notified to the on-map shooting direction display means 110.

  When the shooting direction and the display range are notified, the on-map image display direction display unit 110 displays the display range (from the shooting direction to the current display range, from the image acquisition coordinates (shooting position) on the map on the computer screen). In the above case, polygons are generated in an angle direction of 50 degrees on the left and right, and these polygons are displayed on a map on the computer screen (see the sector in the map display range in FIG. 28).

  Further, the on-image traveling direction mark display means 107 obtains the direction of the connection link (road) included in the shooting range in the shooting direction from the intersection.

  The direction of this connecting link is defined by an angle. For example, as shown in FIG. 32A, when the shooting direction is the camera center (0 degree), the direction of the center line of the road ka from the camera center is θp, and the direction of the center line of the road kb is θi. Can be defined.

  The direction of θp and the direction of θi are determined from the center (0 degree) of the display range, and triangular polygons (through) are displayed three-dimensionally at positions on the screen corresponding to positions on the road (FIG. 32). (C)).

  When this polygon is selected, the captured image display means 103 sequentially extracts the image file of the shooting position of the road in the selected direction and displays it according to the display range.

  On the other hand, the note data registration / display means 109 reads the specified note (XX apartment, XX hospital, XX city hall,...), And is registered in the ID of the image being displayed and the note data related information f6. When the image data IDs match, the close note data is extracted from the map data.

  Next, the extracted note data is displayed on an image on a computer screen.

  The display of the note data reserves the note data related information f6 having the image acquisition coordinates (shooting position) when the shooting position symbol on the road on the computer screen has moved to the shooting position. Then, the character string of the note data is displayed on the computer screen according to the note data display position and direction of the assigned note data related information f6 (see FIG. 24).

  Therefore, the target object can be found immediately by simply looking at the computer screen.

  In the above-described embodiment, when the cut-out image data fi is generated, the cut-out image data fi is displayed. However, the image includes images related to privacy such as a face and a license plate. Such an image related to privacy may be subjected to a blurring process using a Gaussian filter or the like, and stored in the storage device 106 as cut-out image data fi. As a result, privacy-related images are displayed in a blurred manner on the display unit.

It is a block diagram of a map information correlation image data management system and a display system. It is a block diagram of an image photographing subsystem. It is a block diagram of an image data editing subsystem. It is a map displayed by a map data display means. The image data display means is an image displayed on the road surface. It is a figure which shows the method of the image thinning-out by an image thinning means. It is a block diagram of an image data management subsystem. It is a block diagram of a photography plan means. It is a block diagram of an imaging | photography information acquisition means. It is a block diagram of a management means. It is a block diagram of a data extraction means. It is a structure figure of a graph structure database. It is a structural diagram of a camera information database. It is a structure figure of a plan and real imaging | photography information database. It is a structural diagram of a link related information database. It is a structure figure of a physical image database. It is a figure which shows the feature with a graph structure. It is a flowchart explaining the procedure in the image data management system with a geographic information association by embodiment. It is a flowchart explaining the procedure in the image data management system with a geographic information association by embodiment. It is a flowchart explaining the procedure in the image data management system with a geographic information association by embodiment. It is a flowchart explaining the procedure in the image data management system with a geographic information association by embodiment. It is a block diagram of a map information correlation image data display system. It is an example of a display screen of a map information correlation image data display system. It is an example of a note overlay display of a display screen of a map information association image data display system. It is an example of a display screen of a map information correlation image data display system. It is a flowchart of the image search of a map information correlation image data display system. It is the block diagram which made more concrete the image data display system 100 with a geographic information relation. It is explanatory drawing which displays an imaging | photography position and a corresponding image simultaneously. It is explanatory drawing about the more specific structure of clipped image data. It is explanatory drawing explaining the more specific structure of a physical image database. It is explanatory drawing of the thinning-out of this Embodiment, a display of an imaging | photography position, and registration of cut-out data. It is explanatory drawing explaining a traveling direction mark display means. It is explanatory drawing explaining the display to the image of annotation data.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Geographic coordinate related image data management system 20 Image capturing subsystem 21 Vehicle (moving means)
22 Image sensor 23 Hybrid type position sensor 24 Image data collection means 25 Position data collection means 30 Image data editing subsystem 31 Map data display means 32 Image data display means 33 Link start / end point passage time registration means 34 Image thinning means 40 Image data management Subsystem 41 Imaging plan means 42 Imaging information fetch means 43 Management means 44 Data extraction means 50 Storage device 50a Image information database 50b Physical image database 51 Graph structure database 52 Camera information database 53 Plan / actual imaging information database 54 Link related information database 55 Collected image database 56 Collected image attribute information database 100 Image data display system 101 with geographic information association Map display means 102 Shooting position display means 103 Photographed image display hand 104 route search interlocking means 105 image search means 106 storage device 108 map data d0 shooting plan information d1 collected image data d2 collected image attribute information d3 actual shooting result information f1 primary shooting information import file f2 secondary shooting information import file f3 shooting information export File f4 Cutout image data

Claims (18)

When a moving body is equipped with an all-around shooting camera and a GPS-type hybrid position sensor and moves along a traveling path, images are taken at regular intervals, and the shooting position obtained based on the information of the position sensor, all An all-around image of the shooting point is extracted as a first all-around image from a plurality of all-around image information obtained together with the shooting center direction, the shooting date, and the shooting time zone of the surrounding shooting camera. In the image data display system with associated geographic information, the surrounding image is displayed on the screen with a computer graphic in a predetermined display range, and the map including the range of the shooting point is displayed in conjunction with the screen.
ID code and the traveling path start point of the both ends of the node ID number and該進path of said advancing path on the map, route network and having a link information including the vector information with coordinates and directions of the end point,該路-line network The first omnidirectional image at, and for each of the first omnidirectional images, the photographing position, the photographing center direction, the photographing order in the traveling path, and the ID number of the traveling path corresponding to the photographing position are included. A storage device that stores image associated information and stores the map; and
Means for reading image attachment information for each of the first omnidirectional images, extracting the shooting positions included in the image attachment information, and displaying them at corresponding positions on the map;
Means for reading, from the storage device, the image attachment information having the designated shooting position and the first omnidirectional image associated with the image attachment information when the shooting position is designated on the map; ,
Image display means for displaying the read first omnidirectional image as a second omnidirectional image in an image display area different from the map display area by the computer graphic in the predetermined display range;
Accompanying the instruction of continuous display, the imaging order of all the image auxiliary information having the ID number of the traveling path of the read image auxiliary information is read, and these images are arranged in ascending order or reverse order based on the imaging order. The first omnidirectional image associated with the attached information is sequentially read, and each time it is read, the first omnidirectional image is continuously displayed in the image display area as the third omnidirectional image by the computer graphic within the predetermined display range. Captured image display means;
When the photographing order is the number of either end of the both ends, the continuous display of the captured image display means is stopped, and the third omnidirectional image displayed when the stop is stopped is the fourth The ID code of the node at either end of the traveling path having the ID number of the image ancillary information associated with the fourth omnidirectional image is searched for as the omnidirectional image, and the ID code Means for retrieving the ID numbers of all the traveling paths connected to the related nodes, and reading the direction of the road center of the retrieved traveling paths;
For each direction of the road center of all the travel routes read, the direction of all the travel routes read from the photographing center direction on the fourth all-around image based on the direction and the photographing center direction. The azimuth for each direction of the road center is obtained, and the display position of the travel path selection polygon for selecting the travel path for each of these azimuths is displayed on the fourth all-around image at a predetermined height on the road. And obtaining a fifth omnidirectional image obtained by synthesizing the traveling path selection polygon on the fourth omnidirectional image at these display positions, and obtaining the fifth omnidirectional image within the predetermined display range in the computer. Means for graphic display;
And a means for causing the captured image display means to read image ancillary information having the ID number of the link information associated with the travel path selection polygon when the travel path selection polygon is selected. Image data display system with geographic information. Each time the first omnidirectional image related to the shooting position in the shooting order of the read image ancillary information is read, the second currently displayed at the position on the map corresponding to the shooting position. The image data display system with associated geographic information according to claim 1, further comprising a photographing position display means for displaying an identification symbol for informing a display position of the third or fourth omnidirectional image.   The plurality of first surrounding images and image ancillary information stored in the storage device may be selected according to any one of a specified region, a specified shooting date / time, or a specified traveling route type. 3. The image data display system with associated geographic information according to claim 1, wherein the first all-around image extracted from the all-around image information and the image attached information. The first omnidirectional image is a panoramic image obtained by connecting images with a viewing angle in a preset viewing direction,
The second, third, fourth, or fifth omnidirectional image displayed in the image display area is a second, third, fourth, or fifth image in which images of viewing angles in a preset viewing direction are connected. 5 panoramic images,
When the shooting order of the image ancillary information indicates the number of either end of the traveling path, it is determined that the node is located, reads the node information related to the node, and Means for determining the direction of all travel paths of the link information of the ID number of node information;
Means for displaying an arrow corresponding to the obtained direction at a position on the map corresponding to the read travel path;
The center of the fan-shaped polygon whose center angle is the angle corresponding to the predetermined display range of the second, third, fourth, or fifth panoramic image connecting the images of the viewing angles is the intersection of the arrows. And a means for notifying a visual field direction from a direction indicated by a central axis of the fan-shaped polygon by displaying the map on the map. Image data display system. 5. The geographic data-related image data display system according to claim 1, wherein the travel path selection arrow polygon and the fan-shaped polygon are subjected to a transmission process. The first omnidirectional image stored in the storage device is:
The shooting date, shooting date and time zone are added,
The image attached information stored in the storage device is:
The shooting position, the shooting center direction, the shooting order in the traveling path, the ID number of the traveling path corresponding to the shooting position, the shooting date and the shooting date and time band are associated with each other,
For each predetermined distance interval set in advance, the first omnidirectional image having the shooting position closest to the coordinate point for each distance and the image attached information are the shooting date, shooting date and time, 6. The image data display system with associated geographic information according to claim 1, further comprising: means for extracting an ID of a traveling path as a key and storing the extracted information in the storage unit. The map of the storage device includes features and annotation data, the annotation data being associated with the features by coordinates,
Means for retrieving, from the storage device, image ancillary information having the photographing position closest to the coordinates of the annotation data for each of the annotation data, and storing it in association with the annotation data;
The geographical information according to any one of claims 1 to 6, further comprising means for displaying the annotation data of the image attachment information on the all-around image of the image attachment information when the image attachment information is allocated. Related image data display system. The annotation data is obtained by obtaining an orientation from the shooting position of the image ancillary information related to the displayed second, third, fourth, or fifth omnidirectional image to the annotation data. The image data display system with associated geographic information according to claim 7, wherein the overlapping display is performed at a certain eye level.   The display of the second, third, fourth, or fifth omnidirectional image may be a simple display in which the omnidirectional image related to the assigned image attached information is displayed as it is, or the omnidirectional image may be displayed at the photographing center. 9. The map information association image data according to claim 1, wherein camera parameter designation display is performed to display, as a computer graphic, an image taken in an arbitrary direction and at a predetermined viewing angle. Display system. When a moving body is equipped with an all-around shooting camera and a GPS-type hybrid position sensor and moves along a traveling path, images are taken at regular intervals, and the shooting position obtained based on the information of the position sensor, all An all-around image of the shooting point is extracted as a first all-around image from a plurality of all-around image information obtained together with the shooting center direction, the shooting date, and the shooting time zone of the surrounding shooting camera. In a program for displaying image data with associated geographic information for displaying a surrounding image on a screen with a computer graphic in a predetermined display range, and displaying a map including the range of the shooting point on the screen,
On the computer,
ID code and the traveling path start point of the both ends of the node ID number and該進path of said advancing path on the map, the advancing path leading to the link information and said node and a vector information with coordinates and directions of the end point A route network having node information including an ID number, the first omnidirectional image in the route network, the photographing position, the photographing center direction, and the photographing order in the traveling route for each of the first omnidirectional images; means for storing stores the association only in storage means the image accompanying information, and the map including the ID number of the traveling path corresponding to the photographing position,
Means for reading image attachment information for each of the first omnidirectional images, extracting the shooting positions included in the image attachment information, and displaying them at corresponding positions on the map;
Means for reading, from the storage device, the image attached information having the designated taken position and the first omnidirectional image associated with the image attached information when the taken position is designated on the map;
Image display means for displaying the read first omnidirectional image as a second omnidirectional image in an image display area different from the map display area by the computer graphic in the predetermined display range;
Accompanying the instruction of continuous display, the imaging order of all the image auxiliary information having the ID number of the traveling path of the read image auxiliary information is read, and these images are arranged in ascending order or reverse order based on the imaging order. The first omnidirectional image associated with the attached information is sequentially read, and each time it is read, the first omnidirectional image is continuously displayed in the image display area as the third omnidirectional image by the computer graphic within the predetermined display range. Captured image display means,
When the photographing order is the number of either end of the both ends, the continuous display of the captured image display means is stopped, and the third omnidirectional image displayed when the stop is stopped is the fourth The ID code of the node at either end of the traveling path having the ID number of the image ancillary information associated with the fourth omnidirectional image is searched for as the omnidirectional image, and the ID code Means for retrieving ID numbers of all the traveling paths connected to the related nodes and reading the direction of the road center of the retrieved traveling paths;
For each direction of the road center of all the travel routes read, the direction of all the travel routes read from the photographing center direction on the fourth all-around image based on the direction and the photographing center direction. The azimuth for each direction of the road center is obtained, and the display position of the travel path selection polygon for selecting the travel path for each of these azimuths is displayed on the fourth all-around image at a predetermined height on the road. And obtaining a fifth omnidirectional image obtained by synthesizing the traveling path selection polygon on the fourth omnidirectional image at these display positions, and obtaining the fifth omnidirectional image within the predetermined display range in the computer. Means for graphic display,
Wherein when said traveling path selected polygon is selected, in order to function as a means for reading the image auxiliary information having the ID number of the link information associated with the該進path selection polygon on the captured image display means A program for displaying image data with geographic information. In the computer,
Each time the first omnidirectional image related to the shooting position in the shooting order of the read image ancillary information is read, the second currently displayed at the position on the map corresponding to the shooting position. , third or fourth claim 10 geographic information associated with the image data display program according to which functions as a shooting position display means for displaying the identification symbol for informing the display position of the omnidirectional image. The first omnidirectional image and image attached information stored in the storage device are:
The first omnidirectional image and image ancillary information extracted from the plurality of omnidirectional image information under a cutout condition of any one of a designated area, a designated shooting date / time, a shooting time zone, or a specified traveling route type. 12. The program for displaying image data associated with geographic information according to claim 10 or 11. The first omnidirectional image is a panoramic image obtained by connecting images with a viewing angle in a preset viewing direction,
In the computer,
The second, third, fourth, or fifth omnidirectional image displayed in the image display area is a second, third, fourth, or fifth image in which images of viewing angles in a preset viewing direction are connected. 5 panoramic images,
When the shooting order of the image ancillary information indicates the number of either end of the traveling path, it is determined that the node is located, reads the node information related to the node, and Means for obtaining directions of all traveling paths of the link information of the ID number of node information;
Means for displaying an arrow corresponding to the obtained direction at a position on the map corresponding to the read travel path;
The center of the fan-shaped polygon whose center angle is the angle corresponding to the predetermined display range of the second, third, fourth, or fifth panoramic image connecting the images of the viewing angles is the intersection of the arrows. , geographic information related with according a viewing direction by displaying on the map in any one of claims 10 to 12 for functioning as a means for notifying the direction indicated by the central axis of the fan-shaped polygon Image data display program. In the computer,
It said traveling path selection arrow polygon and the fan-shaped polygons, claim 10 or 13 geographic information associated with the image data display program according to which functions as a means for subjected to transmission processing. The first omnidirectional image stored in the storage device is:
The shooting date, shooting date and time zone are added,
The image attached information stored in the storage device is
The shooting position, the shooting center direction, the shooting order in the traveling path, the ID number of the traveling path corresponding to the shooting position, the shooting date and the shooting date and time band are associated with each other,
In the computer,
For each predetermined distance interval set in advance, the first omnidirectional image having the shooting position closest to the coordinate point for each distance and the image attached information are the shooting date, shooting date and time, geographic information associated with the image data display program according to any one of claims 10 to 14 for function of the ID of the traveling path as means for extracting and storing in the storage means as a key. The map of the storage device includes features and annotation data, the annotation data being associated with the features by coordinates,
In the computer,
Means for retrieving image ancillary information having the photographing position closest to the coordinates of the annotation data for each of the annotation data from the storage device and storing the information in association with the annotation data;
When this image accompanying information has been provision, the omnidirectional image in any one of claims 10 to 15 for functioning as a means for displaying overlapping annotation data of the image accompanying information of the image accompanying information Program for displaying image data with geographic information described in 1. In the computer,
The direction from the shooting position of the image ancillary information related to the displayed second, third, fourth, or fifth omnidirectional image to the annotation data is obtained, and this eye direction and a preset eye height are set. geographic information associated with the image data display program according to claim 16 for functioning as a means for the superposed displayed. In the computer,
The display of the second, third, fourth or fifth omnidirectional image is:
A simple display that directly displays the entire surrounding image related to the allocated image attached information, or
A camera parameter designation display is performed for displaying an image obtained by photographing the entire peripheral image in an arbitrary direction from the photographing center and at a predetermined viewing angle with a computer graphic.
Map information associating the image data display program according to any one of claims 10 to 17, characterized in that.