JP2004070929A - Image processing device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To associate real image data with the position of an actual space with a simple operation in an image processing device. <P>SOLUTION: This image processing device for using a photographed image to construct a virtual space in which walk-through is possible is provided with an image data storing part 10 for storing photographed images photographed by a plurality of imaging devices, a map data storing part 20 for storing map data, and an image/map associating part 30 for associating the map data with the photographed images on the basis of a user's instruction. The image/map associating part 30 has an operation screen for displaying a map image corresponding to the map data and displaying an image corresponding to the photographed images, and in this image display, the images photographed by the plurality of imaging devices are synchronously displayed. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing method, an image processing device, and a storage medium. In particular, the present invention relates to an image processing method, an image processing apparatus, and a storage medium having a user interface suitable for associating image data obtained by photographing a real space with position data on a map.
[0002]
[Prior art]
Attempts have been made to photograph a real space with a photographing device mounted on a moving body and to represent the photographed real space as a virtual space using a computer based on photographed real image data (for example, Endo, Katayama, Tamura, Hirose, Watanabe, Tanikawa: "On Cyber-Electronic Imaging of Urban Space Using a Camera Mounted on a Moving Vehicle" (Research Society, PA-3-4, pp. 276-277, 1997), or Hirose, Watanabe, Tanigawa, Endo, Katayama, Tamura: "Construction of a cyber-visual urban space using a camera mounted on a mobile vehicle (2)-Creation of a wide-area virtual space using real images-" Pp. 67-70, 1997)).
[0003]
As a method of expressing the photographed real space as a virtual space based on the real image data captured by the imaging device mounted on the moving object, a geometric model of the real space is reproduced based on the real image data. However, there is a method of expressing by a conventional CG technique, but there is a limit in terms of accuracy, precision, and realism of the model. On the other hand, Image-Based Rendering (IBR) technology for expressing a virtual space using a real image without performing reproduction using a model has recently attracted attention. The IBR technology is a technology for generating an image viewed from an arbitrary viewpoint based on a plurality of real images. Since the IBR technology is based on a real image, it is possible to represent a realistic virtual space.
[0004]
In order to construct a walkable virtual space using such an IBR technology, it is necessary to generate and present an image according to the position of the user in the virtual space. Therefore, it is necessary to associate each frame of the real image data with a position in the virtual space. However, in this case, since the virtual space is a space generated based on the real space, a position in the virtual space can be translated into a position in the real space. That is, it is necessary to correlate each frame of the real image data with the position data in the real space, and to know at which position in the real space each frame was photographed.
[0005]
As a method of obtaining position data in the real space, it is general to use a positioning system using an artificial satellite represented by GPS (Global Positioning System) which is also used in a car navigation system and the like. . As a method of associating position data obtained from a GPS or the like with actual image data, a method of associating the data with a time code has been proposed (Japanese Patent Laid-Open No. 11-168754). In this method, the time data included in the position data is associated with the time code added to each frame of the actually shot image data, so that each frame of the actually shot image data is associated with the position data.
As a method of obtaining position data in the real space, it is general to use a positioning system using an artificial satellite represented by GPS (Global Positioning System) which is also used in a car navigation system and the like. . As a method of associating position data obtained from a GPS or the like with real image data, a method of associating using time code has been proposed (Patent Document 1). In this method, the time data included in the position data is associated with the time code added to each frame of the actually shot image data, so that each frame of the actually shot image data is associated with the position data.
[0006]
[Problems to be solved by the invention]
However, in the case of GPS, for example, there may be cases where position data cannot be obtained due to geographical conditions or an error included in the position data becomes too large to be ignored. For this reason, there is a need for a method of easily obtaining a position for an inaccurate or inaccurate portion of image data and position data, and associating the position with image data.
[0007]
The present invention has been made in view of the above problems, and has as its object to make it possible to associate actual image data with a position in a real space by a simple operation.
Further, another object of the present invention is that, when the position of the real image data is associated with the real space by using the GPS, it is possible to easily associate a portion that cannot be appropriately associated with the position by a simple operation. It is to be.
[0008]
Further, in associating image data with a position by a general method, since the position is associated with each frame, the data amount becomes enormous. Therefore, another object of the present invention is to provide an image data recording that enables the real image data and the real space to be easily associated with each other with a small data amount.
[0009]
Still another object of the present invention is to enable image reproduction based on the image data recording.
[0010]
Further, in the image reproduction, since the actual image data is sequentially reproduced, in order to reproduce a desired position, the reproduction must be advanced along the road to the desired position. For this reason, it takes a considerable amount of time and effort to obtain reproduction at a desired position. Therefore, another object of the present invention is to make it possible to specify a reproduction start position of a photographed image on a map.
[0011]
[Means for Solving the Problems]
An image processing apparatus according to the present invention for achieving at least one of the above objects has the following configuration. That is,
An image processing device for constructing a walkable virtual space using the captured image,
A captured image storage unit that stores captured images captured by a plurality of imaging devices,
A map data storage unit for storing map data,
An image-map associating unit that associates the map data with the captured image based on a user's instruction,
The image / map associating unit includes:
An operation screen for performing a map image display according to the map data and an image display according to the photographed image, wherein the image display synchronously displays photographed images photographed by the plurality of imaging devices. .
[0012]
Further, an image processing method according to the present invention for achieving at least one of the above objects,
An image processing method for constructing a walkable virtual space using a captured image,
A map image is obtained from a map data storage unit for storing map data, and a map image to be displayed is displayed. A providing step of providing an operation screen for displaying
An image / map associating step of associating the map data with the photographed image based on a user instruction on the operation screen,
The display of the captured image on the operation screen displays the captured images captured by the plurality of imaging devices in synchronization.
[0013]
According to the present invention, there is provided a recording medium storing a control program for causing a computer to execute the image processing method.
[0014]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.
[0015]
<First embodiment>
In the present embodiment, for example, a visual user interface is used to associate real image data obtained by photographing with a photographing device mounted on a moving object such as a car with map data as a position in a real space. An image recording device that can be used will be described.
[0016]
FIG. 1 is a diagram illustrating a functional configuration of the image recording apparatus according to the present embodiment. The present image recording system includes an image data collection system 90 and an image recording device 1. The image recording device 1 includes an image data storage unit 10, a map data storage unit 20, an image / map association unit 30, an operation unit 40, and an association data storage unit 50.
[0017]
The image data storage unit 10 stores video data (hereinafter, referred to as image data) as actual image data obtained by an image data collection system 90 described later. The map data storage unit 20 stores map data including image information of a map and coordinate information for representing each position on the map image with coordinates related to longitude and latitude. The map data storage unit 20 stores at least map data corresponding to the real space position of the image data captured by the image data collection system 90 and stored in the image data storage unit 10. The map data is stored in a hard disk (not shown), a RAM, or another external storage device.
[0018]
The image / map associating unit 30 reads out the image data stored in the image data storage unit 10 and the map data stored in the map data storage unit 20 and displays them on the display device of the operation unit 40, The image data and the map data performed by the user using the operation unit 40 are associated with each other. The image data and map data associated in this manner are stored in the association data storage unit 50 as association data. The operation unit 40 includes a display device, a mouse, a keyboard, and the like.
[0019]
FIG. 2 is a diagram illustrating a configuration example of an image data collection system for collecting image data stored in the image data storage unit 10. As shown in FIG. 2, the image data collection system 90 is divided into three parts: a photographing unit 60, a recording unit 70, and an A / D conversion unit 80. Hereinafter, each unit will be described in detail with reference to FIGS.
[0020]
FIG. 3 is a block diagram illustrating the configuration of the imaging unit 60 in detail. In the present embodiment, the photographing unit 60 is used by being mounted on a moving body such as a vehicle. The imaging unit 60 includes n (n ≧ 1) video cameras (60-1 to 60-n) and a synchronization signal generation unit 62. An external synchronization signal from the synchronization signal generator 62 can be input to each of the video cameras 60-1 to 60-n. Using the external synchronization signal output from the synchronization signal generation unit 62, the shooting timings of the n video cameras 60-1 to 60-n are matched.
[0021]
FIG. 4 is a block diagram illustrating the configuration of the recording unit 70 in detail. The recording unit 70 includes a time code generation unit 72 and recording units (video cassette recorder VCRs in this example) 70-1 to 70-n corresponding to the video cameras 60-1 to 60-n. Image outputs from the video cameras 60-1 to 60-n of the photographing unit 60 are input to the recording units 70-1 to 70-n, respectively. Further, the time code generation section 72 supplies a time code representing the shooting time to each of the VCRs 70-1 to 70-n. The VCRs 70-1 to 70-n take in the image input from the corresponding video camera and the time code from the time code generator 72, and store them as image data with the time code.
[0022]
FIG. 5A is a block diagram showing the configuration of the A / D converter 80 in detail. The A / D conversion unit 80 has a personal computer (hereinafter, PC) 84 and video capture boards (hereinafter, capture boards) 82 corresponding to each of the VCRs 70-1 to 70-n. The number of the capture boards is not always necessary for the number of video cameras, and the number of the capture boards that can be mounted on the PC may be shared. The A / D converter 80 converts analog image data supplied from each VCR into digital image data (for example, AVI format) and stores the digital image data in the hard disk 10 connected to the PC 84 or another storage medium. When the image data is captured into the PC 84 through the capture board 82, the capture start part and the end part are determined using the recorded time code, and are acquired via the n VCRs and the capture board. The digital image data is constituted by capture data of the same length, all of which are photographed at the same time. The data format that can be adopted is, for example, the AVI format.
[0023]
FIG. 5B is a diagram illustrating an example of a storage mode of image data in the image data storage unit 10. As shown in FIG. 5B, a frame number and frame data are stored in association with each other for each of the cameras (60-1 to 60-n). Note that the same frame number is assigned to frame data at the same time. In the present embodiment, the image data is stored as image data captured by a normal capturing unit. However, a plurality of frames having the same frame number may be combined and stored as a panoramic image. .
[0024]
FIG. 6 is a block diagram illustrating a hardware configuration of the image recording apparatus according to the present embodiment. The hardware configuration shown in FIG. 6 is equivalent to the configuration of a normal personal computer. In FIG. 6, the disk 105 stores digital image data obtained by the image data collection system 90 described above. The disk 105 constitutes the image data storage unit 10, the map data storage unit 20, and the association data storage unit 50 described above. The CPU 101 executes a program stored in the disk 105 or the ROM 106 or an external storage device (not shown), and associates image data and map data, which will be described later, with reference to the flowchart of FIG. That is, the image / map associating unit 30 is realized.
[0025]
When the CPU 101 issues various display instructions to the display controller 102, desired display is performed on the display 104 by the display controller 102 and the frame buffer 103. In the figure, the CRTC is shown as the display controller 102 and the CRT is shown as the display 104. However, the display is not limited to a cathode ray tube, but may be a liquid crystal display or the like. The mouse 108 and the keyboard 109 are used to input user operations, and input to a GUI described later. Data output by the image / map associating unit is recorded on the disk 105 or an external storage device (not shown). The CRT 104, the mouse 108, and the keyboard 109 constitute the operation unit 40 described above.
[0026]
In the image recording apparatus according to the present embodiment having the above configuration, the correspondence between the image data of the real space stored in the image data storage unit 10 and the map information stored in the map data storage unit 20 is determined according to the operation of the user. The operation of the image / map associating unit 30 to be performed will be described.
[0027]
FIG. 7 is a flowchart showing a processing procedure by the image / map associating unit 30. First, in step S1, the image / map associating unit 30 displays the map data on the CRT 104 and provides a user interface for specifying a section point and a road on the map. Then, in step S2, using the provided user interface, a line segment (hereinafter, referred to as a road) connecting a point (hereinafter, referred to as a "divided point") to the displayed map data and dividing the photographing route into the photographed route. ) Is set for the required number of breakpoints and roads. At this time, an ID capable of uniquely specifying each of the section point and the road is assigned. Note that the line segment connecting the segment points is a straight line or a curve between two segment points, that is, a section.
[0028]
FIG. 8 is a diagram showing an example of a screen display in a user interface for setting a section point and a road. In step S1, the map data displayed on the screen is displayed, and a user interface (GUI) for setting later-described section points and roads is provided on the map 801. In step S2, on the map displayed in step S1, along the photographing route, a dividing point is set mainly at a branch point of the road, and a road whose both ends are sandwiched by the dividing points is set by a user operation. . Here, the section ID and the road ID may be automatically assigned, or may be set and assigned by the user. Also, a name (XX street, XX intersection, etc.) may be given to each road or section point, and search may be made by this name. In addition, the dividing point may be set not only at the branch point of the road but also at an arbitrary point on the road.
[0029]
The setting of the above-mentioned section points and roads is performed by, for example, the following operation. When the mouse is clicked on the map displayed on the screen, a breakpoint is set at the clicked position. Of course, the dividing points may be given as coordinate values on the map from the keyboard. The road can be set by dragging the mouse pointer from one section point to another. Of course, the way may be set by connecting the two points by giving the IDs of the two division points from the keyboard.
[0030]
Various methods are conceivable for storing the set points and roads in this way, but in the present embodiment, it is assumed that the points and roads are stored in a form as shown in FIGS. 9A and 9B. FIG. 9A is a diagram for explaining an example of registration of a section point. FIG. 9B is a diagram illustrating an example of registering a road. As shown in FIG. 9A, in the section list 900, a section point ID 901 for specifying each section point, map coordinate data 902 indicating a position on the map, a name 903 assigned to the section point, Is registered. The name data may or may not be given. Also, as shown in FIG. 9B, in the road list 910, for each road, a road ID 911, a start demarcation point 912 indicating end points of both ends of the road, an end demarcation point 913, and a road name 914 are registered. You. The name data may or may not be given. The start frame 915, end frame 916, and frame position information 917 will be described later.
[0031]
Returning to FIG. 7, next, the image data collected by the image data collection system 90 and stored in the image data storage unit 10 as shown in FIG. 5B is associated with the segment points and roads set by the above processing. For this purpose, a screen (GUI) for the operation is displayed on the CRT 104 in step S3.
[0032]
FIG. 10 is a diagram illustrating a display example of a GUI screen displayed in step S3 for associating image data and map data according to the present embodiment. On this GUI, a map data display unit 301, an image data display unit 302, a road information display unit 303, and an image data reproduction operation unit 304 are displayed. In the map data display unit 301, along with the map data stored in the map data storage unit 20, the dividing points and the roads (registered in the dividing point list and the road list) set in steps S 1 and S 2 described above are displayed. For example, each is displayed as a circle and a thick line.
[0033]
The image data display unit 302 reproduces the image data stored in the image data storage unit 10. In the image data display unit 302, images captured by a plurality of (n in this example) cameras are displayed in synchronization. The road information display unit 303 displays the ID of the road, the name given to the road, the IDs of the breakpoints at both ends of the road, and the frame numbers (start (Start) and end (End)) of the image data corresponding to the road. I do. However, the frame number is not displayed until it is associated with a road (in this case, it may be displayed as undecided). The image data reproduction operation unit 304 includes operation buttons for reproduction, stop, fast forward, rewind, frame advance, and the like.
[0034]
In the following steps S4 to S6, the frames of the image data are associated with the roads using the GUI shown in FIG. First, in step S4, the user selects a desired road from the roads set in the previous steps S1 and S2 and displayed on the map data display unit 301 with a mouse. The selection of a road is performed by performing a double-click operation on or near a desired road displayed on the map data display unit 301, or by opening a dialog and inputting from a keyboard. Here, information on the selected road is displayed on the road information display unit 303.
[0035]
Then, in step S5, a search is made for video frames corresponding to the section points at both ends of the selected road. That is, the user causes the image data display unit 302 to reproduce an image using the reproduction, stop, fast forward, rewind, frame forward buttons, and the like of the image data reproduction operation unit 304, and captures images at the section points at both ends of the selected road. A frame closest to the selected image is searched from the image data. Then, two frame numbers corresponding to the section points at both ends of the road are determined. In step S6, the two frames selected as described above and the image data between the two frames are associated with the selected road and registered as the association data. In this way, the frame numbers determined for the start section point and the end section point at both ends of the road are registered in the start frame 915 and the end frame 916 of the road list shown in FIG. 9B, respectively.
[0036]
FIG. 11 is a conceptual diagram for finding a position where each frame is photographed by associating a map with image data. As described above, the image 1101 displayed by the frame advance operation or the like is selected and associated with the segment point t0, and the image 1102 is selected and associated with the segment point t1. Then, the image frame group 1103 between the two frames associated with the segment points is associated with a road having both ends of the two segment points t0 and t1 on the map. That is, information for associating a road with a frame group is obtained. Further, in the present embodiment, the position in the real space where each image frame is photographed is easily obtained by equally dividing the division points by the number of frames of the image frame group 1103 allocated between the division points. The obtained position information of each frame is registered in the frame position information 917 of the road list shown in FIG. 9B. However, in the present embodiment, the position information of each frame can be obtained by interpolation if the map coordinates of the dividing points at both ends are known, and thus can be omitted.
[0037]
That is, the image frame group 1103 may add (1) position information obtained by the above-described interpolation to each image frame existing between video frames associated with the segmentation points, and store (2). Only the video frames corresponding to the start and end breakpoints of each road are registered, and at the time of image playback, the image frames between the breakpoints of the road are determined according to the number of image frames between the breakpoints and the playback time or distance. You may make it reproduce | regenerate evenly. FIG. 5B shows the storage form according to the method (1), in which the position of the start partition point, the position of the end partition point, or the virtual position is assigned to each frame in the frame position information 917. For example, (u ₁ , V ₁ ) And (u) ₁₀₀ , V ₁₀₀ ) Correspond to the start section point and the end section point of the road R1, and are provided with coordinate values in which frames therebetween are evenly arranged on the road R1. As described above, when the method (2) is adopted, the frame position information 917 can be omitted.
[0038]
When the association of the image data with the selected road is completed as described above, in step S7, if the association is to be performed with another road, the process returns to step S4. If no further association is to be performed, this processing is terminated by a user's termination operation.
[0039]
In this embodiment, images taken from n video cameras are displayed on the image data display unit 302 in a synchronized manner. By doing so, for example, images from cameras facing in the front, rear, left, and right directions can be simultaneously viewed, and it becomes easier to more accurately specify the position of an intersection or the like. However, the display mode in the image data display unit 302 is not limited to this. For example, an image from one of the n video cameras may be displayed. Since only one image needs to be displayed, the load on the CPU is reduced. Further, it may be possible to specify how many images from the video cameras are to be displayed at the same time. Further, when images from one or a desired number of video cameras are simultaneously displayed, it may be possible to select which camera image should be displayed on the image data display unit 302. When images from a desired number of video cameras are simultaneously displayed, the number may be designated.
[0040]
Further, when a frame advance operation is performed, a plurality of continuous frames are displayed on the image data display unit 302 for each image from each camera as shown in the image data display unit 302 in FIG. You may. In the present embodiment, when a desired frame is selected with a mouse, a screen pops up asking which one of the dividing points at the both ends of the road is to be associated (start point or end point). Are associated with the image frame.
[0041]
The data stored in the association data storage unit 50 from the image / map association unit 30 as described above is data in which the image data and the position data on the map are linked in some form. That is, when a desired position on the road on the map is designated, the frame of the corresponding image data is specified and the data can be displayed.
[0042]
FIG. 12 is a diagram showing an example of a recording format of link data stored in the association data storage unit 50. This data format has a tree structure, with map data at the top. The map data 1201 is data stored in the map data storage unit. Road data 1202, section point data 1203, and image data 1204 are linked to the map data 1201. The road data 1202 includes the road list 910 shown in FIG. 9B, and indicates the name of each road, the IDs of the partition points at both ends, the frame numbers of the image data corresponding to the partition points at both ends, and the position of each frame of the image data. Frame position information is recorded. Also, the breakpoint data 1203 includes the breakpoint list 900 shown in FIG. 9A, and records the name of the breakpoint and the ID of the road connected to the breakpoint. The image data 1204 is image data held in the image data storage unit 10 or a panoramic image obtained by synthesizing video frames having the same frame number, and is stored in association with the frame number.
[0043]
That is, the images captured by the video cameras 60-1 to 60-n of the image capturing unit 60 are recorded with a frame number assigned to each video frame and recorded on a map as shown in FIG. When the segment points t0 and t1 are selected and the video frame corresponding to the segment point is selected and associated, the segment points t0 and t1 can be linked to the video frame numbers associated therewith. Will be stored.
[0044]
In FIG. 12, the frame numbers corresponding to the segment points on the map are linked and stored. However, the segment points are divided by the number of frames corresponding to the length of the segment points, and virtually, A frame position may be set, assigned to each video frame, and stored. As described above, when only the video frame corresponding to the segment point is stored, the intermediate video frame is used to reproduce the video frame between the segment points, and the distance between the segment points and / or the reproduction time is used. Are sequentially reproduced in accordance with. The detailed contents of the reproduction system will be described in a third embodiment.
[0045]
As described above, according to the first embodiment, it is possible to easily associate collected real image data with a position on a map.
[0046]
Note that a preview button 305 is provided in the GUI shown in FIG. When a desired road is selected on the map display section 301 and this preview button is pressed, a window pops up, and video frames assigned to a designated section of the selected road are continuously played. By providing such a preview button, the user can check the allocation state at any time while performing the allocation operation, so that the operability is improved.
[0047]
<Second embodiment>
In the first embodiment described above, all of the collected real image data is associated with a position on a map using the GUI shown in FIG. In the second embodiment, by associating the image data with the map information by effectively using the position information obtained by the GPS, the association work can be performed more efficiently. That is, in the second embodiment, when associating the photographed image data with the position data in the real space, the position data is not obtained by the GPS, or the position data obtained has a large error. The position data is associated. That is, the association of the position data using the GUI described in the first embodiment is used to complement the position data obtained by the GPS.
[0048]
FIG. 13 is a block diagram illustrating a configuration of an image collection system according to the second embodiment. In FIG. 13, an image from a video camera 60-1 is recorded on a VCR 70-1. At this time, as described in the first embodiment, the time code from the time code generator 72 is supplied to the VCR 70-1, and the image data with the time code is recorded on the VCR 70-1. The output of the VCR 70-1 is converted into digital image data (for example, AVI format) by the capture board 82-1 and stored in the hard disk 10 or other storage medium connected to the PC 84. Although FIG. 13 shows one set of the video camera, the VCR, and the capture board, it goes without saying that a plurality of sets may be used as in the first embodiment.
[0049]
In FIG. 13, the GPS unit 1301 further supplies the latitude and longitude data and the time code from the time code generation unit 72 to the PC 84. The PC 84 stores information in which the latitude and longitude data are associated with the time code in the hard disk 10. FIG. 14 is a diagram illustrating an example of a storage state of video data and position data (GPS measurement result data) in the image data storage unit 10 according to the second embodiment. Thus, as shown in FIG. 14, the video frame to which the time code is added and the position data to which the time code is added are stored. Therefore, the video frame is associated with the position data via the time code.
[0050]
The calculation of the three-dimensional position in the GPS is known, and a detailed description is omitted here. As described above, the latitude and longitude data P (θ, φ) sequentially obtained by the GPS are stored in association with the time code from the time code generation unit 72 while the image is captured by the n video cameras. Go. Then, based on the time code, each frame of the image is associated with the latitude and longitude data.
[0051]
FIG. 15 is a diagram illustrating a display state of the map data display unit 301 according to the second embodiment. FIG. 16 is a flowchart illustrating a processing procedure according to the second embodiment.
[0052]
First, in step S100, a screen for setting a section point and a road is displayed. This is the same as step S1 in FIG. 7 described above. Next, in step S101, the GPS data is plotted on the map display. In FIG. 15, a plot based on GPS data is indicated by a dotted line.
[0053]
Then, in step S102, as described in step S2 of FIG. 7, a dividing point and a road are set on the map displayed in step S1. At this point, a map, a plot of GPS data, a section point, and a road are displayed on the map display unit 301 as shown in FIG. Thereafter, when the start of the image data association process is instructed, the process proceeds from step S103 to step S104. The start of the image data association process can be determined by providing an appropriate button on the user interface shown in FIG. 10 and performing a click operation.
[0054]
When the image data associating process is started, first, in step S104, the user is caused to specify a road to which image data is to be associated. When a road is designated, in step S105, the dividing points at both ends of the designated road are acquired, and a frame corresponding to these dividing points is searched. In the frame search, the coordinates of the segment on the map are compared with the coordinates of the GPS data of each frame, and the frame closest to the position of the segment is obtained. If the distance is within a predetermined range, the frame is set to a frame corresponding to the section point. In step S105, the acquisition of the frames corresponding to the section points at both ends of the designated road is tried in this way. If the frames corresponding to both the section points can be acquired, the process proceeds from step S106 to step S107. The start frame and the end frame shown in FIG. 9B are registered for the designated road.
[0055]
Further, in step S108, it is checked whether or not there is any one of the positions of the frame group sandwiched between the two frames acquired in step S105 that largely deviates from the designated road or that lacks GPS data. If the positions of the investigated frame group fall within a predetermined distance with respect to the designated road, the process proceeds to step S109, and the frame position information 917 in FIG. 9B is obtained according to the frame group and the position information based on the GPS data. Is registered.
[0056]
On the other hand, if no frame corresponding to the section point is found in steps S105 and S106, the process proceeds to step S110, and manual frame allocation is performed. This processing is similar to the processing described in the first embodiment. That is, the video frame is reproduced and the end and start frames corresponding to both ends of the specified road are specified. Then, in step S111, a frame group between the specified start and end frames is assigned to a road as described with reference to FIG. If there is a GPS data defect in the frame group between the start and end frames, the process proceeds from step S108 to step S111, and the frame group between the start and end frames determined in step S107 is shown in FIG. The process of allocating to the described road is performed.
[0057]
Thereafter, in step S112, it is determined whether or not termination has been instructed, and if not, the process returns to step S104 to repeat the above processing. On the other hand, if the end is instructed, the process ends.
[0058]
Each position (x, y) on the map is obtained by a function f (θ, φ) (the function f differs depending on a projection method when creating a map). That is,
Position (x, y) = f (θ, φ)
, The GPS data is converted into position coordinates on the map, and plotted on the displayed map. The plotted points may be displayed by connecting them with a line. FIG. 15 shows an example in which plots are connected and displayed by dotted lines.
[0059]
The user can recognize from the display contents of the map data display section 301 a portion where the measurement error by GPS is large or a portion where GPS data is missing. In the example of FIG. 15, a GPS error is large at a point 1501, and GPS data is missing at a point 1502.
[0060]
In FIG. 15, if a road 1513 having segment points 1511 and 1512 at both ends is selected, and both segment points and video frames are associated with each other as described above, the portion 1501 having a large measurement error due to GPS will be more correctly determined. Video frames can be associated with position data. In addition, if a road 1523 having segment points 1521 and 1522 at both ends is selected, and the segment points are associated with the video frames as described above, a video frame group including 1502 points where GPS measurement is impossible is performed. Can be correctly associated with the position data.
[0061]
In the above embodiment, when the error of the GPS data with respect to the road is within the allowable range, the registration in the frame position information 917 is performed using the GPS data as the position data as it is, but the present invention is not limited to this. For example, if the start and end frames are determined based on the GPS data, they are associated with the segment points, and the frame group sandwiched between these frames is used without using the GPS data as described in FIG. May be applied to make correspondence with the road.
[0062]
As described above, according to the second embodiment, even if the position data cannot be obtained by the GPS or the error included in the position data is not negligibly large, the position data is simply obtained and the image data is obtained. And position data can be associated with each other. In addition, since the video frames collected by the image data collection system can be associated with the positions on the map simply by compensating for the incomplete portion by the GPS, it is possible to more easily associate the video frames with the positions. Can be.
[0063]
The missing GPS data may be caused by a case where the GPS data cannot be recorded at a moving image rate (30 frames / sec), in addition to the above-described measurement error. That is, as shown in FIG. 14, it is good if measurement result data by GPS is obtained for each frame, but otherwise, position information of frames between frames in which GPS is recorded is obtained by interpolation. Is also good.
[0064]
In this case, for example, frame i (position (u _i , V _i )) And frame i + n (position (u _{i + n} , V _{i + n} ), The position ((u) of each frame x (x = i + 1,..., I + n−1) between the frame i and the frame i + n is recorded. _x , V _x ) Is, for example,
u _x = U _i + (Xi) (u _{i + n} -U _i ) / N
v _x = V _i + (Xi) (v _{i + n} -V _i ) / N
Can be obtained by
[0065]
In the above example, each position of the frame x is obtained by the linear interpolation. However, the position is not limited to the linear interpolation.
[0066]
As described above, when GPS data is sparsely sampled from an image frame, position information can be obtained by interpolation for an image frame for which position information has not been obtained.
[0067]
<Third embodiment>
Next, an image reproduction system using a database in which a map and an image are linked as described above will be described.
[0068]
FIG. 17 is a block diagram illustrating a functional configuration of an image reproducing system. Note that the device configuration of the reproduction system is the same as that of the first embodiment (FIG. 6), and the CPU 101 executes the control program stored in the ROM 106 or loaded into the RAM 107, as shown in FIG. Each function will be realized.
[0069]
In FIG. 17, the association data storage unit 50 holds association data including the above-described road list 910 (FIG. 9B). In the present embodiment, it is assumed that the frames stored in the association data storage unit 50 are panoramic images obtained by combining a plurality of images having the same frame number obtained from a plurality of cameras. Assume that the association data has a form in which the start and end frames of each road are registered, and the frame position information 917 is omitted.
[0070]
Reference numeral 1701 denotes a panorama image frame group acquisition unit that acquires a frame group corresponding to a “road” to be reproduced from the association data storage unit 50, and sequentially outputs frames for display processing in accordance with the reproduction direction. Note that the viewpoint position acquisition unit 1704 is notified of the number of the output frame in the “road”. Reference numeral 1702 denotes a line-of-sight direction image extraction unit that extracts a partial image corresponding to a specified line-of-sight direction from the panorama image acquired from the panorama image frame acquisition unit 1701. A display unit 1703 displays the partial image extracted by the gaze direction image extraction unit 1702 on the CRT 104, and displays a map based on the map data stored in the map data storage unit 20.
[0071]
Reference numeral 1704 denotes a viewpoint position acquisition unit, in which frame of the frame group acquired by the panorama image frame group acquisition unit 1701 the frame whose image is to be extracted by the line-of-sight image extraction unit 1702 (Notified by the panorama image frame group acquisition unit 1701), the current position in the “road” is calculated. The information on the current position is sent to the display unit 1703 and used to display the current position on the map display. Reference numeral 1705 denotes a position / line-of-sight direction instructing unit that indicates the current position and line-of-sight direction of the user.
[0072]
FIG. 18 is a diagram illustrating a display example according to the third embodiment. As shown in FIG. 18, a map display section 301 as a map window and an image data display section 302 as a window for displaying a captured image are displayed on the screen, and a joystick (not shown) and a mouse are used to perform a reproduction operation. Do. The image data display unit 302 displays an image acquired from the panoramic image frame acquisition unit 1701 and extracted by the gaze direction image extraction unit 1702. The gaze direction is detected by the position / gaze direction instruction unit 1705 based on the inclination angle of the joystick, and is notified to the gaze direction image extraction unit 1702.
[0073]
In the map window 301, map information acquired from the map data storage unit is displayed. On the map display section 301, a position mark 320 indicating the position displayed on the image data display section 302 and the line of sight thereof is displayed. Further, by operating the scroll bars 309 and 310 provided on the right and lower sides of the map display unit 301, the map to be displayed can be corrected to an optimum position.
[0074]
In the above configuration, the data shown in FIG. 12 is read, and the image frame group of the road where the viewpoint position exists is sequentially reproduced. For example, if the current viewpoint position is a position corresponding to the j-th frame of the road i, the j-th frame of the road i will be displayed on the image data display unit 302. At the same time, the position of the j-th frame on the map is obtained by interpolation from the coordinates of the two segment points sandwiching the road i on the map, and the position mark 320 is displayed (in the case where there is the frame position information 917 shown in FIG. 9B). Is used to determine the position of the j-th frame on the map). That is, in the reproduction system, an image is displayed based on the frame held in the association data storage unit 50, the map data is also displayed on the screen at the same time, and the current position (the position of the j-th frame) is indicated by an arrow on the map. Is displayed.
[0075]
When the road i is proceeding, a reproduced image is displayed using a frame group corresponding to the road i, and a position is obtained for each frame to be displayed on a map. After traveling on the road i for a while, a frame corresponding to the section point is reached. When the vehicle arrives at the section point, the road to go next is determined based on the direction of the line of sight at that time and the direction of the road connected to the section point.
[0076]
FIG. 19 is a diagram illustrating the determination of the traveling direction. For example, if the vehicle arrives at the section point in the direction of the line of sight 1, the road in the direction [1] is selected. As a result, a frame group corresponding to the road [1] is selected by the panorama image frame acquisition unit 1701, and image reproduction and position display on the map are performed.
[0077]
By the way, when moving in the virtual city space by the real image sequence as described above, the reproduction of the image is started from a predetermined initial position, and in order to move to the desired position, the road on which the frame group on the map exists exists. They have to move along, which wastes time and effort. Therefore, in the present embodiment, when the user specifies a desired position on the map, reproduction can be started from the image closest to the specified position, and the operability is improved.
[0078]
For example, when the user moves the cursor 321 on the map display unit 301 with a mouse or the like and designates a desired position, a click position on the map is calculated, a road closest to the click position is detected, and a click on the road is further performed. Playback starts from the frame having the frame position closest to the position.
[0079]
FIG. 20 is a flowchart illustrating a control procedure for realizing the above-described user interface. In FIG. 20, when the road to be displayed is determined in step S1901, in step S1902, a frame group corresponding to the determined road is acquired. Then, in step S1903, the acquired images are sequentially displayed on the image data display unit 302. It is assumed that the line of sight is given by the direction of the joystick lever. At this time, in step S1904, the position of the frame being displayed is calculated, and an arrow indicating the current playback position is displayed on the map display unit 301 based on the calculation result.
[0080]
Then, when the reproduced image reaches the frame of the section point (that is, the start frame or the end frame of the [road]), the road to go next is determined by the method described above with reference to FIG. 19 (step S1901), Image reproduction is performed by the processing after step S1902.
[0081]
If a desired position on the map display unit 301 is clicked with the mouse during the image reproduction, the process proceeds from step S1906 to step S1907. In step S1907, the road closest to the clicked position on the map is searched, and the frame closest to the clicked position on that road is selected. After that, in step S1908, the reproduction start frame determined in step S1907 is statically displayed, and a standby state is set. When the progress start is instructed by the joystick, the process returns to step S1903, and the image reproduction to the image data display unit 302 is started using the frame group acquired in step S1907.
[0082]
Next, the process of selecting a road and a frame close to the click position in step S1907 will be described with reference to the flowchart in FIG.
[0083]
When the event of clicking the map occurs in step S2001, the process proceeds to step S2002, and the coordinate position (x, y) of the clicked map is acquired. Then, the process proceeds to step S2003, where “roads” are sequentially selected from the road list stored in the association data storage unit, and the distance between the “road” selected in step S2004 and the clicked one is calculated. The processing of steps S2003 and S2004 is performed for all the roads registered in the road list (step S2005).
[0084]
When the distance from all the roads to the clicked position is calculated, the process proceeds to step S2006, and the road with the shortest distance is selected as “the road closest to the clicked position on the map”.
[0085]
Further, in step S2007, a vertical line is lowered from the clicked point on the “road” selected in step S2006, an intersection between the “road” and the vertical line is obtained, and an image frame corresponding to this intersection is obtained. For example, a frame group associated with the selected “road” is obtained, and the “road” is equally divided by the number of frames in the frame group. Then, a division position closest to the intersection is determined, and a corresponding frame is acquired.
[0086]
In step S2008, the ID of the “road” and the image frame number obtained as described above are returned as a result.
[0087]
Note that, in the above embodiment, a case where a desired position is designated during image reproduction is shown, but an initial position of image reproduction may be set as desired on the map display unit 301.
[0088]
In the third embodiment, the closest road is selected by calculating the distance from the click position. However, the correspondence between the click position and the road is tabulated in advance (the road is unique for the click position on the map). May be stored as information associated with each other so as to be able to determine the information. The processing in step S1907 in this case will be described with reference to the flowchart in FIG.
[0089]
In FIG. 22, first, when an event of clicking the map occurs in step S2101, the process proceeds to step S2102, and the coordinate position (x, y) of the clicked map is acquired. Next, the process advances to step S2103 to acquire a road associated with (x, y) from the table. Thereafter, the process proceeds to step S2104, where a perpendicular is dropped from the clicked point on the “road” selected in step S2103, an intersection between the “road” and the vertical is obtained, and an image frame corresponding to this intersection is obtained. For example, a frame group associated with the selected “road” is obtained, and the “road” is equally divided by the number of frames in the frame group. Then, a division position closest to the intersection is determined, and a corresponding frame is acquired. In step S2105, the ID of the “road” and the image frame number obtained as described above are returned as a result.
[0090]
As described above, according to the processing shown in the flowcharts of FIGS. 21 and 22, since a frame is specified after selecting a road close to the clicked position, there is no need to provide position information for each frame, and management is easy. It is. Further, in the case of the processing in FIG. 21, since the distance to the “road” is calculated, it is not necessary to calculate the distance for each frame, and the amount of calculation can be reduced. Further, according to the processing of FIG. 22, high-speed processing is possible because distance calculation is not required.
[0091]
When a plurality of frame sequences corresponding to a plurality of regions are included, a wide-area map including the plurality of regions is displayed, and a frame sequence of the region closest to the click position on the wide-area map can be selected. You may. In this case, first, a map of the area corresponding to the selected frame sequence is displayed, and thereafter, the frame corresponding to the click position is specified by the above-described procedure. If the range of the wide area map is too wide and a plurality of areas are candidates for the click position, a wide area map including the candidate areas (representing a smaller area than the immediately preceding wide area map) May be displayed to narrow down.
[0092]
<Fourth embodiment>
In the fourth embodiment, by specifying an arbitrary position on a map, an image corresponding to the position is displayed, or by specifying a plurality of section points on a map, an image corresponding to a path formed therebetween is displayed. A function for reproducing a plurality of images by moving image or frame by frame will be described. By doing so, a landscape image at an arbitrary position on the map can be reproduced. It is also possible to reproduce images in different directions (images of each camera) at the same point. The specification of the frame corresponding to the specified position on the map is as described above.
[0093]
FIG. 23 shows a user interface for realizing this function. Although some parts are basically similar to the user interface of FIG. 10, they can be used by appropriately switching according to the function to be executed.
[0094]
A map window 301 and an image data display unit 302 displayed on the screen are formed, and each operation is performed by operation buttons of the image data reproduction control unit 304.
[0095]
First, when search button 307 is clicked with a mouse or the like, a space 306 for inputting a place name is displayed, and an arbitrary place name can be input. Subsequently, by clicking an execute button 308, a map of a region corresponding to the input place name is displayed on the map display unit 301. At this time, any still image may be displayed on the image data display unit as an initial image. In this case, the image to be displayed may be an image corresponding to the input area, an image at the closest position, or the like. Further, by operating the scroll bars 309 and 310 provided on the right and lower sides of the map display unit 301, the map to be displayed can be corrected to an optimum position.
[0096]
Next, by clicking on an arbitrary position on the displayed map with a mouse or the like, a breakpoint is displayed at the designated position. In the same figure, the display is indicated by ○.
[0097]
When the segment point is specified and the reproduction button of the reproduction control unit 304 is clicked, an image linked to the segment point is displayed on the image data display unit.
[0098]
If a plurality of segment points are specified in advance, the path between the segment points is displayed (thick line in the figure), and the path is clicked with a mouse or the like to select it (the selected path changes color, etc.). , Display is changed), and when a play button is clicked, a plurality of images linked on the path are successively and sequentially reproduced, and a moving image can be reproduced. In addition, the direction in which the moving image is reproduced may be selected using the forward reproduction button and the reverse reproduction button.
[0099]
Further, if the frame advance button and the frame return button are used, the image on the selected path can be reproduced while being confirmed by the frame advance. During playback, the playback operation can be paused by the pause button.
[0100]
In addition, since the reproduced images to be displayed exist for each of the plurality of cameras, the direction of the reproduced image at each camera, that is, at each point, can be selected by operating the direction selection button 311. In addition, at the time of reproduction, the division point can be set not only at the branch point but also in the middle of the path.
[0101]
With the above system, a map of an arbitrary area can be displayed, and the scenery at an arbitrary position on the map can be confirmed with an image. This system can be used not only for simulation of changes in the scenery according to a map and a moving path, but also for car navigation and the like as a function for previously knowing the scenery, landmarks, and the like of a traveling route. Applicable.
[0102]
An object of the present invention is to provide a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and a computer (or CPU or MPU) of the system or apparatus to store the storage medium. It is needless to say that the present invention can also be achieved by reading and executing the program code stored in the program.
[0103]
In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.
[0104]
As a storage medium for supplying the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, and the like can be used.
[0105]
When the computer executes the readout program code, not only the functions of the above-described embodiments are realized, but also an OS (Operating System) running on the computer based on the instruction of the program code. It goes without saying that a part or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing.
[0106]
Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that a CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.
[0107]
【The invention's effect】
As described above, according to the present invention, the operability of the operation relating to the association between the real image data and the position in the real space is improved.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a functional configuration of an image recording apparatus according to an embodiment.
FIG. 2 is a diagram illustrating a configuration example of an image data collection system for collecting image data stored in an image data storage unit 10;
FIG. 3 is a block diagram illustrating a configuration of a photographing unit 60 in detail.
FIG. 4 is a block diagram showing a configuration of a recording unit 70 in detail.
FIG. 5A is a block diagram showing a configuration of an A / D converter 80 in detail.
FIG. 5B is a diagram illustrating image data stored in an image data storage unit.
FIG. 6 is a block diagram illustrating a hardware configuration of the image recording apparatus according to the present embodiment.
FIG. 7 is a flowchart showing a processing procedure by an image / map associating unit 30;
FIG. 8 is a diagram showing a screen display example in a user interface for setting a section point and a road.
FIG. 9A is a diagram illustrating an example of registration of a breakpoint;
FIG. 9B is a diagram illustrating an example of registering a road.
FIG. 10 is a diagram showing a display example of a GUI screen displayed in step S3 for associating image data and map data according to the present embodiment.
FIG. 11 is a diagram showing a concept of obtaining a position where each image frame is captured by associating a map with image data.
FIG. 12 is a diagram illustrating an example of a recording format of link data stored in an association data storage unit 50;
FIG. 13 is a block diagram illustrating a configuration of an image collection system according to a second embodiment.
FIG. 14 is a diagram illustrating an example of a storage state of video data and position data (GPS measurement result data) in an image data storage unit 10 according to the second embodiment.
FIG. 15 is a diagram showing a display state of a map data display unit 301 according to the second embodiment.
FIG. 16 is a flowchart illustrating a processing procedure according to the second embodiment.
FIG. 17 is a block diagram showing a functional configuration of a reproduction system of image data according to a third embodiment.
FIG. 18 is a diagram illustrating a display example according to the third embodiment.
FIG. 19 is a diagram illustrating a change in a traveling direction during image reproduction.
FIG. 20 is a flowchart illustrating an image reproduction process according to the third embodiment.
FIG. 21 is a flowchart showing a procedure for acquiring a frame to be reproduced from a designated position.
FIG. 22 is a flowchart showing another procedure for acquiring a frame to be reproduced from a designated position.
FIG. 23 is a diagram showing a display example according to the fourth embodiment.

Claims (5)

An image processing device for constructing a walkable virtual space using the captured image,
A captured image storage unit that stores captured images captured by a plurality of imaging devices,
A map data storage unit for storing map data,
An image-map associating unit that associates the map data with the captured image based on a user's instruction,
The image / map associating unit includes:
An operation screen for performing a map image display according to the map data and an image display according to the photographed image, wherein the image display synchronously displays photographed images photographed by the plurality of imaging devices. An image processing apparatus characterized by the above-mentioned. 2. The image processing apparatus according to claim 1, wherein the number of images to be displayed in synchronization can be designated. The image processing apparatus according to claim 1, wherein the image display displays the plurality of continuous captured images. An image processing method for constructing a walkable virtual space using a captured image,
A map image is obtained from a map data storage unit for storing map data, and a map image to be displayed is displayed. A providing step of providing an operation screen for displaying
An image / map associating step of associating the map data with the photographed image based on a user instruction on the operation screen,
The image processing method according to claim 1, wherein the display of the captured image on the operation screen includes displaying the captured images captured by the plurality of imaging devices in synchronization with each other. A recording medium storing a control program for causing a computer to execute an image processing method for constructing a walkable virtual space using a captured image, wherein the image processing method includes:
A map image is obtained from a map data storage unit for storing map data, and a map image to be displayed is displayed. A providing step of providing an operation screen for displaying
An image / map associating step of associating the map data with the photographed image based on a user instruction on the operation screen,
A recording medium according to claim 1, wherein the display of the captured image on the operation screen displays the captured images captured by the plurality of imaging devices in synchronization with each other.

Images