JP7696129B1 - Information processing system, information processing method, and program - Google Patents

Abstract

[Problem] To provide an information processing system, information processing method, and program for supporting management of image data. [Solution] An information processing system comprising: a three-dimensional data storage unit that stores partial model information and structure model information, a virtual space generation unit that generates a virtual space in which a three-dimensional shape model of a structure is arranged based on the structure model information, an image information acquisition unit that acquires an image and shooting condition information associated with the image, a shooting target identification unit that identifies a shooting target model shown in the image based on the shooting condition information from among the partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space, and an image information writing unit that links attribute information associated with the partial shape model identified as the shooting target model to the image. [Selected Figure] Figure 8

Description

This disclosure relates to an information processing system, an information processing method, and a program related to route planning for a moving object.

As shown in Patent Document 1, a system is known that uses a camera to photograph and inspect a given structure.

JP 2019-179422 A

In conventional inspection systems, workers would typically identify the inspection areas to be photographed on a terminal or on paper by referring to design drawings, etc., and then record and manage the identified inspection areas and information obtained during the inspection, such as the inspection results, in a field notebook or similar. In this case, after the inspection, workers would need to manually associate the images taken during the inspection with the inspection areas by referring to the information written in the field notebook or similar.

In this work, it is easy to make mistakes, such as matching an image of an inspection area with an image of a different location. In particular, when a structure has a series of similar structural patterns, mistakes such as mistaking the inspection area when recording in a field notebook or the like and/or when associating images are more likely to occur. For this reason, there is a demand for a more accurate system for managing image data taken during inspections, etc.

The objective of the exemplary embodiment of the present disclosure is to provide an information processing system, an information processing method, and a program for assisting in the management of image data.

An information processing system according to one embodiment of the present disclosure includes:
a three-dimensional data storage unit that stores a plurality of pieces of partial model information for indicating a three-dimensional structure of a portion of a structure, and structure model information for indicating a three-dimensional structure of the structure constructed by integrating the plurality of pieces of partial model information;
a virtual space generation unit that generates a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information;
an image information acquisition unit that acquires an image acquired by photographing a real space with a camera and photographing condition information associated with the image;
a photographing target specifying unit that specifies a photographing target model shown in the image from among partial shape models included in a three-dimensional shape model of the structure arranged in the virtual space based on the photographing condition information;
The image information writing unit links attribute information associated with the partial shape model identified as the photographing target model to the image.

By having an information processing system with the above features, it is possible to automatically record information that identifies the subject of the image in the image data, preventing erroneous recording and optimizing the management of image data.

FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing a hardware configuration of the management server shown in FIG. FIG. 3 is a block diagram showing a hardware configuration of the user terminal shown in FIG. FIG. 4 is a block diagram showing the hardware configuration of the flying object shown in FIG. FIG. 5 is a block diagram illustrating the functions of the management server. FIG. 6 is a conceptual diagram showing an example of operation of the information processing system shown in FIG. FIG. 7 is a diagram for explaining the process related to the identification of the subject model. FIG. 8 is a flowchart showing an example of information processing executed in the information processing system shown in FIG.

The information processing method, information processing system, and program disclosed herein have, for example, the following configuration.
[Item 1]
a three-dimensional data storage unit that stores a plurality of pieces of partial model information for indicating a three-dimensional structure of a portion of a structure, and structure model information for indicating a three-dimensional structure of the structure constructed by integrating the plurality of pieces of partial model information;
a virtual space generation unit that generates a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information;
an image information acquisition unit that acquires an image acquired by photographing a real space with a camera and photographing condition information associated with the image;
a photographing target specifying unit that specifies a photographing target model shown in the image from among partial shape models included in a three-dimensional shape model of the structure arranged in the virtual space based on the photographing condition information;
An information processing system comprising: an image information writing unit that links attribute information associated with a partial shape model identified as the model to be photographed to the image.
[Item 2]
the image information acquisition unit acquires, as the photographing condition information, information indicating an actual photographing position in real space of the camera when the image was photographed, and an actual photographing direction, which is a direction in real space to which an optical center axis of the camera faces;
The information processing system described in item 1, wherein the shooting subject identification unit calculates a virtual straight line in the virtual space extending from a virtual shooting position corresponding to the actual shooting position to a virtual shooting direction corresponding to the actual shooting direction based on information indicating the actual shooting position and the actual shooting direction, and identifies a partial shape model that intersects with the virtual straight line as the shooting subject model.
[Item 3]
a part of the structure represented by each of the partial shape models corresponds to a member included in the structure,
3. The information processing system of claim 1, wherein the attribute information associated with each of the partial shape models includes a name of the component and partition information indicating the partition in which the component is located in the structure.
[Item 4]
4. The information processing system according to item 3, wherein the image information writing unit writes the name of the component associated with the partial shape model identified as the model to be photographed and the partition information into auxiliary information of the image.
[Item 5]
storing a plurality of pieces of partial model information for indicating a three-dimensional structure of a portion of a structure, and structure model information for indicating a three-dimensional structure of the structure constructed by integrating the plurality of pieces of partial model information;
generating a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information;
Acquiring an image acquired by photographing a real space with a camera and photographing condition information associated with the image;
Identifying a photographing target model shown in the image based on the photographing condition information from among partial shape models included in a three-dimensional shape model of the structure arranged in the virtual space;
and linking attribute information associated with the partial shape model identified as the photographing target model to the image.
[Item 6]
storing a plurality of pieces of partial model information for indicating a three-dimensional structure of a portion of a structure, and structure model information for indicating a three-dimensional structure of the structure constructed by integrating the plurality of pieces of partial model information;
generating a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information;
Acquiring an image acquired by photographing a real space with a camera and photographing condition information associated with the image;
Identifying a photographing target model shown in the image based on the photographing condition information from among partial shape models included in a three-dimensional shape model of the structure arranged in the virtual space;
and linking attribute information associated with the partial shape model identified as the model to be photographed to the image.

<Details of the embodiment>
An information processing system according to an embodiment of the present disclosure will be described with reference to the drawings. In each of the attached drawings, identical or similar elements are given identical or similar reference symbols and names, and duplicate descriptions of identical or similar elements may be omitted in the description of the embodiment. Note that the contents shown in each drawing are merely examples for explaining the present embodiment, and are merely schematic examples for easy explanation of the present embodiment. The contents of each drawing may be modified or changed within the scope of no technical problem.

<System Overview>
The information processing system according to the present embodiment is a system that executes a process of automatically recording an image taken during inspection of a structure in association with information on an object depicted in the image. A structure is a structure that is constructed by combining a plurality of materials and/or members (for example, a bridge, a tunnel, a dam, a levee, a building, a factory, or other buildings), and the type of structure to be inspected is not particularly limited. Inspections that are the subject of the operation of the information processing system may be inspections during construction of the structure, inspections at the completion of construction, or periodic inspections after construction, and the type of inspection is not particularly limited. In addition, the method of photographing the structure during inspection is also not particularly limited, and for example, the structure may be photographed with a camera mounted on an unmanned moving body such as a drone, an unmanned aerial vehicle (UAV), or an unmanned ground vehicle (UGV). Alternatively, a worker or other person may operate an imaging device (e.g., a smartphone, a tablet terminal, a digital camera, or other type of camera) to photograph the structure, or the structure may be photographed using a camera mounted on a manned vehicle.

In this embodiment, for the sake of simplicity, the information processing executed by the information processing system will be described in detail by taking as an example a case where a bridge is inspected using a mobile body 4. As shown in FIG. 6, the mobile body 4 is flown around a bridge (a bridge under construction and/or after construction has been completed) and a predetermined portion of the bridge is photographed by a camera 42 mounted on the mobile body 4. The mobile body 4 may be flown by a user (such as a worker) using a remote control, or may be autonomously navigated based on a flight route with waypoints and the like set in advance. There are no particular limitations on the model and specifications of the mobile body 4. For example, the moving object 4 is preferably an aircraft with specifications that allow it to recognize its position relative to the starting position of its movement even in a non-GNSS (Global Navigation Satellite System) environment, and is preferably an aircraft with specifications that allow it to record various shooting conditions, such as the aircraft position at the time of shooting, the aircraft orientation, and the gimbal orientation, as auxiliary information of the captured image (e.g., Exif information (Exif; Exchangeable image file format)).

In the case of bridge inspections such as those shown in Figure 6, conventionally, photographed images have been managed using field notebooks or the like in which workers manually input the inspected parts or components on site. For example, workers record the photographed inspection parts or components in a field notebook or the like while referring to design drawings or the like on a terminal or on paper. Then, at a later date, workers manually determine which inspection part or component the photographed image corresponds to based on the information written in the field notebook or the like, and manually associate the photographed image with the information written in the field notebook or the like. With such conventional recording methods, work errors are likely to occur, such as incorrectly recording an inspection point by associating it with another image taken of a different part. In particular, bridges have the characteristic of having the same structure in succession at a certain interval, so work errors such as incorrectly recording the inspection point on site and associating the wrong image with information are likely to occur.

In the information processing system of this embodiment, a virtual space is generated in which a three-dimensional shape model of a bridge (structure) constructed by combining multiple partial shape models is placed, and in the virtual space, a partial shape model corresponding to an object depicted in a captured image is identified as a captured object model based on shooting condition information (e.g., shooting position, shooting direction, focal length, angle of view, etc.) linked to the captured image. The information processing system then executes a process of linking attribute information associated with the identified captured object model to the captured image. In the information processing system of this embodiment, the above-mentioned information processing can automatically record information identifying the captured object of the image to image data, preventing erroneous recording, optimizing image data management, and improving inspection efficiency.

<System Configuration>
As shown in FIG. 1, the information processing system of this embodiment may have a management server 1, one or more user terminals 2, and one or more mobile objects 4. The management server 1, the user terminal 2, and the mobile object 4 are connected to each other via a network NW so that they can communicate with each other. The illustrated configuration is an example, and is not limited to this. For example, the mobile object 4 may not be connected to the network NW. In that case, the operation of the mobile object 4 may be performed by a transmitter (so-called radio) operated by a user, or image data acquired by a camera of the mobile object 4 may be stored in an auxiliary storage device (e.g., a memory card such as an SD card and/or a USB memory) connected to the mobile object 4, and the image data may be read out from the auxiliary storage device to the user terminal 2 and/or the management server 1 and stored by the user after the fact. The mobile object 4 may be connected to the network NW only for the purpose of operation or for the purpose of storing image data.

<Management Server 1>
Fig. 2 is a diagram showing a hardware configuration of the management server 1. Note that the illustrated configuration is an example, and the management server 1 may have a different configuration.

The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing. The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmission/reception unit 13, an input/output unit 14, etc., which are electrically connected to each other via a bus 15.

The processor 10 is a computing device that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and deployed in the memory 11 to perform various information processing operations.

Memory 11 includes a main memory configured with a volatile storage device such as DRAM (Dynamic Random Access Memory) and an auxiliary memory configured with a non-volatile storage device such as a flash memory or HDD (Hard Disc Drive). Memory 11 is used as a work area for processor 10, and also stores BIOS (Basic Input/Output System) that is executed when management server 1 is started, various setting information, etc.

Storage 12 stores various programs such as application programs. A database that stores data used for each process may be constructed in storage 12. For example, storage unit 120, which will be described later, may be provided in a part of the storage area of storage 12.

The transmission/reception unit 13 is a communication interface that allows the management server 1 to communicate with the user terminal 2 and the mobile object 4, etc., via the communication network. The transmission/reception unit 13 may further include a short-range communication interface for Bluetooth (registered trademark) and BLE (Bluetooth Low Energy) and/or a USB (Universal Serial Bus) terminal, etc.

The input/output unit 14 includes information input devices such as a keyboard and mouse, and output devices such as a display.

Bus 15 is commonly connected to each of the above elements and transmits, for example, address signals, data signals, and various control signals.

<User Terminal 2>
The user terminal 2 shown in Fig. 3 also includes a processor 20, a memory 21, a storage 22, a transmission/reception unit 23, an input/output unit 24, etc., which are electrically connected to each other via a bus 25. The user terminal 2 may be, for example, a remote control device such as a transmitter, a general-purpose computer such as a workstation or a personal computer, or a mobile terminal such as a smartphone or a tablet terminal. The functions of each element of the user terminal 2 can be configured in the same way as the above-mentioned management server 1, and detailed description of each element of the user terminal 2 will be omitted.

<Mobile object 4>
4 is a block diagram showing a hardware configuration of the moving body 4. The flight controller 41 may have one or more processors, such as a programmable processor (e.g., a central processing unit (CPU)).

The flight controller 41 may also have or have access to memory 411. The memory 411 stores logic, code, and/or program instructions that the flight controller can execute to perform one or more steps. The flight controller 41 may also include sensors 412, such as inertial sensors (accelerometers, gyro sensors), GPS sensors, and proximity sensors (e.g., lidar).

The memory 411 may include, for example, a separable medium such as an SD card and random access memory (RAM) or an external storage device. Data acquired from the cameras/sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image/video data captured by a camera or the like may be recorded in an internal memory or an external memory, but is not limited to this, and may be recorded in at least one of the management server 1 or the user terminal 2 from the camera/sensor 42 or the internal memory via the network NW. The camera 42 is installed on the moving body 4 via a gimbal 43.

The flight controller 41 includes a control module (not shown) configured to control the state of the moving body 4. For example, the control module controls the propulsion mechanism (motor 45, etc.) of the flying body via an ESC 44 (electric speed controller) in order to adjust the spatial arrangement, speed, and/or acceleration of the flying body having six degrees of freedom (translational motion x, y, and z, and rotational motion θ _x , θ _y , and θ _z ). The propeller 46 rotates by the motor 45 powered by a battery 48 to generate lift of the flying body. The control module of the flight controller 41 may have a function of controlling the driving of the camera 42 and the shooting conditions (for example, sharpness, focus (focal length), exposure value, shutter speed, ISO sensitivity, lens aperture, etc.). In addition, the control module can control one or more of the states of the mounted parts and sensors.

The flight controller 41 can communicate with a transceiver 47 configured to transmit and/or receive data from one or more external devices (e.g., a transceiver 49, a terminal, a display device, or other remote control). The transceiver 49 may use any suitable communication means, such as wired or wireless communication.

For example, the transceiver 47 may utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, radio, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, and the like.

The transmitter/receiver 47 can transmit and/or receive one or more of the following: data acquired by the cameras/sensors 42, processing results generated by the flight controller 41, specific control data, user commands from a terminal or a remote controller, etc.

The cameras/sensors 42 may include inertial sensors (accelerometers, gyro sensors), GPS sensors, proximity sensors (e.g., lidar), or vision/image sensors (e.g., cameras).

<Functions of Management Server 1>
5 is a block diagram illustrating functions implemented in the management server 1. In this embodiment, the management server 1 may include a virtual space generation unit 101, an image information acquisition unit 102, a shooting target identification unit 103, an image information writing unit 104, and a display control unit 105. In addition, the storage unit 120 of the management server 1 may include various databases such as a three-dimensional data storage unit 121, a shooting image data storage unit 122, and a movement information storage unit 123.

First, various databases of the storage unit 120 will be described in detail. The three-dimensional data storage unit 121 stores a plurality of partial model information for indicating the three-dimensional structure of a part of a structure, and structure model information for indicating the three-dimensional structure of a structure constructed by integrating the plurality of partial model information. The data format of the three-dimensional model data such as the partial model information and the structure model information is not necessarily limited. For example, the three-dimensional model data may be constructed using BIM (Building Information Modeling) data, CIM (Construction Information Modeling) data, CAD data, or GML data such as City-GML, or may be constructed using two or more data formats selected from these data formats. Among the above data formats, it is preferable that the three-dimensional model data is constructed using BIM data and/or CIM data, which includes not only information indicating the three-dimensional shape but also attribute information. In other words, both the partial model information and the structure model information may include a three-dimensional shape model (three-dimensional geometry) that indicates the three-dimensional shape of each target object, and attribute information of the target object.

The "part of the structure" indicated by the partial model information may be a "component" included in the structure, or a cohesive "part" composed of multiple components. In other words, the part of the structure indicated by each piece of partial model information is a part that corresponds to a small unit model that is isolated when the three-dimensional model of the structure is subdivided, and the part may be set by subdividing the structure according to the inspection item.

In the case of a bridge as exemplified in this embodiment, the three-dimensional data storage unit 121 may store partial model information corresponding to each component included in the bridge. Examples of bridge components include decks, main girders, cross beams, vertical beams, upper chords, lower chords, diagonal members, vertical members, supports, piers, gussets, ground guards, and guardrails. A bridge includes multiple components of the same type that have the same shape, but even such components of the same type are placed in different positions. Therefore, for the same type of components included in the bridge, partial model information corresponding to each placement is generated and stored in the three-dimensional data storage unit 121.

Each piece of partial model information as described above includes a partial shape model showing the three-dimensional shape of the target component and attribute information of the component, and the attribute information of the partial model information includes, for example, the name of the component and partition information that identifies the partition in which the component is located on the bridge. A partition is an area that divides a structure into areas that are coarser than the components (e.g., a floor, room, or building of a structure), and a partition on a bridge may be an area separated by a periodic, predetermined interval, such as a span, a span, or the connection interval of a main girder. Note that when multiple components of the same type are included in the same partition, the attribute information of the partial model information may include identification information such as a number for uniquely identifying each component. In addition to the above information, the partial model information may also include information indicating the arrangement of components (e.g., three-dimensional coordinates indicating the position of the component in real space and/or virtual space), information related to the component specifications such as the material, color, dimensions, product number, and performance of the component, information indicating the relationship with other adjacent components (e.g., information indicating which component the target component is connected to, at what position, and how), information related to the procurement history such as the price of the component, and the source of procurement.

The structure model information may include a three-dimensional shape model of the bridge constructed by combining partial shape models, and attribute information associated with the three-dimensional shape model. The attribute information of the structure model information may include, for example, the dimensions and characteristics (e.g., design specifications, etc.) of the bridge when the components are combined, and the position information of the bridge (e.g., three-dimensional coordinates of multiple representative points set on the outer surface of the bridge). The structure model information may also include information on accessories (road signs, lighting, etc.) installed on the bridge, information on the relationship between adjacent components (e.g., which main girder and which cross girder are connected at which position, etc.), information on the surrounding environment of the bridge (e.g., information related to the roads connecting the bridge, information indicating the positional relationship with surrounding facilities, etc.), information on the virtual space in which the three-dimensional shape model is placed (e.g., reference coordinates specified by user operation, settings of spatial areas defined in a three-dimensional coordinate system, scale of the virtual space, etc.), and information on the construction history.

Note that the partial model information and structure model information shown above are merely examples, and each piece of model information may be associated with information accumulated throughout the entire cycle from design to construction and maintenance. The virtual space generation unit 101, which will be described later, reads out the partial model information, structure model information, etc. stored in the three-dimensional data storage unit 121, and generates a virtual space based on this information.

The photographed image data storage unit 122 stores photographed image data that is acquired by photographing a structure with the camera/sensors 42 of the mobile body 4 and transmitted from the mobile body 4 to the management server 1. The photographed image data acquired by photographing with the camera/sensors 42 includes a photographed image that shows at least a portion of the structure and photographing condition information associated with the photographed image.

The shooting condition information includes, for example, the actual shooting position of the camera 42 in the real space at the time when the image was captured, and the actual shooting direction, which is the direction in the real space to which the optical center axis of the camera 42 faces at the time when the image was captured. The actual shooting position is a position expressed in a three-dimensional coordinate system in the real space. When the actual shooting position is acquired using a GNSS sensor or the like, the actual shooting position may be expressed in three-dimensional coordinates of latitude, longitude, and absolute altitude, and when the actual shooting position is acquired in a non-GNSS environment using an inertial sensor (acceleration sensor, gyro sensor), the actual shooting position may be expressed in relative three-dimensional coordinates with the movement start position of the moving body 4 (shooting operation start position) as the reference position (origin). The actual shooting direction may be specified, for example, based on the orientation of the aircraft at the time of shooting, the orientation of the gimbal, etc. In addition to the above, the shooting condition information may also include information indicating the focal length, angle of view, zoom ratio, exposure amount, ISO sensitivity, shooting date and time, etc. of the camera 42 at the time of shooting. This shooting condition information may be metadata attached to the captured image, or may be saved as Exif information.

Each captured image data stored in the captured image data storage unit 122 may be associated with information indicating the subject of the image capture at the capture point, which is identified by the subject of the image capture unit 103 described below and linked by the image information writing unit 104. The information indicating the subject of the image capture may include, for example, identification information that uniquely identifies the subject member that is the main subject, and may include the name of the subject member and section information indicating the section in which the subject member is located. Furthermore, the associated information associated with the captured image data may include, in addition to information indicating the subject of the image capture as the main subject, information indicating other members captured in the capture range.

The movement information storage unit 123 stores, for example, movement information used in movement for the purpose of photographing various objects to be photographed in a structure (for example, each component included in a bridge). The movement information is instruction information for executing movement, and may include, for example, movement route information (including waypoint information), movement speed, flight altitude, imaging conditions (imaging direction, imaging angle of view, imaging focal length, overlap rate of captured images, etc.). The movement information storage unit 123 may also store information acquired during movement that is acquired as an actual result by executing movement. The information acquired during movement may include the actual movement route, movement speed, flight altitude, captured images, and imaging condition information associated with the captured images.

The movement information as the instruction information may be generated, for example, by setting the parameters of various information included in the movement information on the management server 1 or the user terminal 2. In addition, as the movement route, for example, the position of the mobile body storage device may be set as the movement start position and the movement end position, and a movement route passing through each waypoint may be generated. Conversely, without a mobile body storage device, the position where the user carries the mobile body 4 may be set as the movement start position (so-called home point), and the user may collect the mobile body 4 at the movement end position (which may be back to the home point). Alternatively, a movement route may be generated that includes the position of the mobile body storage device selected as the movement start position or movement end position based on information of the mobile body storage device managed by the management server 1 (for example, position information, storage state information, storage machine information, etc.).

Next, each functional part of the processor 10 will be described in detail.

The virtual space generating unit 101 generates a virtual space in which a three-dimensional shape model of a structure is arranged based on the structure model information. FIG. 7B is a diagram showing an example of the generated virtual space, in which a bridge model 60, which is a three-dimensional shape model of a bridge, is arranged. The bridge model 60 shown in FIG. 7B has sections (J21-J26) determined according to the connection interval of the main girders, and each section includes multiple partial shape models (e.g., partial shape models corresponding to members 61-66 in FIG. 7A). When generating the virtual space, each partial model information may also be taken into consideration, and a three-dimensional shape model of the structure may be constructed by combining the partial shape models. In the three-dimensional shape model of the structure arranged in the virtual space, the parts corresponding to each partial model information (e.g., each member of the bridge) may be displayed in an identifiable manner, for example by color coding.

When generating the virtual space, the virtual space generation unit 101 may execute a process of constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space by associating the three-dimensional coordinate system of the real space with the three-dimensional coordinate system of the virtual space. As an example, the virtual space generation unit 101 aligns and associates a reference position in the three-dimensional coordinate system of the real space (e.g., the start position of the movement of the moving body 4) with a reference position 55 (reference number 55 in FIG. 7B) in the three-dimensional coordinate system of the virtual space corresponding to the reference position, thereby constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space. As a result, the virtual space generation unit 101 can convert and express the position in the three-dimensional coordinate system of the real space of the moving body 4 flying in the real space into the position in the three-dimensional coordinate system of the virtual moving body 57 (a virtual object corresponding to the moving body 4 in the real space) in the virtual space. If the scale of the three-dimensional shape model in the virtual space differs from the scale of the corresponding structure in the real space, the scale of the three-dimensional shape model may be adjusted during alignment.

As described above, by constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, it becomes possible to specify a position, such as generating a movement path based on the three-dimensional coordinate system of the moving body 4 in the real space, even if a waypoint is set based on the three-dimensional coordinate system of the virtual space displayed on the user terminal 2. When the three-dimensional model data includes dimensional information, it becomes possible to generate a movement path using a real scale, such as moving straight for 10 m and then turning right, based on the three-dimensional coordinate system of the virtual space.

The virtual space generation unit 101 may generate a display screen showing the generated virtual space and execute a process of transmitting the display screen to the user terminal 2. On the display screen of the user terminal 2, the three-dimensional shape model (bridge model 60) arranged in the virtual space may be enlarged, reduced, rotated, etc., in response to a user operation, and the virtual space generation unit 101 renders an image of the virtual space in response to the user operation and outputs it to the user terminal 2. In addition, the virtual space generation unit 101 may receive a selection operation by the user, such as clicking on a part of the three-dimensional shape model (bridge model 60) of the structure, via the display screen of the user terminal 2 displaying the virtual space, and execute a process of highlighting a partial shape model corresponding to the selected part (component) (e.g., displaying in a coordinated color, enlarging only the selected partial shape model, etc.) or displaying attribute information associated with the partial shape model.

The image information acquisition unit 102 acquires a captured image acquired by capturing an image of a real space with the camera 42, and capturing condition information associated with the captured image. The captured image and capturing condition information acquired by the image information acquisition unit 102 are used as input information in the process of identifying a capture target model by the capture target identification unit 103 described below. The image information acquisition unit 102 acquires, for example, the actual capture position and the actual capture direction as the capture condition information, and may also acquire other information such as the focal length and angle of view when the capture image was captured by the camera 42.

The timing at which the image information acquisition unit 102 acquires the captured image and the shooting condition information is not particularly limited. For example, the image information acquisition unit 102 may receive the captured image and the shooting condition information in real time from the moving body 4 when the moving body 4 performs shooting. Also, the image information acquisition unit 102 may execute a process of importing the captured image and the shooting condition information based on an instruction from a user during the shooting operation by the moving body 4 or at any timing after the shooting operation is completed.

The imaging subject identification unit 103 executes a process of identifying the imaging subject model displayed in the captured image from among the partial shape models included in the three-dimensional shape model of the structure placed in the virtual space, based on the imaging condition information. Figure 7 is a diagram for explaining the imaging subject model identification process executed by the imaging subject identification unit 103, and shows how the imaging subject member 62 displayed in the captured image 50 shown in (a) is identified in the virtual space as shown in (b). The imaging subject identification unit 103 may identify the imaging subject model present at the imaging subject position in the virtual space, for example, based on the center position of the angle of view of the camera 42 when the captured image 50 was captured.

Specifically, the shooting target identification unit 103 obtains a virtual shooting direction in the virtual space corresponding to the real shooting direction of the camera 42 based on information indicating the real shooting position and the real shooting direction among the shooting condition information acquired by the image information acquisition unit 102. The virtual shooting direction can be expressed as a virtual straight line extending from the virtual shooting position in the virtual space corresponding to the real shooting position of the camera 42 to the virtual shooting direction corresponding to the real shooting direction of the camera 42. The virtual shooting position can be specified based on the correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, and the virtual shooting direction corresponds to the center position of the angle of view of the camera 42. The shooting target identification unit 103 obtains a virtual straight line (arrow indicated by a dashed line in FIG. 7B) extending from the virtual shooting position (position of the virtual moving body 57 in FIG. 7B) corresponding to the real shooting position in the virtual space in the virtual shooting direction corresponding to the real shooting direction, and identifies the partial shape model intersecting with the virtual straight line as the shooting target model. In FIG. 7(b), among the components (e.g., components 61-66) included in the shooting range of the captured image 50, the three-dimensional shape model of component 62 (the component shown in gray in FIG. 7(a) "Name: Lighting Widening Section") located in section J24 intersects with the dashed dotted arrow and is identified as the model to be shot.

Note that even if a component is included in the shooting range of the captured image 50, the component (e.g., components 61, 63-66, etc.) that does not intersect with a virtual line extending in the virtual shooting direction may be treated as a non-target component in the process executed by the image information writing unit 104 described below. Alternatively, the partial shape model (component 62) that intersects with the virtual line may be set as the main target model, and the partial shape model (components 61, 63-66, etc.) that is included in the shooting range but does not intersect with the virtual line may be specified as a secondary target model.

In the latter case, the shooting target identification unit 103 may identify the secondary target model by calculating a virtual shooting range corresponding to the real shooting range in the real space based on, for example, the angle of view information and focal length information of the camera 42, which are shooting condition information, and virtual distance information indicating the virtual distance from the virtual shooting position to the shooting target position. Specifically, based on the angle of view information, focal length information, and virtual distance information, the "near plane" and "far plane" of the view frustum in the virtual space are identified, and the range from the "near plane" to the "far plane" is calculated as the virtual shooting range. Then, the shooting target identification unit 103 identifies the partial shape model included in the virtual shooting range (excluding the member that is in the blind spot of the camera 42 and is not shown in the image). The shooting target identification unit 103 identifies the partial shape model (main target model) that intersects with the virtual straight line from the partial shape models included in the virtual shooting range, and identifies the other partial shape models that do not intersect with the virtual straight line as secondary target models.

The image information writing unit 104 executes a process of linking attribute information associated with the partial shape model identified as the photographed object model to the photographed image. The attribute information linked to the photographed image may be identification information for uniquely identifying the photographed object model. For example, the image information writing unit 104 may link the name of the component and/or the section information (information indicating the section in which the component is located in the structure (bridge)) from the attribute information associated with the photographed object model to the photographed image.

The process of "linking" attribute information to a captured image performed by the image information writing unit 104 may be, for example, a process of writing attribute information such as the name of a component and/or section information into the auxiliary information of the captured image and storing (storing) the captured image data including the auxiliary information in the captured image data storage unit 122. The file format (data format) of the auxiliary information is not particularly limited, and the image information writing unit 104 may execute a process of writing attribute information of the captured subject model into the Exif information (an example of auxiliary information) of the captured image. In the case of the example shown in FIG. 7, the image information writing unit 104 executes a process of writing the name of the component 62 identified as the captured subject model, "Lighting Widening Section," and the section number "J24" in which the component 62 is located, into the auxiliary information of the captured image as information indicating the captured subject (capture point).

When the shooting subject identification unit 103 identifies a secondary object model included within the shooting range, the image information writing unit 104 may execute a process of writing attribute information of the secondary object model to the auxiliary information of the captured image. At this time, the attribute information of the secondary object model may be written to the auxiliary information as information indicating other reflected parts other than the main object (photographed object).

The display control unit 105 executes a process of reading out a specific captured image stored in the captured image data storage unit 122 based on a user's request and outputting it to the display screen of the user terminal 2. For example, the display control unit 105 may receive an image output request from the user via the user terminal 2 and output the captured image in response to the request. When receiving an image output request, the display control unit 105 may receive a designation of output conditions such as the shooting date and time, the target component, and the target section, and identify the captured image corresponding to the designated output condition by referring to the accompanying information of each captured image data. When multiple captured images satisfy the output condition designated by the user, the display control unit 105 may output a list (image list) of the multiple captured images that satisfy the output condition. When the display control unit 105 displays the captured image (single image or image list) designated by the user on the user terminal 2, the captured image and information identifying the captured object written in the accompanying information by the image information writing unit 104 (for example, the name of the component that is the captured object, section information indicating the section in which the component is located, etc.) may be displayed side by side.

In addition to or instead of the above-described search process for captured images, the display control unit 105 may perform a filtering process for narrowing down the list of captured images to those that meet the output conditions specified by the user, and a sorting process for rearranging the captured images based on the sorting conditions specified by the user and outputting the list. Even in this case, the display control unit 105 accepts the specification of condition items such as the capture date and time, target material, and target section desired by the user as filtering conditions (narrowing conditions) or sorting conditions.

The conditions (search conditions, filtering conditions, sorting conditions) for outputting the captured images may be input in text format or through a predetermined selection form such as a pull-down list. The display control unit 105 may also accept the user's designation of the target member through a UI (user interface) that displays the three-dimensional shape model of the structure on the user terminal 2. Specifically, the display control unit 105 may accept the designation of any one member or section included in the three-dimensional shape model (for example, by clicking and selecting a desired partial shape model or section) on the UI on which the three-dimensional shape model of the structure is displayed. Then, the display control unit 105 refers to the auxiliary information of each captured image data stored in the captured image data storage unit 122, and identifies the captured image in which the designated member or section is written in the auxiliary information. In this case, the display control unit 105 may switch from a screen displaying the three-dimensional shape model to a screen displaying the captured images, and output the identified captured image (a single image or an image list), or may display the three-dimensional shape model and the identified captured image side by side on the display screen of the user terminal 2.

The functions of the display control unit 105 as described above may be executed by the processor 20 of the user terminal 2. The processor 10 may also have other functional units not illustrated in FIG. 5. For example, the processor 10 may have a functional unit that controls the movement of the moving body 4 for the purpose of capturing real images of various objects inside and outside a structure, based on various movement information stored in the movement information storage unit 123.

<Example of information processing method>
Next, an information processing method (information processing method relating to management of captured image data) by the information processing system according to the present embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart illustrating the information processing method according to the present embodiment.

First, the virtual space generation unit 101 in the information processing system reads out the structure model information (and partial model information) stored in the three-dimensional data storage unit 121 (step SQ101 in FIG. 8). Then, based on the read out structure model information, the virtual space generation unit 101 generates a virtual space in which a three-dimensional shape model of a structure constructed by combining partial shape models is placed (step SQ102). When generating the virtual space in this step SQ102, the virtual space generation unit 101 may execute a process of constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space by associating the three-dimensional coordinate system of the real space with the three-dimensional coordinate system of the virtual space.

Next, the image information acquisition unit 102 acquires the captured image captured by the camera 42 of the moving object 4 and transmitted from the moving object 4 to the management server 1, and the capturing condition information associated with the captured image (step SQ103). At this time, the image information acquisition unit 102 may acquire, as the capturing condition information, information indicating, for example, the actual capturing position in real space of the camera 42 when the captured image was captured, and the actual capturing direction, which is the direction in real space to which the optical center axis of the camera 42 faces, and may also acquire information indicating the focal length and the angle of view of the camera 42 when the captured image was captured.

Next, the shooting target identification unit 103 identifies the shooting target model shown in the captured image from among the partial shape models included in the 3D shape model of the structure placed in the virtual space based on the shooting condition information (step SQ104). For example, based on information indicating the actual shooting position and actual shooting direction, the shooting target identification unit 103 determines a virtual straight line extending in the virtual space from the virtual shooting position corresponding to the actual shooting position to the virtual shooting direction corresponding to the actual shooting direction, and identifies the partial shape model that intersects with the virtual straight line as the shooting target model (for example, (b) of FIG. 7).

Then, the image information writing unit 104 links attribute information associated with the partial shape model identified as the photographed object model by the photographed object identification unit 103 to the photographed image (step SQ105). The attribute information associated with the partial shape model may include, for example, information indicating the name of the component, and partition information indicating the partition in which the component is located in the structure. In step SQ105, the image information writing unit 104 may execute a process of writing the name of the component and the partition information linked to the photographed object model as attribute information to the auxiliary information (e.g., Exif information) of the photographed image.

As described above, in the information processing system according to this embodiment, attribute information such as the names of components corresponding to partial shape models included in the three-dimensional shape model of the structure is centrally managed on the three-dimensional shape model of the structure, and attribute information of the photographed object model identified in the virtual space is automatically linked to the photographed image. In other words, the information processing system can prevent erroneous recording of information associated with the photographed image when using photographed image data for the inspection of a structure, etc. Also, as described above, information identifying the photographed object in the image can be automatically recorded for the photographed image data, and management of the photographed image data can be optimized. Based on these effects, effects such as more efficient inspection work and guaranteed accuracy (certainty) of the inspection results can be expected.

The above-described embodiments are merely examples to facilitate understanding of the present disclosure, and are not intended to limit the interpretation of the present disclosure. This disclosure can be modified and improved without departing from its spirit, and it goes without saying that this disclosure includes equivalents.

For example, in the above embodiment, an information processing system relating to management of captured image data has been described mainly with an example of inspecting a bridge using an unmanned mobile object such as a drone, but application examples of the information processing system are not limited to bridge inspections. The subject of inspection may be other structures such as tunnels and dams, and photographing during inspection may be performed using a portable device such as a smartphone, tablet terminal, or other camera held by a person, or may be performed using a manned mobile object.

In addition, the moving object 4 used to photograph the structure may further include, in addition to the above-mentioned cameras/sensors 42, devices, equipment, etc. used to inspect the presence or absence of a specific event inside and/or outside the structure. More specifically, all devices necessary to know the state of the inspected structure, such as imaging devices (visible light cameras, infrared cameras, metal detectors, ultrasonic measuring devices, etc.), keying devices, etc., detection devices (metal detectors), sound collection devices, odor measuring devices, gas detectors, air pollution measuring devices, detection devices (devices for detecting cosmic rays, radiation, electromagnetic waves, etc.), etc., may be adopted. The attribute information (such as the name of the component) of the photographed object model identified by the photographed object identification unit 103 may be linked to the inspection data acquired by other inspection devices as described above, in the same way as linking to the photographed image.

In addition, the purpose of photography in the information processing system is not limited to inspection, but may also be security, monitoring of infrastructure, surveying, disaster response, etc., and the information processing system of the present disclosure may be applied to image data acquired by photography for these purposes.

Some or all of the functional units, such as the virtual space generation unit 101, image information acquisition unit 102, shooting target identification unit 103, image information writing unit 104, and display control unit 105 described in the above embodiment, may be executed by the processor 20 of the user terminal 2 or the flight controller 41 of the moving object 4, instead of the processor 10 of the management server 1.

1 Management server 2 User terminal 4 Mobile object

Claims (5)

a three-dimensional data storage unit that stores a plurality of pieces of partial model information for indicating a three-dimensional structure of each member included in a bridge , and bridge model information for indicating a three-dimensional structure of the bridge constructed by integrating the plurality of pieces of partial model information;
a virtual space generation unit that generates a virtual space in which a three-dimensional shape model of the bridge is arranged based on the bridge model information;
an image information acquisition unit that acquires an image acquired by photographing a real space with a camera and photographing condition information associated with the image;
a photographing target specifying unit that specifies a photographing target model shown in the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the bridge arranged in the virtual space;
an image information writing unit that links attribute information associated with the partial shape model identified as the photographing target model to the image ,
the attribute information associated with each of the partial shape models includes a name of the component, and section information indicating a section in which the component is located among sections divided at predetermined periodic intervals on the bridge;
An information processing system , wherein the image information writing unit writes the name of the component associated with the partial shape model identified as the model to be photographed, and the partition information, into auxiliary information of the image . the image information acquisition unit acquires, as the photographing condition information, information indicating an actual photographing position in real space of the camera when the image was photographed, and an actual photographing direction, which is a direction in real space to which an optical center axis of the camera faces;
2. The information processing system of claim 1, wherein the shooting subject identification unit determines, based on information indicating the actual shooting position and the actual shooting direction, a virtual straight line extending in the virtual space from a virtual shooting position corresponding to the actual shooting position to a virtual shooting direction corresponding to the actual shooting direction, and identifies a partial shape model that intersects with the virtual straight line as the shooting subject model. The imaging target identification unit is
determining, in the virtual space, a virtual straight line extending from a virtual shooting position corresponding to an actual shooting position to a virtual shooting direction corresponding to an actual shooting direction based on the shooting condition information;
The information processing system according to claim 1 or 2, wherein a partial shape model that intersects with the virtual straight line is identified as the photographed object model, and a partial shape model that is included in the photographed range but does not intersect with the virtual straight line is identified as a secondary object model. Storing a plurality of pieces of partial model information for indicating a three-dimensional structure of each member included in the bridge , and bridge model information for indicating a three-dimensional structure of the bridge constructed by integrating the plurality of pieces of partial model information;
generating a virtual space in which a three-dimensional shape model of the bridge is arranged based on the bridge model information;
Acquiring an image acquired by photographing a real space with a camera and photographing condition information associated with the image;
Identifying a photographing target model shown in the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the bridge arranged in the virtual space;
an image information writing unit links attribute information associated with the partial shape model identified as the photographing target model to the image;
the attribute information associated with each of the partial shape models includes a name of the component, and section information indicating a section in which the component is located among sections divided at predetermined periodic intervals on the bridge;
An information processing method , wherein the image information writing unit writes the name of the component associated with the partial shape model identified as the model to be photographed, and the partition information, into auxiliary information of the image . storing a plurality of pieces of partial model information for indicating a three-dimensional structure of each member included in the bridge , and bridge model information for indicating a three-dimensional structure of the bridge constructed by integrating the plurality of pieces of partial model information;
generating a virtual space in which a three-dimensional shape model of the bridge is arranged based on the bridge model information;
Acquiring an image acquired by photographing a real space with a camera and photographing condition information associated with the image;
Identifying a photographing target model shown in the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the bridge arranged in the virtual space;
and linking , by an image information writing unit, attribute information associated with the partial shape model identified as the photographing target model to the image ;
the attribute information associated with each of the partial shape models includes a name of the component, and section information indicating a section in which the component is located among sections divided at predetermined periodic intervals on the bridge;
A program for causing the computer to execute the following: the image information writing unit writes the name of the component associated with the partial shape model identified as the model to be photographed, and the partition information, into the auxiliary information of the image .

Images