JP2015119372A - Multi-camera photographing system and method of combining multi-camera photographic images - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically and highly accurately match positions, directions and dimensions of captured images at time intervals when photographing using a plurality of high-resolution cameras for a routine check of a structure or the like.SOLUTION: In a photography system, a plurality of images captured by a plurality of cameras with an equal resolution are aligned, a specific range in a marker or a common feature and further, a specific range in a group of captured images is represented by one coordinate system at another time. Two or more markers or feature points at the same positions even at different photographing times are set to a part of the structure that is a subject, the two or more markers or feature points on the subject are detected in the captured images, coordinate transform processing is performed by applying the same coordinate values, and correction values onto the same coordinate system are obtained at time intervals over all the captured images.

Description

  The present invention relates to a configuration of an imaging system and an image matching method for observing a temporal change of a structure using multi-camera imaging for imaging a surface of a structure having a large area with high resolution.

  In recent years, structures such as bridges and tunnels built during the period of high economic growth have reached the end of their useful lives. For example, bridges are regularly inspected for damage every five years and need to be repaired or repaired. Determine sex. Visual inspection by humans on structures such as bridges includes places where work is difficult, such as high places and under bridges, and deterioration judgment varies from person to person, so photography with a camera is desired. However, in the inspection of the damage state of the structure, the area of the inspection range is wide and the crack to be inspected is fine. Therefore, the resolution of the single captured image is insufficient or the whole image cannot be captured. In view of this, as an observation technique based on camera photography, a method of generating a high-resolution image by combining images photographed by a plurality of cameras (multi-cameras) has been considered.

  As one of the techniques for synthesizing images captured by a plurality of cameras into a high-resolution image, Patent Document 1 discloses an object 2 including at least a common image capturing portion in each field of view of two or more adjacent CCD cameras 1. It is shown that a laser beam is temporarily irradiated and used as a common marker for size and position correction to synthesize successive images between adjacent images. However, the laser beam marker is a temporary index marker, and even though the image at the time of inspection at the time of inspection can be synthesized, in comparison with the image of the image at the time of another inspection to grasp deterioration over time, It is necessary to manually superimpose a shooting point from the subject and match the position, size (magnification), and orientation between the two shot images. Images can be matched by manually narrowing down the shooting area from the shot images taken at different shooting times and using this as a reference to match, but the setting of the feature points varies depending on the person searching for. Whether the feature point of the appropriate subject is not in the area you want to use as a reference, or the feature point that should have been used as the reference cannot be found due to changes in shape or state over time. There was a possibility that it could not be secured.

  Instead of humans, it is also possible to apply a method of automatically performing correlation such as luminance distribution of a photographed image by using a recent image recognition technology to identify a common feature point among different subjects at the time of photographing. However, in the correlation comparison process of the image luminance distribution, the amount of processing is increased by the square of the width, so that a huge amount of processing is required, and it is essential to manually narrow down the area.

  Further, a method for specifying an arbitrary position of common features in the image from the entire captured image can be considered. As an example, coordinate conversion is performed between a flat image in which a plurality of reference points are provided on a subject and the entire area including the reference points is captured at a low resolution and an inspection image in which a narrow imaging area including the reference points is captured at a high resolution. Japanese Patent Application Laid-Open No. 2004-133867 discloses a method for specifying the position of an arbitrary point on an inspection image using position coordinates based on a reference point of a planar image.

Japanese Patent Laid-Open No. 9-161068 Japanese Patent No. 4170469

  However, in the method of Patent Document 2 described above, it is necessary to arrange the reference points in all the high-resolution shooting areas, the entire image can be shot with low-resolution shooting that defines the reference points, and [0077] As shown in the figure, there is a restriction that, when the resolution ratio is 10 times or more, there is a risk that a large error occurs in the conversion coefficient. If this method is applied, for example, when observing the observation area of the bottom of a bridge with a bridge width of 10 m and a length of several tens of meters (between bridge girders) as the size of the structure, It is impossible to shoot a whole area with a linear image from several meters to one camera. Also, if you want to divide a 30m long shot and apply low resolution shots in multiple ways, it is currently necessary to shoot at 0.2mm resolution required for crack observation etc. with high resolution shots. Then, even with a camera of 10 million pixels (4200 × 2400 pixels), which is still expensive, 750 reference points corresponding to a total of about 750 images are installed, and at least 8 1 / It is necessary to connect with 10 times lower resolution camera shooting. As described above, the photographing system itself is very complicated and enormous amount of work is generated, which is difficult to apply.

  Therefore, the present invention relates to a combination of a plurality of high-resolution captured images for observing aged deterioration of a structure that is a large-area subject, and the size, position, and orientation of the composite image composed of a plurality of images separated by time. To provide a system for automatically displaying the actual size information for evaluation of cracks and the like in a common image by applying the size information of the image of the subject whose actual size is known automatically with high accuracy And

  In order to solve the above-described problems, the present invention is configured as follows.

A group of images using a common feature of the captured subjects, or an alignment marker between adjacent images temporarily provided between a plurality of images obtained by capturing a target structure with a plurality of cameras having equivalent resolution An imaging system that uses a plurality of cameras that represent a specific range of images in a single coordinate system and that are photographed by a plurality of cameras at different times and similarly represent a specific range of images taken at that time in a single coordinate system. In
Set a fixed two or more markers or feature points that are the same position on the subject even during time-lapse photography on a part of the structure that is a wide range of subjects,
In any image within a specific range of the photographed image, two or more fixed markers or feature points on the subject are detected,
The coordinates of two or more markers or feature points that are fixed in one coordinate system of the above-mentioned photographed image specific range are the same except for the time corresponding to two or more fixed markers or feature points. Applying coordinate values to perform coordinate conversion processing, and obtaining correction values to the same coordinate system at intervals over the entire captured image,
The size, position, and orientation of a composite image composed of a plurality of images separated by time are automatically matched with high accuracy.

  Other means will be described in the detailed description.

  According to the present invention, it is possible to automatically display a plurality of high-resolution images taken at intervals in a common coordinate system in periodic structural inspections every several years. It is possible to provide a system that can easily observe deterioration over time such as changes.

It is an installation figure of the fixed reference marker to a part (end part) of a concrete bridge bottom. It is a flowchart from imaging | photography in a periodic inspection to deterioration comparison. It is a multi camera imaging | photography system figure including imaging | photography of the fixed reference marker of the edge part of a bridge bottom face. It is the imaging | photography figure by the multi camera imaging | photography system which looked at the imaging | photography area | region from right under the bridge bottom. It is an imaging | photography flowchart by a multi camera imaging | photography system. It is a block diagram of the whole picked-up image group containing an image alignment marker image, a fixed reference marker image, and a dimension reference marker image. It is a coordinate conversion principle figure from between adjacent images to real world coordinates. It is a block diagram of a coordinate transformation processing unit. It is the conversion flowchart to the reference | standard image coordinate of imaging | photography, a common image coordinate, and a real world coordinate. It is a block diagram of a time-change comparative image processing part. It is a flowchart of comparative image processing. It is the figure which installed the some fixed reference marker in the bridge width direction of a concrete bridge bottom face, and a bridge axis direction. It is a multi camera imaging | photography system figure which performs several fixed reference | standard marker imaging | photography installed in the bridge bottom face. It is an accumulation figure of the image alignment error from a fixed standard marker. It is an effect (linear simple display of error accumulation) of a plurality of fixed reference markers. It is a conversion flowchart to common coordinates or real world coordinates when a plurality of fixed reference markers are installed. It is a block diagram in the case of using the virtual image of the light source produced by reflection of a convex hemispherical mirror for a fixed reference marker. It is a calculation figure of the reflected image size of a light source when using a spherical mirror as a fixed reference marker. It is a calculation figure of the reflected light intensity when using a spherical mirror as a fixed reference marker, and the reflected light intensity from a to-be-photographed object. It is explanatory drawing of the deviation | shift amount from the spherical surface center of the virtual image of the light emission area | region reflected in a convex hemispherical mirror. It is a figure of the camera and light source space | interval dependence of the deviation | shift amount of the virtual image of the light emission area when using a spherical mirror as a fixed reference marker when the distance between a multi-camera apparatus and a fixed reference marker is used as a parameter. It is a figure of the space | interval dependence of a camera and a light source of the deviation | shift amount of the virtual image of the light emission area | region when using a spherical mirror as a fixed reference marker when a spherical mirror radius is used as a parameter. It is a flowchart which correct | amends the coordinate of the coordinate of a fixed reference marker to the center position of a spherical mirror from the deviation | shift amount of the image of the light emission area | region when using a spherical mirror. FIG. 3 is a structural diagram of a fixed reference marker spherical mirror that prevents salt from adhering to the spherical mirror by sea breeze. It is a block diagram of a system for reliably detecting a fixed reference marker and reading ID and coordinate information when an RFID passive tag and an RFID reader are used or when a code mark sheet is used. It is a figure of the example of presentation of fixed reference marker detection mark by a code mark sheet, ID of related information, and coordinate information. FIG. 10 is a flowchart of specifying a reference mark that is combined with the use of a code mark sheet after receiving RFID, and automatically acquiring ID information or coordinate information of the reference mark.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. In each embodiment, the case where the surface observation for seeing degradation, such as a crack of a concrete bridge bottom, is shown as an example.
(First embodiment)
FIG. 1 is a diagram in which fixed reference markers 1-d1 and 1-d2 are installed at an end portion as a part of the concrete bridge bottom surface 101 in the first embodiment. Here, two fixed reference markers are placed close to each other so as to fall within the photographing range of the same camera. One fixed reference marker 1-d1 is the origin of the common coordinate system in the group of captured images separated from each other, and the other fixed reference marker 1-d2 is coupled with the fixed reference marker 1-d1 to the coordinate axis in the common coordinates. The purpose is to give a direction and use the distance between them as a reference for dimensions. Here, the fixed reference marker 1-d that is intentionally given is used. For example, the fixed reference marker 1-d is not characteristically lost or displaced due to deformation, dirt, or the like even if the subject such as the concrete bridge bottom surface 101 is spaced apart. If there is a clear feature point such as a protrusion shape or a dent shape, the same effect can be obtained even if it is applied as a fixed reference marker.

  FIG. 2 shows, as an example of the present invention, a basic flow chart from photographing to deterioration comparison in periodic inspection of the same concrete bridge bottom surface 101 photographed at regular intervals.

  In step 1 (S-1), a fixed reference marker 1-d serving as a reference mark is set at a predetermined position of the subject as a preliminary preparation prior to periodic inspection. Although the fixed reference marker 1-d may be fixed at all times, there is a mechanism for setting the fixed reference marker 1-d at a predetermined position, and the fixed reference marker 1-d is attached every time shooting is performed. There may be a form of collection.

  In step 2 (S-2), in addition to photographing the concrete bridge bottom surface 101 as a subject, aligning a plurality of cameras in the bridge width direction and continuously connecting the captured images of the cameras in the bridge axis direction The fixed reference marker 1-d, a dimension reference marker installed on or near the surface of the concrete bridge bottom surface 101 at the time of photographing, together with an alignment marker for connecting as a continuous image before and after the images photographed by moving the plurality of cameras. Including the entire bottom of the concrete bridge.

  In step 3 (S-3), an image alignment correction value is derived via the image of each alignment marker for the obtained image group of the entire bridge bottom surface, and the image is synthesized, and the fixed reference marker 1-d and dimensions are combined. Image information of the entire bottom surface in the common coordinate system or the real world coordinate system is obtained by coordinate conversion using the image of the reference marker.

  In step 4 (S-4), in order to compare images taken at different times, a comparison area is set from the structure image displayed for the comparison area, and a common coordinate system or By comparing the images in a coordinate system based on the real world coordinate system, a change in deterioration over time can be obtained. By applying this image group and common coordinate system information or real world coordinate system information to automatic loss assessment and degradation specific image processing, it is also possible to automatically perform quantitative detection and degradation assessment of degradation changes. become.

Hereinafter, a flow of obtaining a comparative image by conversion to a common coordinate system or a real world coordinate system separated from each other using the fixed reference marker 1-d and the size reference marker 1-g will be described.
FIG. 3 shows a multi-camera imaging system configuration including imaging of fixed reference markers 1-d1, 1-d2 at the end of the concrete bridge bottom surface 101 of FIG. The multi-camera imaging system 100 includes cameras 2-1 to 2-e, marker applying units 3-1, 3-2,... For adjusting adjacent images by a laser device or the like, and controls connected thereto. A unit (including timing control) 11 and an image processing unit 12. In addition, a dimension reference marker 1-g for viewing the dimension of deterioration such as the crack 20 is attached to a part of the concrete bridge bottom surface 101 which is a subject prior to photographing. In the present embodiment, since the interval between the fixed reference markers 1-d1 and 1-d2 is narrow and the accuracy is not improved for use in measurement as an actual dimension, a dimension reference for temporarily obtaining dimension information in a captured image. An example using the marker 1-g is shown. The dimension reference marker 1-g may be affixed, but may be mounted on a part of the multi-camera imaging system 100 so as to be in contact with the subject, placed near the subject, or projected from the neighborhood to the subject. Further, if the interval between the fixed reference markers 1-d1 and 1-d2 is wide enough to be used for measurement of the actual dimension and the actual world dimension can be obtained, the common coordinates may be applied as they are as the actual world coordinate information. .

  The cameras 2-1 to 2-e are fixed on the multi-camera photographing system 100 extending in the bridge width direction at a predetermined interval on the straight line and are directed toward the bottom surface 101 (upward). -1 to 6-e. Adjacent shooting ranges (for example, shooting range 6-1 and shooting range 6-2) have a common range that overlaps, and the alignment marker 7-a1 is assigned thereto by the marker applying means 3-1. Similarly, in the imaging ranges 6-2 to 6-e, the alignment marker group is assigned by the marker assigning means 3-2 to 3- (e-1). At least two alignment markers 7-a are arranged in a common shooting area (common range) of adjacent cameras so that each camera can take a picture. When a non-visible light source is used for the marker applying means 3 and a camera capable of simultaneous non-visible light / visible light photographing is used for the camera 2, a visible light photographed image in which the alignment marker 7-a is not captured is obtained. Moreover, the visible light is turned on / off as the marker applying means 3, and a correction value for image synthesis is derived from the alignment marker 7-a image at the on time, and the captured image without the alignment marker 7-a image at the off time is obtained. There is also a form that uses image synthesis without the reflection of the alignment marker 7-a image.

  When the multi-camera photographing system 100 finishes photographing with the plurality of cameras 2-1 to 2-e, the multi-camera photographing system 100 moves in the bridge axis direction and photographs the next region. At this time, the alignment marker 7-b before and after the movement is applied to a part of the imaging region 6-1 on the traveling side of the camera 2-1 before and after the movement in the direction of the bridge axis from the movement direction marker applying means 4. The control unit 11 controls the timing of shooting by the multiple cameras 2-1 to 2-e, marker assignment by the marker assigning means 3-1 to 3- (e-1), and marker assignment by the moving direction marker assigning means 4. The control signals and the image data are transmitted / received via the communication line 21, the plural camera marker applying unit control line 22, and the moving direction marker applying unit control line 23. Here, fixed marker illumination controlled via the illumination means control line 24 so that the captured images of the fixed reference markers 1-d1, 1-d2 and the size reference marker 1-g can be clearly identified from the subject image on the bottom surface 101. The means 5 illuminates with infrared light or visible light, and a high-intensity image is extracted and acquired by simple image processing by binary detection, and the photographed data is transmitted to the image processing unit 12 for processing, storage, or output.

  First, the position of the multi-camera imaging system 100 is adjusted so that the fixed reference markers 1-d1 and 1-d2 installed in advance can be imaged by the camera 2-1. Here, the case where the camera position is adjusted first is shown for easy understanding. However, when the fixed reference marker 1-d can be individually specified as 1-d1 and 1-d2 in the captured image, it will be shown later. Since the same function can be obtained by the coordinate conversion process, there is no necessity of designating the camera that captures the fixed reference marker 1-d. Further, it is sufficient to photograph the dimension reference marker 1-g somewhere in the photographing range. In the initial position adjustment of the camera 2-1, the fixed reference markers 1-d1 and 1-d2 are adjusted so as not to overlap with the alignment marker 7-a of the captured image between the other cameras and the alignment marker 7-b before and after the movement. In this case, the same color marker can be taken simultaneously. However, in other cases, it is necessary to control the colors so that they can be identified by separating the colors of the markers, or by assigning the markers at different timings and photographing them.

  From the essential significance of the use of the fixed reference marker 1-d and the size reference marker 1-g, the illumination means 5 is not essential, and the fixed reference marker 1-d and the size are measured from the subject image to be inspected of the bottom surface 101. As long as the reference marker 1-g can be identified, the reference marker 1-g does not have to have a high luminance. In order to detect a marker image more clearly, surely, and simply, an example using a high-intensity marker is shown here. As described above, the fixed reference marker 1-d can be used as a reference mark if there is a feature point that is clearly a common mark at a certain interval, such as a protrusion or a depression, such as a part of the bottom surface 101. is there. As a technique for obtaining a high-intensity marker luminance, the fixed reference marker 1-d or the size reference marker 1-g is irradiated with external light like the fixed marker illuminating means 5 as in the present embodiment, and the marker In some cases, a mirror, a fluorescent material, or a reflector made of a small reflector may be used, but a self-luminous combination of a power-saving LED and a battery is also applicable. However, in order to prolong the life of the battery, it is necessary to devise measures such as using a method for controlling the light emission time by performing power saving communication with the measurement system.

  In order to separate the marker image in the photographed image from the subject visible light image for comparison, the procedure of marker image extraction processing when infrared rays are used is shown.

  FIG. 4 is a diagram showing a state of photographing by the multi-camera photographing system 100 when the photographing region is viewed from directly below the bridge bottom. (A) is the figure which looked at the concrete bridge bottom face 101 of FIG. 1 from just under, and the following (b)-(d) is a series of imaging | photography flow charts.

  FIG. 5 is a shooting flowchart by the multi-camera shooting system, and the flow will be described below together with a series of shooting states shown in FIG.

  In step 10 (S-10), first, as shown in FIG. 4B, an area including the two fixed reference markers 1-d1 and 1-d2 enters the imaging range 6-1 of the camera. In addition, in this embodiment, the multi-camera imaging system 100 is arranged so that the dimension reference marker 1-g falls within the adjacent imaging ranges 6-2 and 6-3.

  In step 11 (S-11), the fixed reference marker illumination applying means 5 is turned on, and the plural inter-camera marker applying means 3-1 to 3- (e) are also applied to the common area of the plural cameras arranged in the bridge width direction. -1) to infrared alignment markers 7-a1-1 to 7-a (e-1) -1 are added, and images are simultaneously captured by a plurality of cameras.

  In step 12 (S-12), the first moving direction alignment marker 7-b1-1 is given before the movement of the multi-camera photographing system 100 in the bridge axis direction, and the broken line frame in FIG. As shown, the image is taken at the same camera position with respect to the reference camera 2-1.

  In step 13 (S-13), as shown by the solid line frame in FIG. 4C, the multi-camera imaging system 100 is moved a predetermined distance in the bridge axis direction, and a plurality of cameras 2- Photographing is performed in a state where infrared alignment markers 7-a1-2 to 7-a (e-1) -2 are added to the common area 1-2.

  In step 14 (S-14), the next moving direction alignment marker 7-b1-2 is given to the reference camera photographing region 6-1-2 as shown by the broken line frame in FIG. Shoot at the same camera position.

  Thereafter, in step 15 (S-15), the multi-camera imaging system 100 is moved by a predetermined distance in the bridge axis direction, and the operation of FIG. 4D is repeated until imaging of the entire target area of the concrete bridge bottom surface 101 is completed.

  FIG. 6 shows a configuration diagram of the entire captured image group including the adjacent image alignment marker image, the fixed reference marker image, and the size reference marker image. A whole photographed image group obtained by repeating a series of photography is shown as (1, 1) to (e, f). In the drawing, images 10-d1 and 10-d2 of fixed reference markers 1-d1 and 1-d2, images 10-g of dimension reference markers 1-g, and alignment markers 7- of images taken from a plurality of cameras arranged in the bridge width direction. Image group 9-a1-1 to 9-a (e-1) -f of a1-1 to 7-a (e-1) -f, alignment marker 7-b1-1 before and after movement in the bridge axis direction The image groups 9-b1-1 to 9-b1-f of ˜7-b1-f are shown. As for the photographed image, the camera photographed image for e vehicles in the bridge width direction spreads in the bridge axis direction over the movement f times from the initial stage. In FIG. 6, for the sake of simplicity, an arbitrary image is (m, n).

  FIG. 7 shows the principle of coordinate conversion from adjacent images to real world coordinates. (A) is the coordinate relationship for the coordinate conversion in the bridge width direction, and (b) is the coordinate relationship for the coordinate conversion in the bridge axis direction. , (C) shows the coordinate conversion for common coordinate conversion with time, and (d) shows the relationship of coordinate conversion to the real world coordinate system. According to FIG. 7, coordinate conversion between images in the bridge width direction, coordinate conversion between images in the bridge axis direction, and conversion of arbitrary images to the coordinates of the reference image 25, conversion to common coordinates separated by time, The conversion to the real world coordinate system will be described. In general, coordinate conversion from the coordinate system of two coordinate systems (u, v) to (a, b) corresponding to movement, rotation, and change in magnification between two coordinates is expressed by the following equation (1). It is represented by the affine transformation formula shown.

橋幅方向に並んだ撮影画像の共通座標化は、図７に於いて、（ａ）の橋幅方向の座標変換に示される撮影画像（ｍ，ｎ）のマーカ像９−ａｍ−ｎの２つの座標(a1，b1)、(a2，b2)と撮影画像（ｍ，ｎ+１）の同一のマーカ像９−ａｍ−ｎの２つのマーカ像の座標(u1，v1) ，(u2，v2)を使って、以下の式（２）に示すように、上記式（１）中の各係数が導かれる。

The common coordinates of the captured images arranged in the bridge width direction are represented by 2 in the marker image 9-am-n of the captured image (m, n) shown in FIG. The coordinates (u1, v1), (u2, v2) of two marker images of the same marker image 9-am-n of the two coordinates (a1, b1), (a2, b2) and the photographed image (m, n + 1) ) Is used to derive each coefficient in the above equation (1) as shown in the following equation (2).

以上から、これらの係数を使って、撮影画像（ｍ，ｎ＋１）の座標値（ｕ，ｖ）の撮影画像（ｍ，ｎ）の座標系（ａ，ｂ）による表示は各係数により、 以下の式（３）で表される。

From the above, using these coefficients, the coordinate values (u, v) of the photographed image (m, n + 1) are displayed in the coordinate system (a, b) of the photographed image (m, n) depending on each coefficient. It is represented by Formula (3).

このアフィン変換行列をＲｗｍｎとすると、以下の式（４）となる。

When this affine transformation matrix is Rwmn, the following equation (4) is obtained.

橋軸方向での移動前後の撮影画像の共通座標化は、図７の（ｂ）の橋軸方向の座標変換に示される撮影画像（１，ｎ）と撮影画像（１，ｎ＋１）のマーカ像９−ｂ１−ｎの２つの座標 (a1，b1)，(a2，b2)と座標(u1，v1)，(u2，v2)で前記と同様にアフィン変換係数を求めることができ、以下の式（５）で表される。

The common coordinates of the captured images before and after movement in the bridge axis direction are the marker images of the captured image (1, n) and the captured image (1, n + 1) shown in the coordinate conversion in the bridge axis direction of FIG. 9-b1-n The two coordinates (a1, b1), (a2, b2) and the coordinates (u1, v1), (u2, v2) can be used to determine the affine transformation coefficients as described above. It is represented by (5).

以上から、これらの係数を使って、撮影画像（１，ｎ＋１）の座標値（ｕ，ｖ）の撮影画像（１，ｎ）の座標系（ａ，ｂ）による表示は各係数により、以下の式（６）で表される。

From the above, using these coefficients, display of the coordinate value (u, v) of the captured image (1, n + 1) by the coordinate system (a, b) of the captured image (1, n) depends on each coefficient, and It is represented by Formula (6).

このアフィン変換行列をＲａ１ｎとすると、以下の式（７）となる。

When this affine transformation matrix is Ra1n, the following equation (7) is obtained.

任意の撮影画像（ｍ，ｎ）の座標（ｕ，ｖ）を基準撮影画像（１，１）の座標系（ａ，ｂ）でアフィン変換行列を用いて表すと、以下の式（８）となる。

When the coordinates (u, v) of an arbitrary captured image (m, n) are expressed using the affine transformation matrix in the coordinate system (a, b) of the reference captured image (1, 1), the following equation (8) Become.

図７の（ｃ）の基本撮影画像にある二つの固定基準マーカ像の時を隔てた共通座標（x1，y1），(x2，y2）と基準撮影画像の座標系に統一された共通座標系での値（a1，b1），（a2，b2）の座標を用いて、アフィン変換係数を求めることができ、以下の式（９）で表される。

Common coordinate system (x1, y1), (x2, y2) and the coordinate system of the reference photographic image separated from each other by the time of two fixed reference marker images in the basic photographic image of (c) in FIG. The affine transformation coefficient can be obtained using the coordinates of the values (a1, b1) and (a2, b2) at, and is expressed by the following equation (9).

このアフィン変換行列をＲｒ１１とすると、以下の式（１０）となり、

If this affine transformation matrix is Rr11, the following equation (10) is obtained.

基本画像座標系（ａ，ｂ，１）と時を隔てた共通座標系（ｘ，ｙ，１）の関係は、以下の式（１１）となる。

The relationship between the basic image coordinate system (a, b, 1) and the common coordinate system (x, y, 1) separated by time is expressed by the following equation (11).

これより、任意の撮影画像（ｍ，ｎ）の画像の座標からの時を隔てた共通座標表示変換は、以下の式（１２）となる。

Thus, the common coordinate display conversion separated from the coordinates of the image of an arbitrary captured image (m, n) is expressed by the following equation (12).

得られた共通座標系（ｘ，ｙ）で、寸法基準マーカ像１０−ｇの両端の座標を（x1，y1），（x2，y2）とする。寸法基準マーカ像の１０−ｇの長さＤｘｙは、以下の式（１３）で与えられる。

In the obtained common coordinate system (x, y), the coordinates of both ends of the dimension reference marker image 10-g are (x1, y1), (x2, y2). The 10-g length Dxy of the dimension reference marker image is given by the following equation (13).

この寸法基準マーカ１−ｇの実寸法をＤｒとすると、実世界座標系と共通座標系の比率が、Ｒ＝Ｄｒ／Ｄｘｙと表されるので、時を隔てた共通の実世界座標系（Ｘ，Ｙ）は、以下の式（１４）で表される。

When the actual dimension of the dimension reference marker 1-g is Dr, the ratio between the real world coordinate system and the common coordinate system is expressed as R = Dr / Dxy. Therefore, the common real world coordinate system (X , Y) is expressed by the following equation (14).

寸法基準マーカ１−ｇの長さは撮影範囲よりもある程度長い方が画素解像度による誤差の影響を抑えることができるので望ましい。このため、本実施形態では固定基準マーカ１−ｄとは別に長い寸法基準マーカ１−ｇを別途設定している。

It is desirable that the length of the dimension reference marker 1-g is somewhat longer than the imaging range because the influence of errors due to pixel resolution can be suppressed. For this reason, in this embodiment, a long dimension reference marker 1-g is set separately from the fixed reference marker 1-d.

  FIG. 8 shows a coordinate conversion processing block for common coordinate display or real world coordinate display of the image processing unit 12 in FIG. In FIG. 8, a captured image signal 80 is received by an image signal receiving unit 81, a markerless image extracting unit 82 for a visible light image, and a reference camera for processing a marker image from an invisible light image signal. This is transmitted to the moving photographed image group extracting unit 83, the multiple camera simultaneous photographed image group extracting unit 85, the fixed marker image detecting unit 88, and the dimension reference marker image detecting unit 93, respectively.

  FIG. 9 shows a flowchart for converting the pixel coordinates of each marker image of the photographed image group into a common coordinate display or a real world coordinate display via the alignment marker. Hereinafter, the flow of converting to the coordinates of the reference image 25 by connecting the coordinate conversion using the alignment marker between the adjacent images in the bridge width direction and the bridge axis direction will be described.

  In step 20 (S-20), the image signal receiving unit 81 assigns an identification number (image ID) for specifying the reference image 25.

  In step 21 (S-21), an adjacent image is selected from the series of (1,1) to (1, f) images selected by the reference camera movement photographed image group extraction unit 83 in the bridge axis direction in FIG. The shared marker image to be shared is specified, and the coordinates of the marker are acquired.

  In step 22 (S-22), the moving image group coordinate conversion value processing unit 84 derives a correction value (affine conversion coefficient) Ra1n from the coordinate value of the alignment marker between the previous image and the pixel image of the acquired image. To do.

  In step 23 (S-23), in the reference image coordinate conversion processing unit 87, the image of (1, 1) becomes the reference image 25 in this embodiment, but the movement from the moving image group coordinate conversion processing unit 84 is performed. From the direction correction value and the coefficient Ra1n group, the coordinates of the image group connected through the alignment marker in the moving direction are converted into the coordinate system of the reference image.

  In step 24 (S-24), the multiple camera simultaneous photographed image group extracting unit 85 in the bridge width direction shares the adjacent images from the selected series of (1, n) to (e, n). The alignment marker image is identified and the marker coordinates are acquired.

  In step 25 (S-25), the inter-camera image group coordinate conversion value processing unit 86 compares the coordinate values of the markers in the photographed image, and performs a process in accordance with FIG. A coefficient Rwmn of the transformation matrix is derived, and a correction value (affine transformation coefficient) is obtained from the coordinate value of the alignment marker between adjacent images based on the coordinate system of the reference camera moving photographed image group (1, n) in the bridge width direction. Is calculated.

  In step 26 (S-26), the reference image coordinate conversion process is performed by combining the coordinate conversion value from the moving image group coordinate conversion value processing unit 84 and the coordinate conversion value from the inter-camera image group coordinate conversion value processing unit 86. In the unit 87, the conversion coordinate value in the bridge axis direction and the conversion coordinate value corresponding to the position of the image in the bridge width direction are combined to convert the image of an arbitrary captured image (m, n) into the coordinate system of the reference image 25. Deriving a value.

  In step 27 (S-27), the common coordinate value conversion processing unit 89 obtains the image ID and its coordinate value obtained by photographing the fixed reference marker 1-d from the common fixed reference marker image detection unit 88, and Further, from the fixed reference marker common coordinate input 90, a value that becomes the same coordinate value is obtained at intervals, and the coordinate value according to the basic image 23 from the reference image coordinate conversion processing unit 89 is obtained as shown in FIG. To perform common coordinates.

  In step 28 (S-28), in the real world coordinate value conversion processing unit 94, the size reference marker 1-g is detected from the size reference marker image detection unit 93 in the common coordinates from the common coordinate value conversion processing unit 89. 7 is acquired and converted into a coordinate value in the common coordinate system, and the actual dimension is acquired from the actual dimension information input 95 of the dimension reference marker 1-g. ) To obtain a conversion value to real world coordinates. The markerless image composition processing unit 96 obtains real world coordinate image information 97 from the image ID and the real world coordinate conversion value.

  In step 29 (S-29) and step 30 (S-30), the coordinate value of an arbitrary image is converted to the coordinate value of the reference image 25 by repeating up to all images in the bridge width and bridge axis directions, and further Are converted into common coordinates separated from each other, and finally, real world coordinate conversion processing is performed for all images, and real world coordinate image information output 97 is performed.

  In FIG. 9, the flow of converting into the real world coordinate value based on the dimension information through the conversion to the common coordinate value has been described. However, as shown by the broken line in FIG. 8, the fixed reference marker 1-d and the dimension reference By simultaneously providing the information divided into the markers 1-g as the real world coordinate value 98 of the fixed reference marker 1-d, it is possible to convert from the reference image coordinate system to the real world coordinate system at a stretch, and the processing is reduced. . Further, if actual size information is not necessary and only comparison of captured images at time intervals is performed, the image is output directly from the common coordinate conversion value processing unit 96 of FIG. 92 is also possible.

  FIG. 10 is a block diagram for comparing information between images taken at regular intervals between regular inspections, and FIG. 11 is a flowchart thereof. The image processing unit 12 in FIG. 3 considers that an image taken on site can be compared with an image taken on the spot at the time of past periodic inspection, and this function is a part thereof.

  In step 40 (S-40), as the display image condition input 200, the inspection time to be compared and the captured image position and range to be displayed are input. This information may be acquired from the operation of the display image information on the operation screen. In the image control unit 201, the common / real world coordinate system image composition processing unit 99 and the image composition processing unit 203 which are newly acquired as the common coordinate or real world coordinate information range and displayed as display range information are transmitted. Further, the image composition processing unit 201 sends display condition information to the display image processing unit 204.

  In step 41 (S-41), the image composition processing unit 203 calls the image at the shooting time from the image data stored in the past captured image information storage unit 202, and starts from the designated range value to be displayed. Then, the inverse affine transformation is performed to specify the captured image in the range, and the image in the related range is selected.

  In step 42 (S-42), the newly acquired common / real world coordinate system image composition processing unit 99 attaches conversion information to the image data corresponding to the designated range to be displayed and sends it to the image composition processing unit 203. Output.

  In step 43 (S-43), the image data is converted by using the affine transformation coefficient to the common coordinates or the real world coordinates given to the photographed image by the image composition processing unit 203, and the images to be compared are grouped. On the other hand, the stored image information of the image range desired to be displayed and the new captured image information are combined and output to the display unit processing unit 204.

In step 44 (S-44), the display unit processing unit 204 outputs display mode information as a display image signal 205 in accordance with the display condition information obtained from the image control unit 201. As described above, when comparing images taken at regular intervals such as periodic inspections, images are automatically compared at fixed intervals using the fixed reference marker as a reference for common coordinates or real world coordinates. Easy to display. In addition, since real-world coordinate information can be inserted into an image and applied to measurement of a crack size or the like, deterioration with time can be quantitatively performed by image comparison. Although not mentioned in the above flow, the processing information from the new image signal in FIG. 10 is transmitted to the captured image information storage unit 202 together with the image data and becomes future comparison information.
(Effects of the first embodiment)
In the first embodiment, in a configuration in which a plurality of images are combined between adjacent images and combined via a marker, at least two fixed reference markers on the subject or a reference mark that is a common feature point with time is provided. By setting and giving common coordinate values to them over time, the coordinate positions, orientations and dimensions of multiple images over time can be automatically matched, so the same location for each periodic inspection Comparison of surface states including deterioration can be easily performed.

  In addition, by including a dimension reference marker that represents the actual size in the photographing range, it is automatically converted to the real world coordinate system, so that it is possible to easily compare on the basis of the actual size of deterioration such as cracks at the same place.

  In addition, by setting at least two reference landmarks that are common over time and giving them the same real world coordinates over time, an automatic common reference for comparison of the same location degradation on an actual scale basis. Real-world coordinate derivation can be done more easily.

In the above-described embodiment, in particular, the reference mark such as the fixed reference marker or the dimension reference marker has a higher luminance than other subjects as in the case of the alignment marker of the inter-camera image and the image alignment marker before and after the movement. By detection, it is possible to easily identify a marker for coordinate conversion for comparison between images separated by time, so that it is not necessary to perform calculation processing of luminance distribution comparison of images, and the processing can be reduced in weight and speed.
(Second Embodiment)
FIG. 12 is a diagram in which a plurality of fixed reference markers 1-d1 to 1-d8 are installed in the bridge width direction and the bridge axis direction of the concrete bridge bottom surface 101 in the second embodiment. In the first embodiment, two fixed reference markers are installed close to each other at the end of the bridge. However, in this embodiment, two pairs of fixed reference markers (for example, 1- 1) arranged apart in the bridge width direction are used. A plurality of sets d1 and 1-d2) are installed in the direction of the bridge axis (in FIG. 12, four sets up to a pair of 1-d7 and 1-d8 are installed). This configuration will be described while paying attention to the accumulation of errors caused by the image composition processing when the bridge width becomes wide or the bridge shaft becomes long.

  FIG. 13 is a diagram showing a state of photographing by the multi-camera photographing system 100 that photographs a plurality of fixed reference markers installed on the concrete bridge bottom surface 101. In the multi-camera photographing system 100 of FIG. 13, compared to the multi-camera photographing system 100 of FIG. 3, fixed reference marker irradiating means 5-1 and 5-2 and moving direction marker applying means 4-1 and 4-2 are multi-cameras. The difference is that the photographing system 100 is provided at both ends of the bridge width direction. Other configurations are the same as those in FIG. The fixed reference marker group is illuminated by fixed reference marker illumination means 5-1 and 5-2. The control of the marker applying means 3-1 to 3- (e-1) and the moving direction marker providing means 4-1 and 4-2 is always turned on using infrared rays, and the infrared / visible light camera 2 is a concrete object. The visible light image of the bridge bottom surface 101 and the infrared marker image are simultaneously photographed to separate the marker image and the visible light image. The position of the multi-camera imaging system 100 is included in a captured image in which any one of the plurality of fixed reference markers 1-d1 to 1-dN includes any one of the movement direction markers 7-bn and 7-cn. Set to. In this way, if the same captured image includes the moving direction marker image and the fixed reference marker image, the process for coordinate conversion is simplified and the image alignment error can be suppressed to a low level. Even if the positions of the images are different, the effect of suppressing the accumulation of errors can be obtained, and the image processing is somewhat complicated, but there is no major problem. As a technique for avoiding the overlapping of the markers, the colors of the markers can be divided, or the markers can be divided at the timing of image capturing. Hereinafter, in this embodiment, for the sake of simplicity, both markers will be described in the same captured image.

  FIG. 14 shows accumulation of errors in coordinate conversion between adjacent images when the fixed reference markers 1-d1 and 1-d2 of the first embodiment are arranged close to each other in one reference image 25. . (A) includes a common coordinate system or real world coordinate system obtained by performing coordinate conversion via the original common coordinate system or real world coordinate system used as the reference coordinate of the fixed marker and the alignment marker of each image. Represents the maximum error accumulation. The reference image 25 includes two fixed reference marker images 10-d1 and 10-d2, and naturally the original common coordinates or real world coordinates substantially coincide with each other. In the bridge width direction, coordinate conversion is performed by solving affine transformation from the alignment marker image between adjacent images. At this time, since the image is made into particles by the pixels, a particleization error between adjacent images inevitably occurs. In addition, errors due to the effects of optical system aberrations also occur. As a result, repeated coordinate transformation between images results in error accumulation and error accumulation from common or real world coordinates. As a result, as the distance from the reference image 25 including the fixed reference marker image increases, the error accumulation increases monotonously. Similarly, in the bridge axis direction, there is accumulation of errors in coordinate conversion between adjacent images, and errors accumulate as the distance from the reference image 25 including the fixed reference marker images 10-d1 and 10-d2 increases. In the direction of the bridge axis, if the bridge is long, the number of repetitions of photographing becomes large, and this influence becomes more remarkable. For example, when a 30 m long bridge is taken with a high-definition camera (1920 × 1080 pixels) at intervals of 10 cm, the accumulation of 300 times is counted from the number of coordinate transformations, and the number of vertical operation lines of the high-definition image Even in the image of 1080 pixels, the accumulation of pixel particleization errors may cause a shift of up to 1/3 of the image, and the alignment accuracy for image comparison over time is large. It will get worse. The accumulation of this error is statistically proportional to the square root of the number of iterations according to the binomial theorem, but it still causes a misalignment of 51 times, resulting in a certain error. It is necessary to carry out a strict worst-case design, and in this case, statistical application is not possible, so it is necessary to deal with the accumulation of about 300 times. Further, in the bridge width direction, for example, 10 m is photographed at intervals of about 30 cm, so that the accumulation is about 33 times. (B) is a schematic linear approximation (simplified linear display) obtained by simplifying the accumulation of errors for each image, and this linear simplified display will be described below.

  FIG. 15 is a diagram for explaining the effect (linear simple display of error accumulation) of a plurality of fixed reference markers according to the second embodiment. When a plurality of fixed reference markers 1-d1 to 1-d8 are arranged, (a) shows a range surrounded by four fixed reference markers corresponding to one interval of the fixed reference markers in the bridge axis direction. (B) shows the accumulation of errors when discretely arranged by alternately skipping one marker arrangement in the bridge width direction as a modified type. In the case of (a), the accumulation of errors becomes maximum at an intermediate position between the fixed reference markers. When this is applied to the general fixed reference marker quantity N, the cumulative value of the error is reduced and improved in proportion to 1 / (2 * (N-1)) with respect to the total length of the bridge axis. You can see that For example, if six points are arranged at 5 m intervals with respect to the 30 m bridge shaft scale in the previous example, the error is reduced to a 2.5 m long scale, and the cumulative number is reduced from 300 times to 25 times. Substantial statistics are equivalent to five accumulations, and the image correction processing corresponding to the accumulation error is greatly reduced. In the case of (b), although it depends on the distance between the fixed reference marker in the bridge width direction and the bridge axis direction, the bridge axis direction is basically long, so that the installation work of the fixed reference marker is reduced. Since the accumulation of errors in the direction of the bridge axis becomes dominant in order to widen the interval, it is applied in the direction of the bridge axis.

  FIG. 16 shows a flowchart of conversion to common coordinates or real world coordinates when the plurality of fixed reference markers 1-d1 to 1-d8 shown in FIG. 13 are installed.

  In step 50 (S-50), a series of photographed image data is acquired in the image processing unit 12.

  In step 51 (S-51), an image including a fixed reference marker group is specified from the photographed image group.

  In step 52 (S-52), the image ID of the image group including the fixed reference marker image and the number of coordinate transformation combinations between adjacent images of the bridge axis and the bridge width are obtained, and the two nearest fixed reference marker images are obtained. A route to be applied to an arbitrary image and common coordinate transformation or real world coordinate transformation is set so that the number of combinations of the transformation numbers by the alignment marker up to the image including the image is reduced.

  In step 53 (S-53), for each image, the affine transformation value from the alignment marker coordinates of the images taken by the plurality of cameras in the bridge width direction on the route and the alignment marker coordinates before and after movement in the bridge axis direction are obtained. By combining the affine transformation values, the transformation is performed using the common coordinate values or the real world coordinate values of the two nearest fixed reference markers, and the coordinate transformation related information is derived.

  In step 54 (S-54), information related to the derivation of common coordinates and real world coordinates for each image is recorded as image metadata together with the photographed image data and stored.

  In the flowchart of FIG. 16, step 52 (S-52) has been shown to select an image including the nearest fixed reference marker, and this will be described using the entire captured image group of FIG. The coordinate transformation route from the position of the arbitrary value (m, n) to the image including the fixed reference marker 1-d1 is the bridge of the coordinate transformation product in the bridge width direction along the product of the above formula (14). Multiply by the product of the axial coordinate transformation. At this time, as the route distance, m−1 times from (m, n) to (1, n) and n−1 times from (1, n) to (1,1) are added. + | N-1 |, which corresponds to the accumulation of errors. In FIG. 13, there are two routes along the bridge axis direction along (1, n) and (e, n). There are two routes in the bridge width direction: conversion toward (1, n) and conversion toward (e, n). For example, if fixed reference markers are detected at four points (1, 1), (e, 3), (f, 1), and (e, m), the total route distance is | m−1 | + | N-1 |, | e−m | + | n−3 |, | m−1 | + | f−n |, | e−m | + | f−n |. From these four points, the respective route distances are compared, two of the smaller values are selected, and their IDs are selected for conversion to common coordinates or real world coordinates.

  In the above, since the fixed reference markers are arranged at both ends of the bridge width, the route setting is simple, so it is desired to easily set the route length to four fixed reference marker images to be short. There is a marker, and there may be multiple routes to get there. In the multi-camera photographing system 100, there are about two adjacent image alignment routes in the bridge axis direction, and the route configuration basically has a vertical and horizontal structure of the bridge axis and the bridge width. At this time, there are two routes from an arbitrary image position (m, n) to the fixed reference marker image at (i, j). The distance of the route portion in the bridge axis direction in both routes is unique | n−j |. Now, if the two points in the opposite direction from the image (m, n) in the bridge axis direction overlap the route in the bridge axis direction are (k, n) and (l, n), The distance between the image (m, n) and the fixed reference marker image (i, j) is | n−j | + | m−k | + | i−k | and | n−j | + | l−m. | + | L−i |. The shorter of these two is selected. If both are the same, for example, some route is weighted, and the right route may be selected. In this way, a short route to include an arbitrary image position and each fixed reference marker is selected, and each distance from each image to a plurality of fixed reference markers as shown in the previous stage. The two nearest routes are selected by comparison.

  The related information acquired for each image includes the image ID including the two nearest fixed reference marker images of the image, the route information to the image including the nearest fixed reference marker image, the affine transformation coefficient between images obtained from the route, the common Coefficient or coefficient of affine transformation to real world coordinates, etc. When using this image in common coordinates or real world coordinate display, coordinate transformation is performed using the metadata, and the whole image separated by time It can be used for comparison processing with.

  Regarding the route setting of the image including the two nearest fixed reference marker images for each image, an image ID including a plurality of fixed reference marker images may be acquired from all acquired images to select the optimum route. However, the optimal route may be calculated from the image information group acquired during shooting each time and applied to real-time processing, or the shooting route may be set in advance and the optimal route set. Good.

In this embodiment, the structure is described as the bridge axis and the bridge width on the bottom surface of the concrete bridge. However, in a combination of high-definition multiple captured images in a wide area such as a general structure, a vertical and horizontal captured image is a marker for alignment between images. This method can be applied to the case where reference marks such as fixed reference markers and subject feature points are set in the case of compositing via the.
(Effect of 2nd Embodiment)
In the second embodiment, the number of reference marks, for example, fixed reference markers and feature points given to the real object, for sharing the coordinates of the object to be photographed at a common position with respect to the subject at intervals is set to N. By doubling and using this as common coordinates or real world coordinates, the accumulation of errors caused by coordinate conversion processing by image alignment markers between adjacent images during shooting is reduced to 1 / (2 * (N-1)) Therefore, the accuracy of image alignment for comparison of captured images over time is increased. In particular, the effect becomes remarkable in a long scale structure.

In addition, an image including a plurality of reference landmarks from an arbitrary image is derived by calculating a route distance while considering a weight of an error in image alignment with a route of coordinate conversion by an image alignment marker between adjacent images, and the image and the plurality of those images Compare the route distance to the reference mark, select the image and route that include the nearest two reference marks from among them, and use them for the coordinate value of the image alignment, so that the cumulative error is reduced most and the error accuracy is Can be high.
(Third embodiment)
As a third embodiment, a fixed marker illuminating means is provided by installing a convex hemispherical mirror 30 on the concrete bridge bottom surface 101 as an increase in luminance of the fixed reference marker 1-d of FIG. 3 or FIG. 13 in the first and second embodiments. The case where 5 is used in combination will be described. FIG. 17 is a configuration diagram when a virtual image of a light source generated by reflection of a convex hemispherical mirror is used as a fixed reference marker. In actual photographing, the camera 2 and the fixed marker illuminating means 5 are not arranged just below the convex hemispherical mirror, but the basic reflection configuration does not change from the spherical symmetry even if the positional relationship is deviated. In order to explain the basic concept, it is assumed that the configuration is directly below.

  Light rays from the light emitting region 51 of the fixed marker illumination means 5 are reflected by the surface of the convex hemispherical mirror 30 and enter the camera 2. At this time, the dimension of the light emitting region 51 is C, the distance between the multi-camera imaging system 100 and the convex hemispherical mirror 30 is L, and the radius r of the convex hemispherical mirror 30 is assumed. The size of the virtual image 31 by the convex hemispherical mirror 30 in the light emitting area 51 is obtained from the relationship between the distance from the reflection point 34 to the virtual image in the reflection of the convex hemispherical mirror 30. In the reflecting optical system of the convex hemispherical mirror 30, half the radius r / 2 is the focal length. Therefore, when the distance from the reflection point 34 to the virtual image 31 is Q, it is expressed by the following equation (15).

反射点３４から虚像３１までの距離Ｑは、以下の式（１６）となるので、

Since the distance Q from the reflection point 34 to the virtual image 31 is expressed by the following equation (16),

虚像３１の寸法ΔＣは、反射点３４から実像までの距離Ｌと虚像までの距離Ｑの比をとり、以下の式（１７）となる。

The dimension ΔC of the virtual image 31 is a ratio of the distance L from the reflection point 34 to the real image and the distance Q to the virtual image, and is expressed by the following equation (17).

図１８に、凸半球面鏡を固定基準マーカとして用いるときの光源領域の反射像寸法を算出したグラフを示す。発光領域５１の幅Ｃを１０ｍｍとしたときの、虚像３１の寸法の凸半球面鏡３０の半径ｒ依存性を示している。rを５ｍｍ，Ｌを６９０ｍｍとすると、ΔＣ＝０．０３６ｍｍとなる。０．２ｍｍのコンクリートのひび割れを検出すためのカメラの解像度で撮影するに際して、この値は十分に小径な固定基準マーカ１−ｄの寸法を与えることが分かる。理論上の数値は非常に小径となるが、凸半球面鏡３０の表面の歪み，光学系の曇り，研磨時の傷などで実際には多少広がるが、十分に最小ひび割れ以下の寸法の輝点が得られる。仮に、輝点の寸法を広げたい場合には、照明手段５の径を拡げたり、凸半球面鏡３０の半径ｒの径を大きくしたり、あるいは、凸半球面鏡３０の表面を粗くすることにより広げることができる。カメラの画素検出の解像度を同程度に設定するのでこの値は、画素の検出レベルよりも小さなサイズにもなりえる。

FIG. 18 shows a graph in which the reflected image size of the light source region is calculated when the convex hemispherical mirror is used as a fixed reference marker. The graph shows the dependence of the dimensions of the virtual image 31 on the radius r of the convex hemispherical mirror 30 when the width C of the light emitting region 51 is 10 mm. When r is 5 mm and L is 690 mm, ΔC = 0.036 mm. It can be seen that this value gives the dimension of the fixed reference marker 1-d that is sufficiently small in diameter when shooting at the resolution of the camera to detect cracks in the 0.2 mm concrete. Although the theoretical numerical value is very small, it actually spreads somewhat due to distortion of the surface of the convex hemispherical mirror 30, fogging of the optical system, scratches at the time of polishing, etc., but the bright spot with a dimension below the minimum crack is sufficiently large can get. If it is desired to increase the size of the bright spot, the diameter of the illumination means 5 is increased, the radius r of the convex hemispherical mirror 30 is increased, or the surface of the convex hemispherical mirror 30 is roughened. be able to. Since the resolution of pixel detection of the camera is set to the same level, this value can be smaller than the detection level of the pixel.

  Next, the power of light incident on the camera 2 will be described. It can be captured as a light beam Pi supplemented by the solid angle ΔSr stretched on the light emitting region 51 via the convex hemispherical mirror 30 by the lens of the camera 2. When the light receiving radius D at the lens position of the camera 2 and the luminous flux emitted from the light emitting area 51 of the fixed marker illuminating means 5 are Po, the following equation (18) is obtained.

図１９には、凸半球面鏡を固定基準マーカとして用いるときの反射光強度と被写体からの反射光を面状の乱反射モデルとしたときの強度の算出図を示す。固定マーカ照明手段５として１００Ｗの電熱球相当を用いたときのカメラ２への凸半球面鏡３０からの反射光束密度と、周囲の被写体からの反射光束密度を示したもので、カメラ２のレンズの受光半径Ｄを２ｍｍとしている。図１８から、周囲の被写体からの反射に対して、３０ｄＢ程度強い輝度差をもつことが分かる。ここでは、凸半球面鏡について説明したが、凹球面鏡についても上記の式（１５）の係数を変えて、以下の式（１９）を適用することにより、同様な効果が得られる。

FIG. 19 shows a calculation diagram of the intensity of the reflected light when the convex hemispherical mirror is used as a fixed reference marker and the intensity when the reflected light from the subject is a planar irregular reflection model. This shows the reflected light flux density from the convex hemispherical mirror 30 to the camera 2 and the reflected light flux density from the surrounding subject when a 100 W electrothermal bulb equivalent is used as the fixed marker illumination means 5. The light receiving radius D is 2 mm. From FIG. 18, it can be seen that there is a strong luminance difference of about 30 dB with respect to reflection from surrounding subjects. Although the convex hemispherical mirror has been described here, the same effect can be obtained for the concave spherical mirror by changing the coefficient of the above equation (15) and applying the following equation (19).

凸半球面鏡３０を利用する場合には、発光領域５１とカメラ２の位置関係により、その反射虚像３１の位置が凸半球面鏡３０の中心３２からずれて撮影されることになる。図２０に、凸半球面鏡に反射する発光領域の虚像３１の球面中心からのずれ量の説明図を示す。図２０の（ａ）には虚像３１の被写体表面の射影３３の球面鏡の中心位置３２からの位置ずれの様子を示し、図２０（ｂ）には、撮影画像における、固定基準マーカ１−ｄ像の撮影画像内位置から、位置ずれ量を導出して補正するための画像条件を示す。図２０（ａ）では凸半球面鏡３０の真下位置からからカメラ２と固定マーカ照明手段５がＤｓ分ずれて、両者の凸半球面鏡３０の反射点３４が鉛直線に対してθ程傾いた位置に配置されている。このとき、凸半球面鏡３０の球の中心３２と反射点３４を結ぶ線分５２とすると、線分５２上での反射点３４から実像までの距離はＳ＝Ｌ／ｃｏｓ（θ），線分５２上での発光領域５１の反射点３４からの発光領域５１までの距離はＳｃ＝Ｓ−ｄ／２＊ｓｉｎ（θ），線分５２上での反射点３４からの発光領域５１の虚像３１までの距離はＳｃ‘＝ｒ＊Ｓｃ／（２＊Ｓｃ＋ｒ），虚像３２の線分５２からの高さがΔｈ＝ｄ／２＊ｃｏｓ（θ）＊Ｓｃ’／Ｓｃとなるので、カメラ２から見た、コンクリート橋の底面１０１上での虚像３１の射影像の中心３２からのずれΔｄは、以下の式（２０）となる。ここで、θ＝ｔａｎ^-1（Ｄｓ／（Ｌ＋ｒ））である。

When the convex hemispherical mirror 30 is used, the position of the reflected virtual image 31 is shifted from the center 32 of the convex hemispherical mirror 30 due to the positional relationship between the light emitting region 51 and the camera 2. FIG. 20 is an explanatory diagram of the amount of deviation from the center of the spherical surface of the virtual image 31 in the light emitting area reflected by the convex hemispherical mirror. FIG. 20A shows the state of the projection 33 of the object surface of the virtual image 31 displaced from the center position 32 of the spherical mirror, and FIG. 20B shows the fixed reference marker 1-d image in the photographed image. The image conditions for deriving and correcting the amount of positional deviation from the captured image position are shown. In FIG. 20A, the camera 2 and the fixed marker illuminating means 5 are deviated by Ds from the position directly below the convex hemispherical mirror 30, and the reflection point 34 of both convex hemispherical mirrors 30 is inclined by about θ with respect to the vertical line. Is arranged. At this time, assuming that a line segment 52 connecting the center 32 of the sphere of the convex hemispherical mirror 30 and the reflection point 34, the distance from the reflection point 34 to the real image on the line segment 52 is S = L / cos (θ), line segment. The distance from the reflecting point 34 of the light emitting region 51 on the light emitting region 51 to the light emitting region 51 is Sc = S−d / 2 * sin (θ), and the virtual image 31 of the light emitting region 51 from the reflecting point 34 on the line segment 52 is shown. Sc ′ = r * Sc / (2 * Sc + r), and the height of the virtual image 32 from the line segment 52 is Δh = d / 2 * cos (θ) * Sc ′ / Sc. The seen deviation Δd from the center 32 of the projected image of the virtual image 31 on the bottom surface 101 of the concrete bridge is expressed by the following equation (20). Here, θ = tan ⁻¹ (Ds / (L + r)).

図２０（ｂ）には、カメラ２で撮影した画像が示されている。この画像の横軸方向に固定マーカ照明手段５の発光領域５１が配されている。また、撮影領域の幅Ｗは、被写体までの距離：Ｌ＋ｒと撮影倍率により求まる。画像内発光領域５１の凸球面鏡３０の反射虚像３１までの、カメラ２と発光領域５１を結ぶ方向６３の射影の中心２１からの画素距離をｄｓとする。画素座標系での撮影範囲の幅をｗとすると、以下の式（２１）となる。

FIG. 20B shows an image photographed by the camera 2. The light emitting area 51 of the fixed marker illumination means 5 is arranged in the horizontal axis direction of this image. Further, the width W of the shooting area is obtained from the distance to the subject: L + r and the shooting magnification. Let ds be the pixel distance from the projection center 21 in the direction 63 connecting the camera 2 and the light emitting region 51 to the reflected virtual image 31 of the convex spherical mirror 30 in the light emitting region 51 in the image. When the width of the imaging range in the pixel coordinate system is w, the following equation (21) is obtained.

このＤｓを用いて、以下の式（２２）にてθを求めることで、

By using this Ds and obtaining θ in the following equation (22),

射影のずれ量Δｄは、以下の式（２３）と表せる。

The projection deviation amount Δd can be expressed by the following equation (23).

図２１には、マルチカメラ装置と固定基準マーカまでの距離をパラメータとしたときの、球面鏡を固定基準マーカとして用いるときの発光領域の虚像のずれ量のカメラと光源間隔依存性を示す。虚像３１の射影３２と凸球面鏡３０の中心３２からのずれΔｄのカメラ２と発光領域５１の凸半球面鏡３０の鉛直方向からのずれＤｓ依存性を示すが、Ｄｓ依存性はほとんどない。

FIG. 21 shows the dependency of the amount of deviation of the virtual image in the light emitting area on the camera and the light source when the spherical mirror is used as the fixed reference marker when the distance between the multi-camera apparatus and the fixed reference marker is used as a parameter. Although the deviation Δd from the projection 32 of the virtual image 31 and the center 32 of the convex spherical mirror 30 is shifted Ds from the vertical direction of the camera 2 and the convex hemispherical mirror 30 of the light emitting area 51, there is almost no Ds dependency.

  FIG. 22 is a diagram showing the distance dependency between the camera and the light source of the amount of deviation of the virtual image in the light emitting area when the spherical mirror is used as a fixed reference marker when the radius of the spherical mirror is used as a parameter. When the distance d is 0 mm, that is, when the camera 2 and the light emitting area 51 are at the same distance from the convex hemispherical mirror 30 and θ is 0, it means that there is no deviation. On the other hand, if the distance is away from the convex hemispherical mirror 30, for example, L = 690 mm, r = 10 mm, and Ds = ds * W / w, so that w = 1920 image pixels and W = 384 mm. In some cases, Ds = ds * 0.2 mm, and if d = 100 mm, the following equation (24) is obtained and a large deviation occurs. When the distance d is about 100 mm, the amount of displacement is considerably larger than the resolution of 0.2 mm. In such a case, it is applied to correct the deviation amount from the deviation amount deriving method.

図２３に、球面鏡を用いたときに、その発光領域の像のずれ量から、固定基準マーカの座標を球面鏡の中心位置に座標を補正するフローチャートを示す。

FIG. 23 shows a flowchart for correcting the coordinates of the fixed reference marker to the center position of the spherical mirror from the amount of deviation of the image of the light emitting area when the spherical mirror is used.

  In step 60 (S-60), a captured image of the fixed reference marker 1-d is acquired.

  In step 61 (S-61), the image center coordinates are acquired and set from the captured image.

  In step 62 (S-62), parameters necessary for the correction amount, the distance d between the light emitting area 51 and the center of the camera 1, the radius r of the spherical mirror, the distance L between the multi-camera imaging system 100 and the spherical mirror, and the subject. The angle γ in the light emitting region direction 63 is acquired from the image capturing range W of the image, and the pixel axis in the image.

  In step 63 (S-63), a vector from the pixel coordinates 60 (u0, v0) of the center of the image to the pixel coordinates (u1, v1) of the image of the spherical mirror reflection image 61 in the light source region 51 is derived.

In step 64 (S-64), the angle δ = tan ⁻¹ ((v1−v0) / (u1−u0)) of the vector 64 from the image center 60 of the image of the spherical mirror reflection image 61 in the light source region 51 is set. Then, the projection distance ds of the marker image in the light emitting area direction 63 is obtained by the following equation (25).

ステップ６５(S-65)に於いて、このｄｓから、上記の式（２２），（２３）により、球面鏡中心からの射影のずれ量Δdを求める。

In step 65 (S-65), from this ds, the projection deviation amount Δd from the center of the spherical mirror is obtained by the above equations (22) and (23).

  In step 66 (S-66), the following equation (26) is obtained by using this deviation amount as the value of each pixel axis in the pixel axis direction, and using this deviation amount, On the contrary, correction is made from the coordinates of the image of the spherical mirror reflection image 61 in the light source region 51 to the coordinates of the center 32 of the original spherical mirror.

ステップ６７(S-67)に於いて、画像内の固定基準マーカのメタデータとして種々の画像ＩＤや画像寸法情報，固定基準マーカＩＤ情報，固定基準マーカ座標などと共に補正値として記録する。

In step 67 (S-67), it is recorded as correction values together with various image IDs, image size information, fixed reference marker ID information, fixed reference marker coordinates, etc. as metadata of fixed reference markers in the image.

In the above description, a convex hemispherical mirror is used. However, a concave hemispherical mirror can be calculated equally by applying equation (19) instead of equation (15).
(Effect of the third embodiment)
In the third embodiment, by using a convex hemispherical mirror or a concave hemispherical mirror and an external light source different from the resolution size of the camera as a fixed reference marker for converting a plurality of photographed image groups separated into time into a common coordinate system. Since a fixed reference marker that is very small and has a much higher luminance than the subject can be obtained, it is possible to perform processing that is easy to detect, and also to perform image alignment processing with a high accuracy by using a fixed reference marker at a time interval.

Also, the deviation of the reflected image of the light source from the convex or concave hemispherical mirror and the center position of the spherical mirror in the captured image is the coordinate vector from the center of the captured image, information on the camera and light emitting area direction, its distance, the radius of the spherical mirror Since the position can be obtained with high accuracy by using the correction amount obtained from a simple deviation derivation formula depending on the distance to the spherical mirror, image alignment processing using a highly accurate fixed reference marker can be performed.
(Fourth embodiment)
As the fourth embodiment, the installation environment of the fixed reference marker 1-d when the convex hemispherical mirror 30, the concave hemispherical mirror 36, and the light emitting area 51 are used as the fixed reference marker 1-d in the third embodiment is as follows. The assumed long-term stable configuration will be described. The spherical mirrors 30 and 36 installed on the bottom surface of the concrete bridge 101 constructed on the sea often have a risk that the sea surface will be sprayed as the cross wind 140 and salt contained in the sea breeze will adhere to the surface, making it impossible to maintain the mirror surface. Need to avoid. FIG. 24 is a structural diagram of a fixed reference marker spherical mirror for preventing salt from adhering to the spherical mirror by the crosswind 140. It is necessary to protect the form in which light from the light emitting area 51 from a predetermined range is reflected toward the camera 2 from the sea breeze. FIG. 24A shows a mounting structure of the convex hemispherical mirror 30. A concrete screw anchor 131 is driven into the concrete bridge bottom surface 101, and the inside thereof has a screw 131 structure. A base 134 that supports the convex hemispherical mirror 30 is attached to a support column 133 and is fixed to the concrete with a screw portion 132. A wall 135 that surrounds the periphery of the convex hemispherical mirror 30 and prevents cross wind is attached to the base 134. At this time, a window structure that secures a reflection range of the spherical mirror necessary for a predetermined photographing range from the multi-camera photographing system 100 is adopted.

As for the window angle 41 of the convex spherical mirror wall to be secured for securing the reflection region, the optical path must be secured within the range where the photographing range 38 and the light emitting region 51 of the camera 2 are installed. Since the light emitting area 51 which is the maximum range in the shooting areas of the two adjacent cameras is located at the end of the shooting range 38 of the camera 2, the maximum fixed reference marker illumination angle addition 4 with respect to the shooting range angle 39 is the shooting range 38. As a result, the window angle 41 of the convex mirror wall is set to be not less than twice the shooting range angle 39. The actual window radius dw is given by dw = r * sin (ω / 2) using the radius r of the convex hemisphere, where ω is the distance L between the light emitting region 51 and the bridge bottom 101. If the imaging range is W, it is given by ω = 2 * tan ⁻¹ (W / L). Here, an arbitrary illumination light emitting region 51 is assumed. However, when closer to the camera 2, the window angle 41 can be narrowed. Further, the wall 135 can also be used as a holding portion for installation work on the bottom surface 101 of the convex hemispherical mirror fixed reference marker 1-d.

  FIG. 24B shows a case where the concave hemispherical mirror 36 is used. The concave hemispherical mirror 36 eliminates the protrusion from the bottom surface 101 in terms of applicability to the sea breeze, so that the influence of salt adhesion by the cross wind 140 is greatly reduced. In the figure, a lid 137 with an open window is attached for protection. The window of the lid 137 is dimensioned so that a necessary reflection image is formed due to the relationship between the light emitting area 51 and the predetermined range of the camera 2 in FIG. The window angle 42 of the concave hemispherical mirror to be secured for securing the reflection area is the same as that of the convex spherical mirror 30 in FIG. Must be secured. Since the light emitting area 51 which is the maximum range in the shooting areas of the two adjacent cameras is located at the end of the shooting range 38 of the camera 2, the maximum fixed reference marker illumination angle addition 40 with respect to the shooting range angle 39 is the shooting range 38. As a result, the window angle 42 of the concave half mirror wall may be set to be not less than twice the imaging range angle 39, similarly to the window angle 41 of the convex spherical mirror wall. It is conceivable to provide a separate protrusion for the fixing work, but it is conceivable to dig a groove 138 from the lid 137 to the base 136.

Installation on the bottom surface 101 of the concrete bridge, which is an actual subject, is simple with installation work that is fixed with screws 132, but it is also possible to adopt a structure in which the pillar 133 can be attached and removed as an insertion structure at a position fixed by the screws 132. In this case, installation is required for each photographing, but conversely, the influence of salt adhesion to the surface due to the sea breeze can be suppressed as much as possible.
(Effect of the fourth embodiment)
In the above embodiment, by providing around a convex hemispherical mirror provided with a window of a predetermined size obtained from the dimensions of the light source, camera direction and shooting conditions, salt adhesion due to crosswinds at sea is obtained while obtaining an appropriate reflection image. The wall can be used to prevent the convex hemispherical mirror from being easily installed on the bottom of the concrete bridge.

Furthermore, by applying the structure of a concave hemispherical mirror to the lid provided with a window of a predetermined value from the dimensions of the shooting conditions in the light source and the camera direction, while obtaining functional performance similar to that of a convex hemispherical mirror, It is possible to significantly reduce salt adhesion due to crosswinds at the bottom.
(Fifth embodiment)
As a fifth embodiment, a method for more reliably finding a reference mark such as a fixed reference marker 1-d or a dimensional reference marker 1-g or a feature point included in a subject in the first and second embodiments will be described. Furthermore, as another function of the method, a method of simultaneously acquiring and automatically using the reference mark ID, the common coordinate value, or the real world coordinate value on the spot will be described.

  As shown in FIG. 18 of the third embodiment, when a spherical mirror is used, a reflected image with very high luminance can be obtained with a diameter smaller than the resolution of 0.2 mm. In general, in order to eliminate an error signal, the camera function performs a process of ignoring the high luminance signal in units of pixels so as to obtain an easy-to-see image. When this function is activated, there is a risk of eliminating the high-luminance fixed reference marker 1-d when the reflected image 31 of the light source region 51 has a size smaller than the detection level of the pixel. When photographing in the vicinity of the fixed reference marker 1-d, it is necessary to suppress this function, and when suppression of such an error signal is further suppressed, it is necessary to discriminate it from a high-intensity error signal.

  Further, in the first to fourth embodiments, the use of high-intensity markers or subject feature points on the concrete bridge bottom surface 101 has been described. However, for example, when photographing the bottom surface 101 of the concrete bridge from below, the brightness of the sky is backlit at the end of the bridge width, and there is leakage light from the hole in the damaged part on the bridge that is severely deteriorated. Even if the marker 1-d or the dimension reference marker 1-g has a high luminance, there is a risk of misdetecting the others. Further, when using feature points included in a subject, when there are a plurality of similar feature points, it is necessary to identify them.

  As a method for stably specifying the reference mark, FIG. 25 shows a case where a fixed reference marker is reliably detected when an RFID passive tag and an RFID reader are used in the vicinity of the reference mark or a code mark sheet is used. The block diagram of the reading system of ID of a reference mark and coordinate information is shown.

  In FIG. 25, the radio 73 from the RFID reader 72 for power supply on the multi-camera imaging system 100 side is transmitted to the RFID passive tag 71 with an antenna installed on the bottom surface 101 of the concrete bridge, and the RFID passive with antenna is attached. The tag 71 uses the received power of the radio 73 to transmit the held information as a radio 74 from the RFID passive tag 71 as a reply, and the RFID reader 72 receives it. Since the RFID passive tag 71 that holds information does not require a battery, the RFID passive tag 71 can be used without doing anything even if it is taken only every few years. Since the RFID passive tag 71 can communicate at a distance of 2 m to 3 m by applying a radio wave system, the RFID reader 72 can be arranged as necessary to obtain information from the RFID passive tag 71. Is to be placed. Since the radio wave reach range can be limited by adjusting the antenna size and sensitivity of the RFID passive tag 71, the ID of the fixed reference marker 1-d when grounded at a plurality of locations can be specified. Furthermore, since information can be rewritten remotely using RFID, it is possible to change ID and coordinate information corresponding to a change in the situation with respect to the RFID passive tag 71 from a remote location such as on the multi-camera imaging system 100. . Since this RFID is wireless, before photographing the fixed reference marker image 10-d, the RFID wireless detects that there is a fixed reference marker 1-d in the vicinity, and the detection of the high-intensity reference mark is limited. If you select a shooting mode that reduces halation, such as reducing the aperture, increasing the shutter speed, or shooting a code mark sheet, or if you want to increase the contrast and emphasize the border of light and dark so that you can easily detect a large area detection pattern, Unlike the subject shooting mode, it can be used to increase the accuracy of image determination and increase its accuracy.

  FIG. 26 shows an example of presentation of the fixed reference marker detection mark, the related information ID, and the coordinate information by the code mark sheet. FIG. 26A shows an example of the configuration of the code mark sheet 65. First, in order to easily specify the code mark sheet 65 itself from the photographed image, circular code mark sheet specifying patterns 67 are arranged at three corners as simple marks. If the three circular detection patterns 67 are circular, the shape can be easily specified regardless of the screen direction, and the arrangement direction of the information on the code mark sheet can be expressed by the arrangement of three points. Further, when a detection pattern 67 having a high brightness and a reflected image or the like that is used for the fixed reference marker 1-d is larger than the fixed reference marker, the code marker sheet can be easily detected by binary detection of the brightness. . Next, code marks that describe information will be described. In general character expressions, even if a part is lost due to dirt or peeling, the reading becomes severe, but using a code with a combination of simple shapes makes it more resistant to dirt and partial deterioration. The information is obtained by arranging squares and converting them into binary code with the black and white arrangement, and displaying the fixed reference marker ID information 68 and the fixed reference marker information 69. Regarding reading of these information, unlike normal photographing of an object, edge enhancement processing, contrast enhancement, and shape recognition processing optimized for reading of the marks 68 and 69 only when the code marker sheet is detected. Processing can be reduced by strengthening. (B) is provided with a bridge shaft position information code mark sheet 65, a bridge width position information code mark sheet 66, and the like made of a coded pattern on the subject surface in the vicinity of the reference mark. The code mark sheet detection patterns 67 are arranged vertically and horizontally at three points, and further, a reference mark is provided on an extension line of the detection patterns 67 of the bridge axis position code mark sheet 65 and the detection patterns 67 of the bridge width position code mark sheet 66. If it is arranged so that 10-d comes, the reference mark with a small diameter can be reliably specified by the processing in the acquired image.

  FIG. 27 shows a flowchart of identification of a reference mark combined with use of a code mark sheet triggered by reception of RFID, and automatic acquisition of ID information or coordinate information of the reference mark. The RFID passive tag 71 or the code mark sheets 65 and 66 installed in the vicinity of the reference mark such as the fixed reference marker 1-d, the dimension reference marker 1-g, or the feature point of the subject is used.

  In step 70 (S-70), the RFID reading operation when the RFID is used is advanced.

  In step 71 (S-71), it is determined whether a new RFID signal has been received. If received, the process proceeds to the next process. If not, the process returns to step 70 (S-70).

  In step 72 (S-72), when the ID information of the reference mark of the RFID signal is received, the coordinate information in the RFID signal is acquired, and the detection operation of the detection pattern of the code mark sheet installed around the reference mark is started. . At this time, a photographing mode suitable for photographing the code mark sheet is activated. Since the detection process of a specific pattern in an image is heavy, in order to reduce this process as much as possible, reception of an RFID signal is used as a trigger. For detecting the detection pattern, when a high-brightness detection pattern is used, a light-weight one such as binary detection can be used.

  In step 73 (S-73), it is determined whether the detection pattern is detected. If detected, the process proceeds to the next step. If not detected, the confirmation of the reference mark with high accuracy by the coat mark sheet is given up and the process proceeds to a combination with RFID reception.

  In step 74 (S-74), the reference mark detection operation is started. In this case, a binary detection operation is performed in the case of high luminance detection, but an optimized shooting mode such as a shooting operation control that does not delete even if the diameter of the reference mark is as small as a pixel is set.

  The reference mark is detected from the combination direction of the detection patterns. In addition, a shooting mode suitable for reading a code mark around the detection pattern is activated. The photographing mode includes a boundary strength enhancement and a mark template matching process so that the code mark on the code mark sheet can be easily read.

  In step 75 (S-75), it is determined whether a reference mark and a hidden image are detected. If not, the process goes to step 82 (S-82) because it is impossible to detect the reference mark.

  In step 76 (S-76), the position of the reference mark is specified from the arrangement of the detection patterns, and the consistency between the reference mark and the position of the opening image is confirmed.

  In step 77 (S-77), it is confirmed whether the position matches the position from the detection pattern. If the intersection of the vertical and horizontal extension lines of a plurality of detection patterns is the position of the reference mark, and the position of the reference mark and the approximate image match, it is confirmed that the reference mark is not a bright spot due to an error signal. If they do not match, go to step 78 (S-78), and if they match, go to the next.

  In step 78 (S-78), the ID information of the reference mark by the code mark sheet and the inability to detect from the code mark sheet are recorded.

  In step 79 (S-79), as a mode for photographing and processing the code mark of the code mark sheet, the boundary strength is increased and the template alignment process of the mark is performed so that the code mark of the code mark sheet can be easily read. Read and record the reference mark ID and coordinate information.

  In step 80 (S-80), since the code mark sheet cannot be used, the reference mark detection operation based on the received information of the RFID is started.

  In step 81 (S-81), it is determined whether a reference mark and a hidden image are detected. If not, the process goes to step 82 (S-82) because it is impossible to detect the reference mark.

  In step 83 (S-83), it is confirmed whether there is another reference mark and a ghost image. If there is no unique mark as the reference mark, the process goes to step 84 (S-84). As a result, the process goes to step 85 (S-85).

  In step 84 (S-84), the ID acquired by the RFID and the unconfirmed result are recorded.

  In step 85 (S-85), it is confirmed by RFID, and the ID and coordinate information of the reference mark are read and recorded.

  As described above, the method for confirming the reference mark in the case of combining the ease of RFID radio reception and the highly accurate position specification by the code mark sheet has been shown, but in the combination of only the RFID radio reception and the reference mark, In the form of no RFID ID information in step 82 (S-82), the process proceeds to the conclusion of step 84 (S-84) and step 85 (S-85). In the confirmation of the reference mark by the code mark sheet, step 78 (S- 78), step 79 (S-79), and step 82 (S-82).

When using RFID, for example, it is possible to change the real world coordinates due to the change of the reference point of the real world coordinate system, and to change the writing of information when the measured value of the real world coordinates changes, so the change work is easy It becomes. When the acquisition information is limited, there is also a method of acquiring only the ID information and reading preset real world coordinates from the database based on the ID. In this way, ID information or real world coordinate information source of the fixed reference marker is set in the vicinity of the fixed reference marker, and it is automatically read on-site to save the time and effort of setting the real world coordinate. You can eliminate mistakes.
(Effect of 5th Embodiment)
In the fifth embodiment, based on the information of the reference mark in the vicinity through the wireless communication by the RFID reader of the RFID passive tag installed near the reference mark such as the reference marker or the feature point of the subject, the photographing control and the image are performed. By tuning the processing method to control for detection of reference markers and feature points, it is possible to lighten the processing of the whole shooting and identify landmarks for coordinate setting more reliably than detecting only the reference landmarks by image processing Will be able to.

  As another method, the detection pattern of the code mark sheet placed in the vicinity of the reference mark has a simple structure such as a circular pattern or a high-intensity marker with a large diameter. Only when the camera is identified, the processing is performed by recognizing the presence of the reference marker or the feature point of the subject in the vicinity of the shooting camera. By tuning to control for detection of reference markers and feature points, it is possible to reduce false detection when there is only a reference mark.

  Further, by arranging the plurality of detection patterns of the code mark sheet so as to indicate the position of the reference mark, the mark in the photographed image can be easily specified very accurately.

  Further, only when a pattern for detecting the code mark sheet is detected, the processing is reduced by performing boundary strength enhancement and mark template alignment processing so that the code mark on the code mark sheet can be easily read.

  Furthermore, by simultaneously using the radio signal of the RFID passive tag and the image capture of the code mark sheet, taking advantage of both, the RFID detects that the reference mark is in the vicinity and tunes the image capture control or image processing, and the coat mark sheet Since the position of the detailed mark is specified by the captured image, the feature point of the subject having a small diameter or inconspicuousness can be identified more reliably.

Since the reference mark ID, the fixed reference marker, the common coordinates of the feature point of the subject or the real world coordinates, or the dimension information of the dimension reference marker can be automatically acquired through an RFID radio signal or from a photographed image of the code mark sheet. , Image position, direction, dimension alignment for real time comparison and real-world coordinate conversion can be performed fully automatically, eliminating manual input operations, simplifying and manually Missed elements can be eliminated.
(Modification)
The present invention is not limited to the above-described embodiment, and includes various modifications. The above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to the one having all the configurations described.

  A part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is also possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

As modifications of the present invention, for example, there are the following (a), (b) and (c).
(A) In the above embodiment, the processing of captured images is performed by the image processing unit attached to the multi-camera capturing system. However, it is also possible to perform a series of processing of the captured information separately on a remote server or storage via communication. It is.
(B) In the above embodiment, the convex hemispherical mirror and the concave hemispherical mirror are described. However, the hemispherical surface may be a partial shape of the spherical mirror with no strictness and a reflection area.
(C) In the above-described embodiment, the coordinate system of the captured image with time separated is described in the form of uniquely matching, but the cumulative value of the image alignment error for each shooting does not necessarily match, and the pixel of the image Since the positions are not exactly the same, it is necessary to define an error range and finally combine the correlation processing of the luminance distribution in that range when comparing captured image groups over time. However, at that time, the common coordinates obtained in the embodiment can be used as a reference value for the correlation process, and a value based on the accumulated error can be used as an image range for the correlation process.

1-d fixed reference marker 1-g dimension reference marker 2 camera 3 inter-camera marker applying means 4 moving direction marker attaching means 5 fixed reference marker illuminating means 6 imaging region 7-a plural inter-camera markers 7-b, 7-c Movement direction alignment marker 9-a Marker image 9-b between multiple cameras Movement direction marker image 10-d Fixed reference marker image 10-g Dimension reference marker image 11 Control unit 12 Image processing unit 20 Crack 21 Communication line 22 Add multiple camera markers Means control line 23 Moving direction marker giving means Control line 24 Illumination means control line 25 Reference image 30 Convex hemispherical mirror 31 Virtual image 32 Center of convex hemispherical mirror 33 Projection of virtual image 34 Convex antireflective mirror 36 Convex hemispherical mirror 38 Shooting range 39 Photographing Range angle 40 Maximum fixed reference marker illumination angle addition 41 Window angle of convex spherical mirror wall 42 Window angle of concave spherical mirror 51 Light emitting area 52 Opposite to center Line segment 60 connecting launch points Image center 61 Light emitting area reflection image 63 Light emitting area direction 64 Vector 65 Bridge axis position code mark sheet 66 Bridge width position code mark sheet 67 Code mark sheet detection pattern 68 Fixed reference marker ID information 69 Fixed reference marker information 71 RFID Passive tag 72 RFID reader 73 Radio 74 from RFID reader Radio 80 from RFIF passive tag New image signal
81 Image signal receiving unit 82 Image extracting unit without marker image 83 Reference camera moving photographed image group extracting unit 84 Moving image group coordinate conversion value processing unit 85 Multiple camera simultaneous photographed image group extracting unit 86 Inter-camera image group coordinate conversion value processing unit 87 Reference image coordinate conversion processing unit 88 Fixed reference marker image detection unit 89 Common coordinate value conversion processing unit 90 Fixed reference marker common coordinate input 91 Markerless image synthesis processing unit 92 Common coordinate image information output
93 Dimension Reference Marker Image Detection Unit 94 Real World Coordinate Value Conversion Processing Unit 95 Dimension Reference Marker Actual Dimension Information Input
96 Markerless image composition processing unit 97 Real world coordinate image information output 98 Real world coordinate value 99 Common / real world coordinate system image composition processing unit 100 Multi-camera photographing system 101 Concrete bridge bottom surface 131 Concrete screw anchor 132 Screw 133 Post 134 Base 135 Wall 136 Base 137 Lid 138 Groove 140 Crosswind 200 Display image condition input 201 Image controller
202 Past image information storage unit
203 Image composition processing unit 204 Display image processing unit 205 Display image signal

Claims (15)

Using the common feature of the captured subject or the alignment feature between the adjacent images that temporarily provided a plurality of images of the target structure captured by the plurality of cameras having the same resolution. In a multi-camera imaging system that represents a specific range in one coordinate system, and that represents a specific range of an image group that was captured at the same time and captured at the same time using another plurality of cameras.
A fixed reference marker or feature point of two or more fixed points whose positions are the same on the subject is set on a part of a structure that is a subject over a wide range, even during time-lapse photography,
In any image within a specific range of the image group, the fixed two or more markers or feature points are detected,
For the coordinates of two or more fixed reference markers or feature points that are fixed in one coordinate system of a specific range of the image group, the same coordinates apart from the time corresponding to the fixed reference markers or feature points A multi-camera imaging system characterized in that a coordinate conversion process is performed by applying a value to obtain correction values for the same coordinate system at intervals over the entire captured image. In the multi-camera photographing system according to claim 1,
Further, a dimension reference marker consisting of at least two points for which the actual dimension is known is set on or near the subject, or a feature point on the subject whose size is defined from the subject is selected,
In any image of the image group, two or more dimensional reference markers or feature points on the subject are detected,
Apply the actual dimension value between the two points or the corresponding points to perform the coordinate conversion process,
A multi-camera imaging system characterized by obtaining correction values for real-world coordinates in the same coordinate system over time in the entire captured image. In the multi-camera photographing system according to claim 1,
A multi-camera imaging system using a real world coordinate system as the coordinate system. The multi-camera photographing system according to any one of claims 1 to 3,
An image in which the number of the fixed reference markers or feature points is N (N ≧ 3) and the feature points on the subject are photographed between two nearest fixed reference markers from any arbitrary image in the entire photographed image area. Arrange so that the number of coordinate conversion between adjacent camera shooting until
Applying the same N fixed reference markers or feature points common coordinates or real world coordinate values over time, and performing coordinate conversion processing,
A multi-camera imaging system characterized by obtaining correction values for the same common coordinates or real world coordinate system at intervals over the entire captured image. The multi-camera photographing system according to claim 4,
When specifying an image including the two nearest reference marks from an arbitrary image with respect to the reference mark such as the fixed reference marker or the feature point, adjacent to the image including the reference mark from the position of the arbitrary image Deriving at least one route distance to spell the image alignment marker
Select the route from each route distance to the image that contains the nearest said reference landmark that has a short distance to any image,
Applying the same common coordinates or real world coordinate values over time to the coordinates of the two nearest reference landmarks, perform coordinate transformation processing,
A multi-camera imaging system characterized by obtaining correction values for the same common coordinates or real world coordinate system at intervals over the entire captured image. The multi-camera photographing system according to any one of claims 1 to 4,
Install a reflection point with higher reflectivity than the surrounding object as the fixed reference marker or the dimension reference marker and irradiate light from the camera system side, or the fixed reference marker or the dimension reference marker itself emits light A multi-camera photographing system that emits light having higher brightness than the surrounding subject. The multi-camera photographing system according to claim 6,
Provide an external light source,
The fixed reference marker is a convex spherical mirror or a concave spherical mirror,
The multi-camera photographing system, wherein the fixed reference marker is illuminated by the external light source, and the camera captures a reflection image of the external light source, which is used as the fixed reference marker having high luminance light. The multi-camera photographing system according to claim 7,
The direction of the external light source from the camera in the captured image of the camera, distance information between the camera and the external light source in real space, radius information of the convex spherical mirror or concave spherical mirror, and the multi-camera imaging system Distance information to the convex spherical mirror or concave spherical mirror, photographing range information on the subject including the convex spherical mirror or concave spherical mirror in the real space of the camera, and the external by the convex spherical mirror or concave spherical mirror in the photographed image A value obtained by obtaining a deviation value from the center position of the convex spherical mirror or the concave spherical mirror on the subject of the reflected image of the external light source as the image of the fixed reference marker from the position information of the reflected image of the light source, and correcting the center position. Is a position of the fixed reference marker. The multi-camera photographing system according to claim 7,
The fixed reference marker is the convex spherical mirror,
The shooting range of the camera, the camera, and the convex spherical mirror are viewed from the center of the sphere of the convex spherical mirror so that a reflection image of the external light source by the convex spherical mirror is included in any shooting range under arbitrary shooting conditions. A multi-camera photographing system characterized in that a wall is provided around the convex spherical mirror with a window gap of at least twice the angle formed by the distance. The multi-camera photographing system according to claim 7,
The fixed reference marker is the concave spherical mirror,
The camera as viewed from the bottom of the center position of the spherical surface of the concave spherical mirror at the center of the circular section of the concave spherical mirror so that a reflection image of the external light source by the spherical mirror is included in any imaging range under an arbitrary imaging condition A multi-camera photographing system characterized by having a structure in which a window gap that is at least twice the angle formed by the distance between the photographing range and the camera and the concave spherical mirror is opened. The multi-camera photographing system according to any one of claims 1 to 4,
In the vicinity of a reference mark such as the fixed reference marker or the feature point or the dimension reference marker, a plurality of patterns having a simple shape with a large size, clear brightness, or high brightness or easy to recognize an image are arranged.
Photographing the reference mark, the hidden image, and the pattern in the vicinity thereof by the same photographing technique, and starting the photographing control or image processing for detecting the reference mark triggered by the detection of the pattern, or the plurality The multi-camera photographing system is characterized in that the reference mark is specified based on a combination of information on the direction and the reference mark and the hidden image by disposing the reference mark according to the positional relationship of the patterns. . The multi-camera photographing system according to any one of claims 1 to 4,
In the vicinity of a reference mark such as the fixed reference marker or feature point or the dimension reference marker, an RFID passive tag is installed,
When the RFID passive tag is detected, a multi-camera imaging system is provided that activates imaging control suitable for detecting the reference mark or activates image processing to identify the reference mark. The multi-camera imaging system according to claim 12,
An RFID reader,
Identifying the fixed reference marker or the feature point or the dimension reference marker by obtaining a wireless signal from the reference mark, a hidden image and the RFID passive tab,
The RFID passive tag has identification information of the fixed reference marker or feature point or the dimension reference marker or coordinate information or real world coordinate information common over time,
The multi-camera imaging system, wherein each information is acquired by the RFID reader and automatically provided as information for coordinate conversion processing by the fixed reference marker or feature point or the dimension reference marker. The multi-camera imaging system according to claim 11,
Along with the plurality of patterns, the fixed reference marker or the feature point or the identification information of the dimension reference marker or a code mark sheet having common coordinate information or real world coordinate information with time separated,
In response to the detection of the pattern, the position of the reference mark is specified from the positional relationship of the plurality of patterns, and shooting control suitable for reading the code mark sheet is activated,
A multi-camera imaging system, wherein each piece of information acquired from the photographed image of the code mark sheet is automatically provided as information for coordinate conversion processing using the fixed reference marker, the feature point, or the dimension reference marker . The multi-camera photographing system according to any one of claims 1 to 4,
In the vicinity of a reference mark such as the fixed reference marker or the feature point or the dimension reference marker, an RFID passive tag and a simple shape having a large size, a clear contrast, or high brightness, or a plurality of patterns that are easy to recognize images Installing the fixed reference marker or the feature point or the identification information of the dimension reference marker or the code mark sheet having the common coordinate information or the real world coordinate information over time,
When the RFID passive tag is detected, a suitable photographing control for detecting the reference mark is activated or image processing is activated, and the photographing control for detecting the plurality of patterns is activated,
Triggered by the detection of the pattern, the position of the reference mark is specified from the following relationship among the plurality of patterns, and shooting control suitable for reading the code mark is activated, and acquired from the image of the shot code mark sheet The multi-camera imaging system, wherein the information is automatically provided as information for coordinate conversion processing using the fixed reference marker, the feature point, or the dimension reference marker.

Images