JP6022330B2 - Camera system - Google Patents

Description

  The present invention relates to a camera system for automatically recognizing a reference point used for calibration of various camera parameters between a plurality of cameras, shape recognition of a photographed object, and the like from an image photographed by a camera, and particularly, a reference installed in a real space. The present invention relates to a camera system that allows an operator to easily set points.

  A camera system for tracking a person using a surveillance camera has been studied. When constructing such a camera system, it is necessary to obtain camera parameters such as the position and orientation of the camera and to adjust the correspondence between the virtual space recognized by the camera system and the monitoring space that is the real space. This adjustment work is called calibration.

  For example, in order to perform calibration, a plurality of reference points serving as reference points for position measurement are set in a monitoring space within the camera field of view, and the reference points are photographed by the camera. Then, each reference point is detected from the photographed image, and camera parameters are measured from the geometric relationship between the detection position of the reference point on the image and the reference point position in the monitoring space.

  Camera parameters such as camera position and orientation can be measured with a single camera if the 3D coordinates of the reference point are given, but can be measured without giving the 3D coordinates of the reference point if a common field of view of multiple cameras is used. it can. That is, a geometrical feature in which a plurality of reference points are set in a common field of view and the same reference point is detected from each image captured by a plurality of cameras sharing the field of view, and the detection positions of these reference points are the same position. It is possible to measure the camera parameters of each camera by using the relationship.

  In addition to such calibration, the reference point is also used for measuring a three-dimensional position / shape of a fixture or the like installed in the monitoring space. For example, a plurality of reference points are set on the surface of a fixture, etc., each reference point is detected from an image captured by a camera, and the geometric relationship between each reference point and the camera position / posture is applied to the surface of the fixture, etc. By obtaining the three-dimensional position of each installed reference point, the position or shape of a fixture or the like may be measured.

  Conventionally, in order to install such a reference point, there has been proposed an image processing apparatus that irradiates spot light from a light emitting device onto a monitoring space and realizes camera calibration using the irradiated point as a reference point (Patent Document 1). .

JP 2004-46772 A

  By the way, in response to the recent drop in camera prices, a camera system has been proposed in which a large number of surveillance cameras are installed to monitor a wide surveillance space. In such a case, it becomes difficult to set the reference point by irradiating the spot light from the outside of the monitoring space using the light emitting device as in the prior art. For this reason, it is conceivable that the operator holds a portable laser pointer or the like, and irradiates the spot light while moving in the field of view of the camera to set the reference point.

  However, in that case, the operator and the reference point are simultaneously present in the field of view of the camera and the reference point is set. Depending on the position of the camera and the operator, the blind spot generated by the operator may be the reference point that should have been set. And the reference point may be missed. If the reference point is missed, the reference point needs to be re-installed.

  That is, there is a problem that it is difficult for an operator to correctly recognize a standing position that does not form a blind spot at a reference point and to proceed with work efficiently.

  The camera system according to the present invention includes one or a plurality of cameras that capture an image of a predetermined space, a control device that processes an image captured by the camera, and a reference point that can be captured by the camera and is set by an operator. The control device includes at least a camera parameter including a position / posture of the camera in the space and a required position in the space where the reference point needs to be installed. A storage unit for storing, a reference point detection unit for detecting the reference point from an image captured by the camera, and using the camera parameters and the requested position of the camera, the line-of-sight direction from the camera to the requested position An operator position calculating unit that calculates a position as a blind spot forming worker position at which the requested position cannot be captured from the camera when the worker exists; And an output unit for outputting blind spot formation operator position.

  In the camera system according to the present invention, the output unit may output the requested position.

  In another camera system according to the present invention, a plurality of the cameras are arranged so as to have a common field of view in the space, and the control device further includes a calibration unit that calibrates the camera parameters using the reference points. The calibration unit has the same reference point image in the space when the reference point detection unit detects a reference point image as the reference point in each of the images simultaneously captured by the plurality of cameras. The camera parameter is calibrated by applying a constraint condition that it is a position.

  In the camera system according to the present invention, the control device captures the reference point only when the reference point detection unit detects the reference point existing at the required position in the common visual field. A configuration may further include a requested position setting unit that deletes the requested position from the storage unit.

  Further, in the camera system according to the present invention, the control device may be configured such that the required position is based on a segmented area in which an image captured by the camera is radially divided at predetermined angles from an intersection of an optical axis of the camera and an imaging plane. It is possible to adopt a configuration further including a required position setting unit for setting.

  According to the present invention, since the operator knows the standing position where no blind spot is formed at the position where the reference point is installed, the reference point can be efficiently detected with less redo and trial and error, and calibration work and measurement work can be performed. Is easy and quick.

1 is a schematic diagram illustrating a schematic configuration of a camera system according to an embodiment of the present invention. It is a typical top view of monitoring space. It is the block diagram which showed the structure of the outline of a control apparatus. It is a schematic diagram which shows the data structure of a camera parameter in a table format. It is a schematic diagram which shows the example of the request | requirement position set to the camera pair which consists of the camera 2B and the camera 2C. It is a flowchart which shows the outline of the operation | movement in the calibration process of the camera system which concerns on embodiment of this invention. It is a flowchart which shows the outline of the presentation process of the recommended standing position which the camera system which concerns on embodiment of this invention performs by a calibration process. It is a schematic diagram which shows the data structure of reference | standard point information in a table format. It is a schematic diagram which shows the data structure of request information in a table format. It is a schematic diagram which shows the data structure of a chain determination table in a table format. It is a schematic diagram of the image area | region which shows the example of calculation of the blind spot formation position calculated | required directly in the image area | region of the cameras 2B and 2C. It is a schematic diagram of the image area | region which shows the calculation example of the worker standing position corresponding to FIG. It is a schematic diagram of the image area | region which shows the example of narrowing down of the recommended standing position corresponding to FIG. It is an example of the display screen which presents the recommended standing position in a display part. It is another example of the display screen which shows the recommended standing position in a display part. It is a flowchart which shows the outline of the presentation process of the recommended standing position which an image process part performs by a calibration process corresponding to the display screen of Fig.15 (a). It is a flowchart which shows the outline of the operation | movement in the measurement process of the camera system which concerns on embodiment of this invention. It is a schematic diagram of the image area | region which shows the example of a setting of the installation request information in a measurement process. It is a schematic diagram which shows the example which installs a reference point with apparatuses other than an irradiator.

  Hereinafter, a camera system according to an embodiment of the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings.

[Configuration of Camera System 1]
FIG. 1 is a schematic diagram illustrating a schematic configuration of a camera system 1 according to the embodiment. The camera system 1 includes a plurality of cameras 2 (2A, 2B, 2C, 2D, 2E, 2F, 2G, 2H), an irradiator 3, a control device 4, and a display unit 5. FIG. 1 shows a situation where an operator 10 carrying the irradiator 3 and the display unit 5 irradiates the floor surface (reference surface 11) in the monitoring space to form a light image on the floor surface. The object 12 is installed on the floor.

  In the present embodiment, the center of gravity of the optical image is the reference point 13. The worker 10 moves in the monitoring space and installs reference points 13 at various places. The camera 2 captures the reference point 13, and the control device 4 detects the reference point 13 from the captured image, and performs calibration of the camera 2 and three-dimensional measurement of the object 12. Further, the control device 4 calculates a worker standing position where a blind spot by the worker 10 does not occur at a place (requested position) where the installation of the reference point 13 is required and outputs the calculated position to the display unit 5. The operator 10 can perform the work efficiently by reducing the failure by performing irradiation from the displayed standing position.

  The camera 2 is a surveillance camera, and each camera 2 is connected to the control device 4. Each camera 2 is arranged in a chain with at least one other camera providing a common field of view, and the sum of the fields of view of the plurality of cameras 2 becomes a monitoring space. The camera 2 captures a reference point 13 that moves together with the worker 10 at predetermined time intervals, and sequentially outputs the captured images to the control device 4.

  In the example shown in FIG. 1, the camera system 1 includes eight cameras 2A, 2B, 2C, 2D, 2E, 2F, 2G, and 2H, and the control device 4 includes A, B, C,. A camera ID of H is assigned and managed. The plurality of cameras 2 are arranged in a grid pattern on the ceiling with their optical axes oriented vertically downward. FIG. 2 is a schematic plan view of the monitoring space 20, and the position of the object 12 within the reference plane 11 and the arrangement of the eight cameras 2 (2A to 2H) are indicated by “☆”. The distance between the cameras 2 is set so that adjacent cameras have a common field of view in consideration of the angle of view of the cameras. In this example, camera 2A has at least a common field of view with camera 2B, camera 2B has at least a common field of view with cameras 2A and 2C, and camera 2C has at least a common field of view with cameras 2B, 2D, 2G, and 2H. , Camera 2D has a common field of view with at least cameras 2C, 2G, 2H, camera 2E has a common field of view with at least camera 2F, camera 2F has a common field of view with at least cameras 2E, 2G, and camera 2G has at least The cameras 2C, 2D, 2F, and 2H have a common visual field, and the camera 2H has at least a common visual field with the cameras 2C, 2D, and 2G. In the figure, the field of view of the camera 2B is indicated by 100b, and the field of view of the camera 2C is indicated by 100c. The other visual fields of the camera 2 are not shown. A region 200 indicates a common field of view of the field of view 100b of the camera 2B and the field of view 100c of the camera 2C. Note that the number of cameras 2 is arbitrary as long as it is two or more, and the position and orientation of the camera 2 are arbitrary within a range where at least common fields of view are chained.

  The irradiator 3 is a light source that outputs spot light. For example, a laser pointer or a collimated LED light can be used. The irradiator 3 is carried by the operator 10 and irradiates light to the object 12 such as the reference plane 11 or the fixture by the operation of the operator 10. By this irradiation, the reference point 13 is set on the reference surface 11 or the surface of the object 12. The reference point 13 moves in the monitoring space as the operator 10 operates and moves. Incidentally, the operator 10 irradiates the reference plane 11 in the common visual field in the calibration work of the position and orientation of the camera 2, and irradiates the object 12 existing in the common visual field in the three-dimensional measurement work of the object 12. The irradiator 3 is preferably one that can be easily obtained and used without requiring a license.

  The control device 4 is connected to each camera 2 via a LAN such as Ethernet (registered trademark) or a wiring such as a coaxial cable. The display unit 5 is connected by a wireless line such as Bluetooth. The control device 4 acquires an image taken by the camera 2 and detects the reference point 13 from the image. The control device 4 calculates a worker standing position that does not hinder the detection of the reference point 13 at the requested position, and outputs information including the requested position and the worker standing position to the display unit 5. The detected reference point 13 is used to calibrate the camera 2 or measure the object 12.

  The display unit 5 includes a wireless communication interface and is connected to the control device 4 through a wireless line. The display unit 5 includes display means including an image display device such as a liquid crystal touch panel display and an arithmetic device, and displays information input from the control device 4 so as to be visible.

  FIG. 3 is a block diagram showing a schematic configuration of the control device 4. The control device 4 includes an image acquisition unit 40, a storage unit 41, an image processing unit 42, and a display communication unit (output unit) 43.

  The image acquisition unit 40 includes a camera communication unit 400 and a synchronization control unit 401, and synchronizes images from the cameras 2 and outputs the images to the image processing unit 42.

  The camera communication unit 400 is a communication interface that communicates with the camera 2 and sequentially receives images captured by each camera 2. The camera ID of the camera 2 that captured the image is assigned to the received image and the synchronization control unit 401 receives the image. Output. Here, if the camera 2 is an analog camera, the camera ID can be specified from the correspondence with a preset connection terminal, and if it is an IP camera, the camera ID can be specified from the correspondence with a preset transmission source address.

  The synchronization control unit 401 synchronizes the image capturing times received by the camera communication unit 400 from the respective cameras 2. Specifically, each camera 2 captures and transmits an image with a capturing period sufficiently shorter than the reference period, and the synchronization control unit 401 synchronizes the received image with less than an allowable time difference set to be considered as simultaneous. It is assumed that the taken simultaneous image is taken. The synchronization control unit 401 extracts a simultaneously shot image for each reference period, assigns a common frame number to the simultaneously shot images, and outputs the same to the image processing unit 42. The synchronization control unit 401 generates a buffer (not shown) that accumulates images from each camera 2 for at least the time of the reference period, and generates time information to determine the coincidence, in order to extract simultaneously shot images for each reference period. Timing means (not shown). For example, the reference period may be 3 fps (frame per second), the imaging period may be 15 fps, and the allowable time difference may be 66 ms (mili second), for example. Note that the synchronization timing may be directly controlled by outputting a synchronization pulse from the synchronization control unit 401 to each camera 2 for each reference period.

  The storage unit 41 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 41 stores various parameters, a program of the image processing unit 42, and the like, and inputs / outputs such information to / from the image processing unit 42. The various parameters include a camera parameter 410, request information 411, an irradiation limit value 412, reference point information 413, object measurement information 414, and a chain determination table 415.

  The camera parameters 410 are internal parameters and external parameters of each camera 2 and are associated with the camera ID of the camera 2. FIG. 4 is a schematic diagram showing the data structure of the camera parameter 410 in a table format.

Internal parameters include focal length and optical center position of the lens. In FIG. 4, for example, the focal length of the camera 2A is represented as f _A , and the optical center position of the lens is represented as c _A. Accurate values are set in advance for the focal length and the optical center position of the lens, and do not change during calibration of the camera 2.

  The external parameters include at least information on the position and orientation of the camera 2 in the monitoring space. Specifically, the installation height and rotation matrix of the camera 2 expressed in a common coordinate system (world coordinate system) common to the plurality of cameras 2. , Translation vector, photographing plane normal vector, and inter-camera distance.

  Here, the world coordinate system is a right-handed orthogonal coordinate system XYZ that imitates a monitoring space. In this embodiment, the Z axis is set vertically upward, the floor surface that is the reference surface is set to height Z = 0, and the reference surface The point below the camera A at the vertical is the coordinate origin. In addition, the horizontal direction and the vertical direction of the image of the camera A are the x-axis and y-axis of the photographing plane, and the x-axis and y-axis projected onto the reference plane are the X-axis and Y-axis, respectively.

For example, the rotation matrix R _B of the camera 2B defines the rotation around the optical axis of the camera 2B. The normal vector n _{B on the} imaging plane defines the direction of the optical axis of the camera 2B. The translation vector is a two-dimensional vector that defines the relative positional relationship between the two cameras 2 in the XY plane. Translation vector in the present embodiment is defined the position of the camera 2A starting, for example, the translation vector t _B of the camera 2B is starting from a position of the camera 2A, 2B in each XY plane, and the end point. The inter-camera distance is the distance of a certain camera 2 from the reference camera 2 and is the distance in the XY plane of both cameras 2. In the present embodiment, the inter-camera distance is defined with reference to the camera 2A. For example, in FIG. 4, the inter-camera distance d _AB indicates the distance between the cameras 2A and 2B. The inter-camera distance may be a distance between the three-dimensional coordinates of the camera 2.

  Among these external parameters, the installation height is set to an accurate value in advance and does not change during the calibration work of the camera 2. The rotation matrix and the normal vector of the imaging plane are set in advance as initial values that indicate a vertically downward value, which is a value in the installation plan, and are used as a calibration work target of the camera 2. Initial values are not set for the translation vector and the inter-camera distance, but are set by the calibration operation of the camera 2.

  The request information 411 can simultaneously photograph a request position that is a position in a monitoring space where it is necessary to request the worker 10 to set the reference point 13 for calibration and three-dimensional measurement, and the request position. Information associated with a pair of cameras. In the request information 411, coordinate data representing a requested position is set in association with the camera ID of the camera pair by a request setting unit 422 described later. The requested position can be a point or a line, but is preferably an area. By setting the required position as an area, a request from the camera system 1 to the worker 10 becomes a request to set the reference point 13 at an arbitrary position in the area. Accordingly, the burden on the operator 10 is reduced because the target of irradiation increases. Therefore, in this embodiment, the requested position is an area, and the area of each requested position is also referred to as an area hereinafter. FIG. 9 used in the description of the subsequent processing schematically shows the data structure of the request information 411. In FIG. 9, symbols such as “A5” indicating the request position indicate areas as described later. Further, the request information 411 includes a request condition that defines the installation requirement of the reference point 13 at the required position.

  The irradiation limit value 412 is an irradiation limit distance value set in advance in consideration of the characteristics of the irradiator 3. In a place that is too far from the light source, the luminance per unit area decreases due to the increase in the diameter of the optical image, and there is a high possibility that the reference point 13 cannot be detected. The irradiation limit distance value is a threshold value for determining whether or not the requested position is so far that the reference point 13 cannot be detected.

  The reference point information 413 is generated by a reference point detection unit 420 described later, and is used by the general calibration unit 421, the precision calibration unit 424, and the object measurement unit 425. The reference point information 413 is information of the reference point 13 detected from the captured images of the plurality of cameras 2 captured in synchronization with the reference period, and an image of the reference point 13 (reference point image) detected in the captured image. Are associated with the camera ID and frame number assigned to the image from which the reference point image is detected. FIG. 8 used in the description of the subsequent processing schematically shows the data structure of the reference point information 413. In FIG. 8, (57, 90) indicating the detection position indicates the x coordinate and y of the reference point image in the captured image. It represents a pair with coordinates.

  The object measurement information 414 is information obtained by measuring the three-dimensional shape of the object 12 existing in the monitoring space, and is generated by the object measurement unit 425 based on the reference point information 413. The object measurement information 414 is used as background information when the camera system 1 detects or tracks a person in the monitoring space.

  The linkage determination table 415 is information used for managing the progress status of the precision calibration process, which will be described later. As shown in FIG. 10 used in the description of the subsequent process, the camera pair to be subjected to the precision calibration and the completion of the precision calibration are completed. The data structure is associated with / incomplete.

  The image processing unit 42 is configured using an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit), and detects a reference point by reading a program from the storage unit 41 and executing it. Functions as a unit 420, a rough calibration unit 421, a request setting unit 422, a recommended standing position calculation unit 423, a precision calibration unit 424, an object measurement unit 425, and the like.

  The reference point detection unit 420 detects the reference point 13 from the image captured by the camera 2. Specifically, a reference point image detection process is performed on images captured by each camera 2 at a reference period, and the detection position of each reference point image is assigned to the image from which the reference point image is detected. The reference point information 413 of the storage unit 41 is stored in association with the ID and the frame number. In the detection process, a difference area is extracted based on a background difference or an inter-frame difference, a labeling process is performed on the extracted difference area, and an area in a size range set in advance based on the size of the light image of the irradiator 3 is emitted. The image is selected as an image, and the center of gravity of the selected light image is detected as the reference point 13. By providing a restriction on the size range, the difference area caused by the operator is excluded.

  Here, the reference point 13 set in the common visual field of the plurality of cameras 2 is basically photographed simultaneously by each of the plurality of cameras 2, and one reference point 13 is set at a temporary point by the operator 10. It is. Therefore, a camera pair having a common field of view can be detected by searching for the reference point 13 associated with the same frame number in the reference point information 413. The reference point detection unit 420 extracts a reference point pair having the same frame number from the reference point information 413 and stores the extracted camera ID of the reference point 13 in the request information 411 as a camera pair having a common field of view.

  In this embodiment, the calibration operation of the camera 2 is performed in two stages: rough calibration (rough calibration) and higher-precision calibration (precision calibration). The general calibration process is performed by the general calibration unit 421, and the precise calibration process is performed by the precise calibration unit 424. The rough calibration unit 421 and the precision calibration unit 424 perform respective calibration processes based on the reference point information 413 generated by the reference point detection unit 420. Here, the precision calibration unit 424 uses the reference points 13 at more required positions than the general calibration unit 421 to perform calibration with higher accuracy than the general calibration.

  The rough calibrating unit 421 and the precision calibrating unit 424 extract, from the reference point information 413, reference point images that are commonly detected in the simultaneously captured images, and the constraint that these reference point images correspond to the same position in the monitoring space. The position / orientation of the camera 2 is calculated by applying the conditions, and the camera parameter 410 of each camera 2 is updated with the position / orientation. Specifically, the position of the reference point image detected by the camera pair to be calibrated is corrected for distortion with the internal parameters of the camera 2, and the detected position of the reference point image and the installation height of the camera 2 are used. A homography matrix is derived. Then, a rotation matrix, a translation vector, a normal vector of the imaging plane, and an inter-camera distance of each camera constituting the camera pair are calculated from the homography matrix for each camera pair, and these are converted into a common coordinate system to be used for each camera 2. Find the position / posture. How to find the position and orientation using the homography matrix is known from “Study on Optimal Triangulation of Planar Scene” Information Processing Society of Japan, CVIM, 2010-CVIM-171 (4), 1-8, 2010-08-11) It is. In addition to the method using a homography matrix, various calibration methods based on epipolar geometry can be applied.

  The object measurement unit 425 applies a constraint condition that the reference point images commonly detected from the simultaneously captured images by the reference point detection unit 420 correspond to the same position in the monitoring space, and the three-dimensional object 12 existing in the common visual field. Measure the position. The three-dimensional measurement function by the object measurement unit 425 is used after the precision calibration unit 424 has finished the precision calibration process. That is, the three-dimensional measurement process (object measurement process) is executed in a state where the precise position / orientation of the camera is stored in the camera parameter 410 of the storage unit 41.

  Specifically, the object measurement unit 425 determines the triangulation principle at the position / posture of the camera pair determined by the camera parameter 410 and the detection position of the reference point 13 captured by the camera pair stored in the reference point information 413. Is applied to derive the three-dimensional position of each reference point 13. Further, the object measuring unit 425 derives a plane for each requested position by regarding a plurality of reference points 13 having the same corresponding requested position among the reference points 13 as points on the same plane. Further, the object measuring unit 425 derives a tangent line between the derived planes, and sets the tangent line as a side of each plane. The monitoring space is a solid surface composed of a reference surface (floor surface), a virtual ceiling having the same height as the camera installation height, and a virtual wall defined by a finite range of XY coordinates, and a tangent line is defined as a reference surface, a virtual ceiling, and a virtual wall. Derived even between.

  The object measurement unit 425 stores the data of each plane in which the edge is derived and the contour is defined in the object measurement information 414 of the storage unit 41 as the three-dimensional shape data representing the surface of the object 12.

  As described above, the reference point 13 detected by the reference point detection unit 420 is used in the rough calibration process, the precision calibration process, and the object measurement process. A request setting unit (requested position setting unit) 422 sets a plurality of required positions where the installation of the reference point 13 is required within the common visual field for each camera 2. Specifically, the request setting unit 422 generates the above-described request information 411 and stores it in the storage unit 41. Further, after storing the request information 411, the request setting unit 422 sequentially refers to the reference point information 413 and the request information 411, and the reference point detection unit 420 determines the reference point 13 existing at each request position in the common visual field. Whether or not it has been detected is confirmed, and only when the reference point 13 is detected, the requested position where the reference point 13 is photographed is deleted from the storage unit 41. In addition, the setting criteria of the request | requirement position in each process differ, and each is demonstrated below.

(1) Required position in rough calibration process In the calibration process (rough calibration process and precision calibration process) of a camera pair having a common field of view, it is necessary to detect a plurality of reference points 13 from the reference plane of the common field of view. In this embodiment, as described above, calibration using a homography matrix is applied to each camera pair. In this case, it is required that four or more reference points 13 that are not on the same straight line for each camera pair are detected from the simultaneously photographed image, that is, a total of eight or more reference points 13 are detected by the two cameras 2 that form a pair. Is done. From this point of view, the request setting unit 422 sets a plurality of request positions in units of divided areas obtained by dividing the images of the cameras 2. Preferably, the request setting unit 422 divides the image captured by each camera 2 radially at predetermined angles from the intersection of the optical axis of the camera and the imaging plane. The blind spot area generated by the presence of the worker 10 is basically located along a line extending from the intersection of the optical axis of the camera 2 and the imaging plane toward the position of the worker 10, According to this method, the required position where the blind spot area is formed is reduced. In other words, there is an effect that it is difficult to be affected by blind spots. For example, it can be divided into 8 areas in 45 degree increments radially.

  In order to ensure the accuracy of calibration, it is desirable that these reference points 13 are dispersed. From this point of view, apart from the above-mentioned plurality of radial areas, there is a method of classification in which an area where the intersection between the optical axis of the camera 2 and the imaging plane is located in the image area of the camera 2 is additionally set as one area. Is preferred. Thereby, it is possible to avoid the crowding of a plurality of radial areas in the central portion of the radial section, and it is possible to eliminate the situation where the reference point 13 is detected in a concentrated manner in a narrow region of the central portion. That is, it is possible to prevent the degree of dispersion of the detected reference point 13 from being lowered, and to ensure calibration accuracy. Therefore, in the present embodiment, the request setting unit 422 includes 9 areas including a circular area corresponding to the vicinity of the bottom of each camera 2 and 8 areas obtained by radially dividing the area remaining outside the camera 2 in 45 degree increments. Is set as the requested position.

  FIG. 5 is a schematic diagram showing the required positions set as described above, and FIG. 5 shows an example of the required positions set for the camera pair composed of the camera 2B and the camera 2C. FIGS. 5A and 5B show image areas 21b and 21c photographed by the cameras 2B and 2C arranged in the monitoring space 20 shown in a plan view in FIG. Incidentally, the range shown in the image areas 21b and 21c in the monitoring space 20 is represented by the fields of view 100b and 100c of the cameras 2B and 2C, respectively, in the plan view of the monitoring space 20 shown in FIG. Requested positions B1 to B9 are shown in the image area 21b of the camera 2B shown in FIG. 5A, and required positions C1 to C9 are shown in the image area 21c of the camera 2C shown in FIG. 5B. Yes. Specifically, an area B1 having a radius R on the reference plane 11 is set as an image area 21b as a circular area corresponding to the vicinity of the camera 2B vertically below, and lines connecting the cameras 2B and 2C vertically below each other are set. The image area around the area B1 is divided into 8 areas in increments of 45 degrees as 0 degrees, and areas B2 to B9 are set in order. Similarly, an area C1 which is one circular area with respect to the camera 2C and areas C2 to C9 obtained by radially dividing the periphery are set.

  The circular area in the center is set for the purpose of dispersing the radial area, and the size thereof is determined so that the object can be achieved. For this purpose, the central area is not limited to a circle but may be a polygon such as a quadrangle.

  For each camera 2, the request setting unit 422 reads the initial value of the camera parameter 410 from the storage unit 41, classifies the image area of the camera 2 based on this, and sets nine areas as the required position in the rough calibration process. To do. Then, for each camera 2 of the camera pair, the reference point 13 (reference point image) is detected in any four of the nine areas, and the reference point 13 (reference point image) is detected in a total of eight areas for the camera pair. Is set as a requirement. Two or more detections are allowed in one area.

  The requested position in the image area is not limited to the radial section, and the requested position can be dispersed in the image area by a block (lattice) section.

(2) Required Position in Precision Calibration Process The description generally given as the calibration process in (1) above is common to the rough calibration process and the precision calibration process, and the request setting unit 422 in this embodiment is a precision calibration process. The above nine areas are also set as required positions. For each camera 2, the request setting unit 422 reads the camera parameters 410 roughly calibrated from the storage unit 41, classifies the above-described image area of the camera 2 based on the camera parameters 410, and sets nine areas as required positions in the precision calibration process. Set. Then, for each camera 2 of the camera pair, the reference point 13 (reference point image) is detected in all the nine areas, and the camera pair requires that the reference point 13 (reference point image) is detected in a total of 18 areas. Set as a condition. Two or more detections are allowed in one area.

  That is, the difference between the required position in the precision calibration process and the required position in the rough calibration process is that the number of reference points 13 required by the required position in the precision calibration process is larger than that in the rough calibration process, and is used for setting the area. The camera parameter 410 is an initial value in rough calibration, whereas it is a value that is roughly calibrated in precision calibration.

(3) Required position in measurement processing The required position in measurement processing is a position where detection of the reference point 13 required for three-dimensional measurement of the object 12 is required, and is based on images taken by each camera 2. Generated.

  In the present embodiment, it is assumed that the object surface is a collection of planes, and the request setting unit 422 sets an area estimated as a plane constituting the object surface as a required position. Further, in order to specify a plane, the request setting unit 422 sets detection of three or more reference points 13 that are not on the same straight line as the required condition for each region.

  Specifically, the request setting unit 422 divides the image of each camera 2 into a group of adjacent pixels that are similar in color to each other, and sets each of the divided areas as a required position. For example, an image can be divided by performing clustering using the pixel color and the pixel position as parameters.

  In another embodiment, the request setting unit 422 generates an edge image by filtering the image of each camera 2 using a Sobel filter or the like, and divides the edge image into closed regions each surrounded by an edge. Each can be set to a required position.

  The recommended standing position calculation unit 423 uses the requested position of the camera 2 to be noticed and the camera parameters 410 of the camera 2 to determine the position in the line-of-sight direction from the camera 2 to the requested position in the monitoring space. Is a worker position calculation unit that calculates a blind spot forming position (dead spot forming worker position) that forms a blind spot at the requested position. The recommended standing position calculation unit 423 outputs the information on the blind spot formation position calculated with respect to the requested position to the display communication unit 43 together with the information on the requested position.

  The line-of-sight direction can be obtained from a straight line connecting the camera position and the required position, and an area formed by a group of straight lines connecting the vertical bottom position of the camera and the required position on the reference plane can be obtained as the blind spot forming position. In this embodiment, the blind spot formation position is approximately calculated on the image. That is, the recommended standing position calculation unit 423 calculates a region formed by a group of straight lines connecting the image center and the required position as a blind spot forming position.

  In the precision calibration process, the recommended standing position calculation unit 423 calculates the blind spot forming position in units of areas. Specifically, the recommended standing position calculation unit 423 calculates an area intersecting a straight line connecting the image center and the requested position as a blind spot forming position. Thereby, it is possible to absorb the influence of the error caused by performing the calculation using the approximate position / orientation of the camera.

  In this embodiment, a blind spot forming position is calculated for each camera pair that performs precision calibration processing or measurement processing. In this case, the reference point 13 needs to be photographed simultaneously from the camera pair. Therefore, the recommended standing position calculation unit 423 calculates a blind spot forming position for both cameras forming a camera pair, and calculates a sum area of the calculated multiple blind spot forming positions as a blind spot forming position.

  The recommended standing position calculation unit 423 outputs the standing position of the worker 10 (worker standing position) that can avoid the formation of the blind spot as information on the blind spot forming position. Therefore, the recommended standing position calculation unit 423 calculates a position where the distance from the requested position is less than the irradiation limit distance value and excludes the blind spot forming position as the worker standing position. Further, the position closest to the required position among the worker standing positions can be output as the final worker standing position. When a plurality of requested positions are set, the product area of the worker standing positions calculated for each requested position may be output as a common worker standing position for the set of these requested positions. By doing so, it is possible to present the worker standing position where the reference point 13 can be installed at a plurality of required positions from one place, and the efficiency of the worker is further improved. At this time, it is only necessary to output the worker standing position separately without requesting the required position without the product area.

  The display communication unit 43 is a communication interface that performs communication between the control device 4 and the display unit 5. The display communication unit 43 functions as an output unit that outputs information on the requested position input from the recommended standing position calculation unit 423 and information on the blind spot forming position with respect to the requested position to the display unit 5.

[Operation in Calibration Processing of Camera System 1]
FIG. 6 is a flowchart showing an outline of the operation in the calibration process of the camera system 1. FIG. 7 is a flowchart showing an outline of a recommended standing position presentation process performed by the camera system 1 in the calibration process. The operation in the calibration process of the camera system 1 will be described with reference to FIGS.

  When the camera 2 is installed and the control device 4 and the display unit 5 are turned on, the calibration operation starts. Immediately after startup, the camera parameter 410 is an initial value and is in a state before rough calibration. Each camera 2 sequentially captures its field of view and transmits the captured image to the control device 4.

  The image acquisition unit 40 of the control device 4 sequentially inputs the simultaneously captured images to the image processing unit 42 of the control device 4. In this state, the operator irradiates the floor with the irradiator 3 while moving in the monitoring space.

  Every time a simultaneously captured image is input from the image acquisition unit 40 to the image processing unit 42, the processing from step S1 to step S14 is repeated. Hereinafter, the repeating time unit is referred to as a frame.

  When the image processing unit 42 acquires the simultaneously shot image (S1), the image processing unit 42 operates as a reference point detection unit 420, performs difference processing on each simultaneously shot image, and detects the reference point 13 (reference point image) from the simultaneously shot image. (S2). The reference point detection unit 420 associates the detected coordinates of the reference point 13 on the image with the camera ID of the image and the frame number of the image and stores them in the reference point information 413 of the storage unit 41.

  Next, the image processing unit 42 operates as the general calibration unit 421 to detect a new camera pair (S3). That is, the rough calibration unit 421 searches the reference point information 413, extracts a camera ID pair associated with the reference point 13 having the same frame number as a camera pair, and extracts it with reference to the camera parameter 410. It is confirmed whether or not the camera pair is a new camera pair that is not set as a precision calibration target. The detected camera pair is a camera sharing at least a part of the field of view. It may be added to the extraction condition that there are four or more pairs of reference point images having the same frame number.

  When one or more new camera pairs are detected (in the case of “YES” in S3), the image processing unit 42 operates as a request setting unit 422 to generate request information 411 for rough calibration for each camera pair. The data is stored in the storage unit 41 (S4). Specifically, the request setting unit 422 reads out the camera parameters 410 of each camera 2 constituting the new camera pair, sets nine areas for each camera, and requests position where the reference point 13 is detected in the camera pair. The required condition is set so that the number of cameras is any 4 areas for each camera and 8 areas in total for the camera pairs.

  Subsequently, the request setting unit 422 confirms whether or not each new camera pair satisfies a rough calibration requirement condition (S5). Specifically, the request setting unit 422 refers to the reference point information 413 and request information 411 of the new camera pair, and the area (request request) to which the coordinates of the reference point image associated with the camera ID of the camera pair belong. (Position) is deleted, and it is confirmed from the number of remaining requested positions whether the detected area is 4 areas or more for each camera, and a total of 8 areas or more.

  If the new camera pair includes a camera pair that satisfies the required conditions (in the case of “YES” in S5), the image processing unit 42 operates as the general calibration unit 421 and uses the reference point information 413 of the camera pair. Approximate calibration is performed (S6), the approximate position / orientation values as calibration results are written in the camera parameter 410, and the camera pair is set as a precision calibration target (S7).

  When a new camera pair is set as a precision calibration target, the image processing unit 42 operates as the request setting unit 422, generates precision calibration request information 411 for the camera pair and stores it in the storage unit 41 (S8). That is, the request setting unit 422 reads the camera parameters 410 of each camera 2 constituting the new camera pair, sets nine areas for each camera, and sets the number of requested positions at which the reference point 13 is detected in the camera pair. The required conditions are set to 9 areas for the camera and 18 areas in total for the camera pair.

  Since the area set in step S8 is calculated using the camera parameter 410 roughly calibrated, the area is more suitable for the actual position and orientation of the camera 2 than the area set in step S4.

  When a new camera pair is not detected (in the case of “NO” in S3), the image processing unit 42 skips steps S4 to S8 for the current frame, and proceeds to step S9. If a new camera pair is detected but does not satisfy the rough calibration requirement (in the case of “NO” in S5), the image processing unit 42 skips steps S6 to S8 for the current frame and skips step S9. Proceed to

  Through the processing so far, the camera pairs irradiated by the operator with four or more common fields of view are sequentially listed as precision calibration targets by the image processing unit 42, and request information 411 for each camera pair to be precisely calibrated is set. (S8).

  Next, the image processing unit 42 operates as the request setting unit 422 to check whether there is a camera pair that has not been subjected to precision calibration among the camera pairs that are subject to precision calibration (S9). After updating the required position based on the detection result of the reference point 13 in step S2 (S10), it is confirmed whether or not the camera pair satisfies the required conditions for precision calibration (S11). Specifically, the request setting unit 422 refers to the camera parameter 410 to check whether there is a camera pair that has not been subjected to precision calibration (S9), and the reference point information 413 and request information 411 of the camera pair that has not yet undergone precision calibration. , After deleting the area (required position) to which the coordinates of the reference point 13 associated with the camera ID of the camera pair belong (S10), whether the remaining required positions have become zero, That is, it is confirmed whether the reference point 13 is detected in 18 areas (S11).

  When a camera pair that satisfies the required conditions is included in the camera pair that has not been subjected to the precision calibration (in the case of “YES” in S11), the image processing unit 42 operates as the precision calibration unit 424 and operates as a reference point of the camera pair. Precision calibration is performed using the information 413 (S13), the value of the precise position / orientation as the calibration result is written in the camera parameter 410, and information indicating that the camera has been precision calibrated is stored in the chain determination table 415. It is written and it is confirmed whether or not the precision calibration of all cameras is completed (S14).

  In step S14, the precision calibration unit 424 searches the camera parameter 410 to count the number of cameras that have been precision calibrated, and confirms the connection relationship of the camera pair that has been precision calibrated in the chain determination table 415. If the count matches the number of all cameras and there are no isolated camera pairs, it is determined that the precision calibration of all the cameras has been completed. When the precision calibration of all the cameras is completed (in the case of “YES” in S14), the calibration process is terminated (S15). By the way, if redundant precision calibration targets are set, calibration may be completed even if there are still uncalibrated camera pairs.

  On the other hand, if the count is smaller than the total number of cameras or if there is an isolated camera pair, the process is returned to step S1 as incomplete, and the process for the simultaneously shot image of the next frame of the next frame is performed (S14). "NO" → S1).

  With reference to FIG. 8 to FIG. 10, a processing example when the required condition is satisfied (in the case of “YES” in S11) will be described. As described above, FIGS. 8, 9, and 10 schematically show the data structure of the reference point information 413, the request information 411, and the chain determination table 415, respectively, and show the changes between two consecutive frames. Show. FIGS. 8 (a), 9 (a), and 10 (a) show states in the 41st frame, respectively, and FIGS. 8 (b), 9 (b), and 10 (b) show the states in the 41st frame, respectively. The state in 42 frames is shown.

  The reference point information 413 in the 41st frame stores information on the reference point 13 detected by the reference point detection unit 420 in the shooting in the 1st to 41st frames (FIG. 8A). From the 37th frame to the 41st frame, the reference point 13 is detected in the common visual field of the cameras B and C.

  In addition, the rough calibration unit 421 roughly calibrates the camera pairs “A, B”, “A, C”, “B, C”, and “C, D” until the 41st frame, and as a result, the chain determination table. It is recorded in 415 that these camera pairs are set as precision calibration targets (FIG. 10A). Further, the chain determination table 415 records that the precision calibration by the precision calibration unit 424 has been completed for the camera pair “A, B” and the precision calibration has not been completed for the other camera pairs (FIG. 10 (a)).

  Of the 18 areas of areas B1 to B9 and areas C1 to C9 set in the request information 411 as the initial request position by the request setting unit 422 for the camera pair “B, C” at the time of the 41st frame, the area B3 and Two areas of C4 remain as request positions (FIG. 9A). That is, when the reference point 13 is detected in the areas B3 and C4 in the subsequent frames, the camera pair “B, C” can be precisely calibrated. It should be noted that the requested position relating to the camera pair “A, B” that has completed precision calibration has already been deleted from the request information 411.

  Next, in the 42nd frame, the reference point detection unit 420 newly detects a reference point image from each of the images of the cameras B and C, and the detection result is added to the reference point information 413 (FIG. 8B). The request setting unit 422 confirms that the reference point image newly detected on the camera B side is in the area B3 and that the reference point image newly detected on the camera C side is in the area C4. Then, the area B3 and the area C4 that were the requested positions of the camera pair “B, C” were deleted from the request information 411 (step S10, FIG. 9B). As a result of the update of the required position, it is determined that the reference points have been detected at all the required positions necessary for the precision calibration of the camera pair “B, C” (“YES” determination in step S11), and the precision calibration unit 424 is determined. Performed precision calibration using the reference point information 413 of the camera pair “B, C”, and recorded that the precision calibration was completed in the chain determination table 415 (step S13, FIG. 10B).

  When the precise calibration of the camera pair “B, C” is completed, there are two camera pairs, “A, B” and “B, C”, for which the precision calibration has been completed (FIG. 10B). . That is, the precision calibration is completed for the three cameras 2 with the camera IDs “A”, “B”, and “C”. Here, the cameras “A” and “C” are connected via the camera “B”, and it is determined based on the chain determination table 415 that these three cameras 2 are connected.

  At this time, since the five cameras 2 having the camera IDs “D” to “H” remain as incomplete precision calibration, it is not yet determined in step S14 that all calibrations have been completed. Since it can be determined from the chain determination table 415 that the cameras “A”, “B”, and “C” are in a connection relationship, precise calibration of the camera pair “A, C” has not been completed in the chain determination table 415. Even if it is recorded, it is determined that all calibrations have been completed. In other words, in this example, the precise calibration of the camera pair “A, C” is not an indispensable condition for the completion of the entire calibration when determining the completion of the entire calibration.

  If there is no camera pair that has not been subjected to precision calibration in step S9 (in the case of “NO” in S9), the precision calibration unit 424 skips steps S10 to S13 and confirms completion of step S14.

  When there is no camera pair that satisfies the request information 411 in step S11 (in the case of “NO” in S11), the image processing unit 42 operates as the recommended standing position calculation unit 423, and the reference point to the request position A worker's standing position suitable for installation is calculated and presented to the worker (S12).

  The recommended standing position presentation process (S12) will be described with reference to FIG.

  First, the recommended standing position calculation unit 423 selects one presentation camera pair that presents the recommended standing position on the display unit 5 from the camera pairs for which precision calibration has not been completed (S20). Specifically, the recommended standing position calculation unit 423 suppresses the flapping of the presentation by selecting the presentation camera pair selected one frame before. As a result, the camera pair that the worker is currently irradiating the common field of view is selected, so that unnecessary movement of the worker can be reduced. The recommended standing position calculation unit 423 stores the presentation camera pair selected in the current frame in the storage unit 41 in preparation for selection in the next frame.

  On the other hand, when a new reference point 13 is not detected for a certain period from the pair of presentation cameras selected one frame before, the pair of presentation cameras is changed. In other words, in this case, the recommended standing position calculation unit 423 refers to the reference point information 413 and selects a camera pair that has not been precisely calibrated for the camera ID associated with the reference point image with the latest frame number.

  Next, the recommended standing position calculation unit 423 refers to the request information 411 of the presented camera pair, and presents the recommended standing position for the camera having the larger remaining number of requested positions among the cameras constituting the camera pair. A camera is selected (S21). In preparation for selection in the next frame, the recommended standing position calculation unit 423 stores the presentation camera selected in the current frame in the storage unit 41. When the number of requested positions is the same, fluttering of presentation is suppressed by selecting the presentation camera selected one frame before. If one of the pair of presentation cameras cannot be selected as the presentation camera even if the number of requested positions and the conditions of the previous presentation camera are imposed, the camera with the younger camera ID is selected.

  The subsequent processing S22 to S25 of the recommended standing position calculation unit 423 will be described with reference to specific examples illustrated in FIGS. In the examples shown in these drawings, the presentation camera pair is “B, C”, and the presentation camera is “B”. FIG. 11 shows an example of the blind spot formation position obtained directly in the image areas 22b and 22c of the cameras 2B and 2C, and FIG. 12 shows an example of calculation of the worker standing position corresponding to FIG. FIG. 13 shows an example of narrowing the recommended standing position. 11A and 12A show the image areas 22b and 23b of the camera 2B, and FIGS. 11B and 12B show the image areas 22c and 23c of the camera 2C, respectively. .

  The recommended standing position calculation unit 423 projects the requested position to be processed on the presentation camera side onto the other side of the presentation camera pair (S22), and the recommended standing position calculation unit 423 is based on the requested position and its projection image. A blind spot formation position is calculated (S23).

  Specifically, the recommended standing position calculation unit 423 determines an area intersecting a line of sight connecting the position of the presentation camera (image center of the presentation camera) and the requested position as a blind spot formation position on the presentation camera side. In FIG. 11, the area B3 on the camera 2B side (within the thick line in the image area 22b) is the requested position to be processed. On the camera 2B side, an area 220 where the straight line connecting the position of the camera 2B and the required position B3 intersects (an oblique lattice portion of the image area 22b) is calculated as a direct blind spot forming position, and an area B1 including the area 220 is an area. It is calculated as a blind spot formation position in units.

  Next, the recommended standing position calculation unit 423 performs projection by coordinate conversion using the camera parameter 410 of the presenting camera pair, and intersects the line of sight connecting the position of the other camera (image center of the other camera) and the projected image of the requested position. The area to be performed is determined as the blind spot forming position on the other camera side. In the example shown in FIG. 11, the blind spot formation position for the area 221 corresponding to the requested position B3 (the hatched portion of the image area 22c) is calculated on the camera 2C side. Specifically, the projected position (area 221) is calculated by projecting the required position B3 on the camera 2C side, and an area 222 (an oblique lattice of the image area 22c) where a straight line connecting the projected image and the position of the camera 2C intersects. (Part) is calculated as a direct blind spot forming position on the camera 2C side. Then, as shown in FIG. 12, areas C1, C3 to C5 including the region 222 (within the thick line of the image region 23c) are calculated as blind spot formation positions in area units.

  Subsequently, the recommended standing position calculation unit 423 calculates an area excluding the blind spot formation position calculated in step S23 as a recommended standing position (S24). Specifically, the recommended standing position calculation unit 423 backprojects the blind spot formation position on the other camera side to the presentation camera side by coordinate conversion using the camera parameter 410 of the presentation camera pair, and displays the presentation camera from the entire image on the presentation camera side. A region excluding the blind spot forming position on the side and the back projection image of the blind spot forming position on the other camera side is calculated as a recommended standing position. In the example shown in FIG. 12, areas C1, C3 to C5, which are blind spot formation positions on the camera 2C side, are projected on the camera 2B side to calculate a projected image. The reference point 13 of the required position needs to be photographed simultaneously from the camera pair. From this condition, on the camera 2B side, the area B1 that is the blind spot forming position in area units on the camera 2B side and the area on the camera 2C side The sum area 230 (projected lattice portion of the image area 23b) with the projected image of the unit blind spot forming position becomes a prohibited area that prohibits the presence of the worker when setting the reference point with respect to the required position B3. 231 (the horizontal line portion of the image area 23b) is the recommended standing position of the operator on the camera 2B side.

  Further, the recommended standing position calculation unit 423 narrows down the recommended standing position (S25). For example, the recommended standing position calculation unit 423 obtains the center of gravity of the requested position on the presentation camera side, sets a circle centered on the center of gravity while sequentially increasing its radius, and the recommended standing position calculated in step S24. The process of calculating the overlapping area is repeated until the number of overlapping areas reaches four or until the area of the overlapping area reaches 50% of the area of the recommended standing position. Then, the recommended standing position calculation unit 423 sets the overlapping area when the overlapping area becomes four or when the area of the overlapping area reaches 50% of the recommended standing position as the final recommended standing position. In the example shown in FIG. 13, the recommended standing positions are further narrowed down within a certain distance (in the dotted line) from the prohibited region 230, and four regions 232 (regions in the horizontal line portion in FIG. 13). Is obtained as the recommended standing position.

  The recommended standing position calculation unit 423 generates a composite image by superimposing the requested position, the recommended standing position narrowed down in step S25, and the message on the image of the presentation camera (S26), and outputs the composite image to the display communication unit 43. (S27).

  The display communication unit 43 to which the composite image is input transmits the composite image to the display unit 5, and the display unit 5 that has received the composite image displays the composite image on the display. FIG. 14 is an example of a display screen presenting a recommended standing position on the display unit 5 corresponding to the example of FIG. 13. The recommended standing position (area 240), requested position B 3 (area 241), and message 242 are displayed by the camera 2 B. A composite image synthesized with the captured image is displayed. Incidentally, the recommended standing position and the required position B3 are preferably displayed so as to be easily identified by color coding or the like. For example, in FIG. 14, the recommended standing position is displayed in red and the required position B3 is displayed in green.

  FIG. 15 is another example of a display screen presenting a recommended standing position on the display unit 5 corresponding to the example of FIG. FIG. 15A shows an example of displaying the prohibited area 243, that is, the blind spot forming position, together with the specified requested position (area 241). By displaying the blind spot forming position, it is indicated to the worker 10 that the outside is the recommended standing position. FIG. 15B displays and presents a belt-like region 244 surrounding the blind spot forming position as a recommended standing position.

  Thus, the required position and the recommended standing position are presented to the worker, and the worker stands at the recommended standing position and irradiates the requested position. By doing so, the operator can efficiently set the reference point 13 while avoiding the failure to form a blind spot at the required position, so that the calibration work can be completed with a small work load.

  Further, the recommended standing position is narrowed down to a position close to the required position by the process of step S25. Since the brightness of the light image formed at the requested position increases as irradiation is performed from a position closer to the requested position, the standing position where the reference point 13 is easily detected in addition to the standing position that does not form a blind spot at the requested position is operated. Can be presented to the person. As a result, a failure that cannot be detected even if the reference point 13 can be photographed can be further avoided, so that the calibration work can be completed with a small work load.

  Further, by setting the recommended standing position as a plurality of overlapping areas or overlapping areas having a certain area in the narrowing down in step S25, it is possible to prevent only the area that is not the floor surface such as the object 12 from being the recommended standing position. Furthermore, the operator can determine the position where the operator actually stands by being superimposed on the photographed image in the composition of step S26.

  Depending on the arrangement of the camera 2, there may be no recommended standing position when there are many required positions. At this time, only the required position may be combined with the captured image and output.

  When the recommended standing position is presented (S12), the image processing unit 42 returns the process to step S1 in FIG. 6 and performs the process on the simultaneously captured image of the next frame.

  Note that the recommended standing position may be narrowed down to a band-like region around the blind spot forming position by narrowing down in step S25. In this case, in the example of the cameras 2B and 2C described above, the recommended standing position is an area indicated by a horizontal line in FIG. The width of the band should be about the width of a person.

  In the above-described embodiment, the required position and the recommended standing position are displayed on the display unit 5. On the other hand, by displaying the required position and the blind spot forming position on the display unit 5, the outside of the blind spot forming position may be presented as the recommended standing position. This corresponds to the display example of FIG. FIG. 16 is a flowchart showing an outline of the recommended standing position presentation process in this case. Steps S20 to S23 are the same as those in FIG. Subsequent to step S23, the requested position and the blind spot forming position are combined with the image of the presentation camera (S28), and the combined image is output to the display unit 5 via the display communication unit 43 (S29).

[Operation in 3D Measurement Processing of Camera System 1]
After completing the calibration process, the camera system 1 shifts to a measurement process for three-dimensional measurement of the monitoring space.

  FIG. 17 is a flowchart showing an outline of the operation in the measurement process of the camera system 1. With reference to FIG. 17, the operation | movement in the measurement process of the camera system 1 is demonstrated.

  When the three-dimensional measurement process is executed, each camera 2 sequentially captures its own field of view and transmits the captured image to the control device 4 as in the calibration process described above. The image acquisition unit 40 of the control device 4 sequentially inputs the simultaneously captured images to the image processing unit 42 of the control device 4. In this state, the worker irradiates the object 12 such as a fixture existing in the monitoring space by the irradiator 3 while moving in the monitoring space.

  When the image processing unit 42 acquires the simultaneously shot image (S50), the image processing unit 42 operates as the request setting unit 422, divides each image into a group of adjacent pixels having similar colors, and generates divided regions (S51). Request information 411 is generated and stored in the storage unit 41 for each region as a request position for requesting detection of three or more reference points 13 (S52). FIG. 18 is a schematic diagram of an image area showing an example of setting the request information 411 in the measurement process. FIG. 18 shows an example of the request information 411 for the cameras 2B and 2C. As shown in the image area 24b, for the camera 2B, the request positions b3 and b4 on the surface of the fixture and the request positions b1 and b1 on the wall surface. b5 and the required position b2 are set on the floor. Further, as shown in the image area 24c, for the camera 2C, the required positions c2, c3, and c4 are set on the surface of the fixture, the required positions c5 and c6 are set on the wall surface, and the required position c1 is set on the floor surface. .

  After the generation of the request information 411, the image processing unit 42 repeats the processing of steps S53 to S62 every time a simultaneously shot image is input.

  First, when the image processing unit 42 obtains the simultaneous captured image (S53), the image processing unit 42 operates as a reference point detection unit 420, performs difference processing on each simultaneous captured image, detects the reference point 13 from the simultaneous captured image, and performs the reference processing. The camera ID of the image and the frame number of the image are associated with the coordinates on the image of the point 13 and stored in the reference point information 413 of the storage unit 41 (S54).

  Next, the image processing unit 42 operates as the request setting unit 422 and refers to the reference point information 413 and the request information 411 to check whether or not there is a request position where three or more reference points 13 are detected ( S55).

  If there is a corresponding request position (in the case of “YES” in S55), the image processing unit 42 operates as the object measurement unit 425, and applies the principle of triangulation to the reference point 13 detected at the request position. Then, three-dimensional measurement is performed (S57), and the plane of the divided area that is the required position is derived (S58). Then, the derived plane information is stored in the object measurement information 414 of the storage unit 41 (S59). At this time, the request setting unit 422 deletes the request position where the three-dimensional measurement is performed from the request information 411.

  On the other hand, when there is no request position where three or more reference points 13 are detected (“NO” in S55), the image processing unit 42 operates as the recommended standing position calculation unit 423, and three reference points 13 are present. The recommended standing position with respect to the requested position that is less than the recommended position is presented (S56). The process in step S56 is the same as the process described with reference to FIG. As a result, as in the case of the calibration process, the required position and the recommended standing position are presented to the worker, and the efficiency of the operation of detecting the reference point 13 from the required position is improved. In the measurement process that requires detection of a plurality of reference points 13 at each required position, the screen display of the display unit 5 should be performed so that the operator can know the number of reference points 13 that have already been detected for each required position. You can also. For example, the area of the requested position and the number of detections in the area can be displayed on the screen, or the color intensity can be changed for each area according to the number of detections. As a result, not only the area displayed as the requested position is no longer the requested position but is no longer displayed on the display unit 5, and more detailed work progress can be presented to the worker, thereby improving work efficiency. .

  Here, the measurement process may include a required position where it is difficult to install the reference point 13. For example, when the height of the object 12 is high, it is difficult for the worker 10 standing on the floor to irradiate the required positions b3 and c2 that are the upper surface of the object 12. In such a case, the operator inputs a request release by designating a requested position that is difficult to install in the display image of the display unit 5. In step S60, the request setting unit 422 confirms the request cancel input, and if there is an input, deletes the corresponding request position from the request information 411 (“YES” in S60 → S61). On the other hand, if there is no request release input, step S61 is skipped (in the case of “NO” in S60).

  Subsequently, the object measuring unit 425 confirms the request information 411 (S62), and ends the measurement process if there is no request position information (“YES” in S62 → S63). On the other hand, if the requested position remains (“NO” in S62), the image processing unit 42 returns the process to step S53 and performs the process on the simultaneously captured image of the next frame.

[Modification]
(1) In the above-described embodiment, an example in which the XY coordinates of the camera and the approximate calibration value in the optical axis direction are set using the reference point 13 has been shown. It may be set and rough calibration may be omitted.

  (2) The irradiator 3 may be a light source that outputs pattern light such as a predetermined cross pattern instead of the above-described output of spot light. In this case, the reference point detection unit 420 may detect a predetermined shape point in the pattern, such as an intersection in the cross pattern, as the reference point 13 instead of the center of gravity of the optical image.

  (3) In the above-described embodiment, the reference point 13 is set by irradiating with the laser pointer carried by the worker 10. However, the present invention is not limited thereto, and the worker can form a blind spot at the required position. The present invention can also be applied to the configuration. FIG. 19 is a schematic diagram showing an example of another apparatus for forming the reference point 13 by an image. For example, when the operator 10 moves the roller 32 (FIG. 19A), the carriage 33 (FIG. 19B), etc., to which the light emitting device 31, which is the reference point 13, is attached, the reference point 13 is installed. In addition, the present invention is suitable.

  (4) In the embodiment described above, the recommended standing position calculation unit 423 calculates and outputs the recommended standing position to the worker 10 so as not to be a blind spot from both cameras of the camera pair. In another simple embodiment, the recommended standing position calculation unit 423 omits the blind spot formation position calculation on the other camera side, and calculates and outputs the recommended standing position only from the blind spot formation position on the presentation camera side. This configuration is advantageous in that the amount of calculation can be reduced and the possibility that the recommended standing position is lost is reduced, and it is effective when there are many required positions. In this configuration, since the recommended standing position can form a blind spot at a part of the required position with respect to the other camera, the possibility of resetting the reference point remains correspondingly, but depending on the position of the newly set reference point 13 The reference point 13 is detected from the photographed image of the other camera, and resetting is unnecessary. Therefore, although not as much as in the above embodiment, an effect of improving work efficiency can be obtained. Furthermore, when there are a plurality of cameras (other cameras) having a common field of view with the camera of interest, it can be expected that one of the other cameras will detect the reference point 13 simultaneously with the camera of interest, and resetting may be necessary. Workability is improved.

  (5) The irradiator 3 has a function of measuring its own position, detecting the irradiation direction and tilt angle of the light beam, and measuring the distance to the irradiation point by laser distance measurement, etc., and the irradiator based on the information In the configuration in which 3 or the control device 4 calculates the position of the irradiation point, the camera can be calibrated by detecting the reference point with a single camera. In this configuration, the request setting unit 422 does not set a request for each camera pair, but sets a required position (for example, the above-described nine areas) for each camera 2 to avoid a blind spot formation with respect to the camera of interest. It is sufficient to calculate and output

  In addition, in the configurations (4) and (5), which are sufficient to avoid the blind spot with a single camera, the method of presenting the blind spot forming position to the worker 10 can be performed in the same manner as in the above embodiment. For example, the display communication unit 43 may output the blind spot formation position itself as information on the blind spot formation position, or display a position other than the blind spot formation position, that is, the worker standing position that can avoid the formation of the blind spot as the same information. May be.

  (6) As described in the above embodiment, the calibration process of the camera 2 generally requires that four or more reference points 13 that are not on the same straight line are detected from the simultaneously captured image. However, if some of the degrees of freedom of the position and orientation of the camera 2 are known and calibration is not necessary, the minimum number of reference points 13 required for calibration processing is less than 4. You can also.

  (7) In the above embodiment, the output of the blind spot forming worker position is an image output from the display communication unit 43 to the display of the display unit 5, but the output form is not limited to this. For example, it can be voice output or printing on a medium such as paper.

  1 camera system, 2A, 2B, 2C, 2D, 2E, 2F, 2G, 2H camera, 3 irradiator, 4 control device, 5 display unit, 10 operator, 11 reference plane, 12 object, 40 image acquisition unit , 41 storage unit, 42 image processing unit, 43 display communication unit, 400 camera communication unit, 401 synchronization control unit, 410 camera parameter, 411 request information, 412 irradiation limit value, 413 reference point information, 414 object measurement information, 420 reference Point detection unit, 421 rough calibration unit, 422 requirement setting unit, 423 recommended standing position calculation unit, 424 precision calibration unit, 425 object measurement unit.

Claims (5)

A camera configured to include one or a plurality of cameras that capture an image of a predetermined space, a control device that processes an image captured by the camera, and a reference point that can be captured by the camera and is set by an operator A system,
The control device includes:
A storage unit that stores at least camera parameters including the position and orientation of the camera in the space and a required position in the space where the reference point needs to be installed;
A reference point detection unit for detecting the reference point from an image captured by the camera;
Using the camera parameters and the required position of the camera, the position in the line-of-sight direction from the camera to the required position is defined as the blind spot forming worker position that cannot capture the required position from the camera when the operator exists. An operator position calculation unit to calculate,
An output unit for outputting the blind spot forming worker position;
A camera system comprising:   The camera system according to claim 1, wherein the output unit outputs the requested position. A plurality of the cameras are arranged to have a common field of view in the space,
The control device further includes a calibration unit that calibrates the camera parameter using the reference point,
In the calibration unit, when the reference point detection unit detects a reference point image as the reference point in each of the images simultaneously captured by the plurality of cameras, the plurality of reference point images are at the same position in the space. Calibrating the camera parameters by applying a constraint condition with
The camera system according to claim 1 or 2, characterized by the above.   The control device requests to delete the requested position where the reference point is photographed from the storage unit only when the reference point detection unit detects the reference point existing at the requested position in the common visual field. The camera system according to claim 3, further comprising a position setting unit.   The control device further includes a required position setting unit that sets the required position in units of segmented areas obtained by radially dividing an image captured by the camera from an intersection between the optical axis of the camera and the imaging plane for each predetermined angle. The camera system according to any one of claims 1 to 4, characterized in that:

Images