JP2024136150A - Information processing device, information processing system, information processing method, and program - Google Patents

Abstract

To provide an information processing apparatus, an information processing system, an information processing method, and a program for facilitating calibration work in a system for specifying the position of an object using an imaging device for imaging an object moved by a moving device and three-dimensional position information acquired by another means.SOLUTION: An information processing apparatus 100 is provided with: a first acquisition unit 402 for acquiring first object position information, which is two-dimensional position information of an object, from a first image captured by a first imaging device; a second acquisition unit 403 for acquiring three-dimensional position information of the object; a first calculation unit 404 for calculating second object position information, which is two-dimensional position information on the first image, from the three-dimensional position information of the object acquired by the second acquisition unit 403; a second calculation unit 405 for calculating a distance between a first locus of the object based on the first object position information and a second locus of the object based on the second object position information; and a correction unit 406 for performing correction processing for correcting the distance between the first locus and the second locus.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing device, an information processing system, an information processing method, and a program.

There is known an information processing device that processes position information of objects such as cargo or pallets transported by a mobile object such as a forklift (see, for example, Patent Document 1).

In addition, an information processing device is known that stably obtains the position of a moving body based on the position of the moving body determined using a camera mounted on the moving body, such as a transport vehicle that transports an object, and the position of the moving body determined using a camera installed in a location where the moving body can be observed (see, for example, Patent Document 2).

For example, as in the technology disclosed in Patent Document 2, when identifying the position of a moving object using a camera mounted on the moving object and another camera installed in a location where the moving object can be observed, it is necessary to grasp the relationship between the position and attitude of the camera and the other camera in advance. Then, in order to grasp the relationship between the positions and attitudes of multiple imaging devices (cameras), a calibration work of the imaging devices (cameras) is required. Conventionally, the calibration work of the imaging device (camera) requires adjusting the parameters of the imaging device (camera) by photographing a planar marker at various positions and attitudes with the imaging device (camera) to be calibrated, and there is a problem that it takes a lot of time, such as continuing trial and adjustment until the desired imaging device (camera) state is set.

Note that this problem is not limited to cases where the position of a moving object is to be determined, but is also common to various systems that determine the position of an object using, for example, an imaging device that captures an image of an object moved by a moving device that moves the object, and three-dimensional position information of the object obtained by another means.

One embodiment of the present invention has been made in consideration of the above problems, and facilitates calibration (correction) in a system that identifies the position of an object using an imaging device that captures an image of an object moved by a moving device and three-dimensional position information acquired by another means.

In order to solve the above problem, an information processing device according to one embodiment of the present invention is an information processing device capable of communicating with a first imaging device that captures an object moved by a moving device, and includes a first acquisition unit that acquires first object position information, which is two-dimensional position information of the object, from a first image captured by the first imaging device, a second acquisition unit that acquires three-dimensional position information of the object, a first calculation unit that calculates second object position information, which is two-dimensional position information on the first image, from the three-dimensional position information of the object acquired by the second acquisition unit, a second calculation unit that calculates the distance between a first trajectory of the object based on the first object position information and a second trajectory of the object based on the second object position information, and a correction unit that executes a correction process to correct the distance between the first trajectory and the second trajectory.

According to one embodiment of the present invention, in a system that identifies the position of an object using an imaging device that captures an image of an object moved by a moving device and three-dimensional position information acquired by another means, calibration work is made easier.

1 is a diagram illustrating an example of a system configuration of an information processing system according to a first embodiment. FIG. 2 is a diagram showing an example of a first image according to the first embodiment; FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer according to an embodiment. FIG. 1 is a diagram illustrating an example of a functional configuration of an information processing system according to an embodiment. FIG. 11 is a diagram illustrating another example of a functional configuration of an information processing system according to an embodiment. 10 is a flowchart illustrating an example of a correction process according to an embodiment. FIG. 11 is a diagram showing an image of reception timing of a captured image according to an embodiment. FIG. 13 is a diagram showing an image of two trajectories according to an embodiment. FIG. 13 is a diagram illustrating an image of distance minimization according to an embodiment. 11 is a flowchart illustrating an example of a process for calculating a distance between two trajectories according to an embodiment. 11 is a flowchart illustrating an example of a process for calculating a distance between a polygonal line and a point according to an embodiment. FIG. 2 is a diagram for explaining first to third regions according to an embodiment. FIG. 13 is a diagram for explaining a modified example. FIG. 1 is a diagram for explaining a positional relationship between a second imaging device and an object according to the first embodiment; FIG. 2 is a diagram for explaining a positional relationship between the second imaging device and an object according to the first embodiment. FIG. 11 is a diagram illustrating an example of a system configuration of an information processing system according to a second embodiment. FIG. 13 is a diagram illustrating an example of a system configuration of an information processing system according to a third embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings. In each drawing, the same components are given the same reference numerals, and duplicated explanations will be omitted.

<System Configuration>
[First embodiment]
Fig. 1 is a diagram showing an example of a system configuration of an information processing system according to a first embodiment. In the example of Fig. 1, the information processing system 1 includes a first imaging device 10, a mobile device 20, a second imaging device 21 mounted on the mobile device 20, an information processing device 100, and the like.

The first imaging device 10 is, for example, a fixed camera or the like that is installed on a ceiling 2, captures an image of an object 22 moving with the mobile device 20, and transmits the captured image (hereinafter referred to as a first image) to the information processing device 100. Note that the first imaging device 10 is not limited to being installed on the ceiling 2, and may be installed on, for example, a wall, a pillar, or a support. Here, the following explanation will be given assuming that the first imaging device 10 is installed on the ceiling 2 of, for example, a warehouse or the like.

The moving device 20 is, for example, a transport device such as a forklift that holds an object 22, such as a baggage or a pallet carrying baggage, and transports the object 22 while holding it. Note that the moving device 20 is not limited to a forklift, and may be, for example, a robot that moves the object 22, a drone, etc. Here, the following explanation will be given assuming that the moving device 20 is a forklift that moves the object 22 in a warehouse, etc.

The second imaging device 21 is mounted on the mobile device 20 and includes one or more cameras that capture images of the surroundings of the mobile device 20 and objects 22, etc.

As an example, the mobile device 20 has a communication function such as wireless LAN (Local Area Network) communication, and a computer configuration. Here, the "mobile device 20" includes the second imaging device 21, and the communication function and computer configuration may be possessed by the second imaging device 21, or may be provided in a part of the mobile device 20 other than the second imaging device 21.

The mobile device 20 (or the second imaging device 21) measures three-dimensional position information indicating the position of the second imaging device 21 for each video frame from an image captured by the second imaging device 21 (hereinafter referred to as the second image) using, for example, a known Visual SLAM (Simultaneous Localization and Mapping) technique. In addition, the mobile device 20 (or the second imaging device 21) calculates three-dimensional position information of the object 22 from the three-dimensional position information of the second imaging device 21 and the positional relationship (relative position) between the second imaging device 21 and the object 22, and transmits the calculated three-dimensional position information of the object 22 to the information processing device 100.

Here, SLAM is a known technology that acquires three-dimensional position information of surrounding three-dimensional objects as point cloud information and estimates the position of the vehicle itself and surrounding three-dimensional objects. Visual SLAM (hereinafter referred to as VSLAM) is a known technology that performs SLAM using images captured by a camera. Note that the mobile device 20 may measure three-dimensional position information of the second imaging device 21 using other positioning technologies, such as radio waves or ultrasound.

As another example, the mobile device 20 (or the second imaging device 21) may have a communication function such as wireless LAN communication, and transmit the second image captured by the second imaging device 21 to the information processing device 100. In this case, the information processing device 100 executes a position information calculation process to calculate three-dimensional position information of the object 22 based on the second image captured by the second imaging device 21.

The information processing device 100 is an information processing device having a computer configuration, or a system including multiple computers. The information processing device 100 is also communicatively connected to the first imaging device 10 and the mobile device 20 (or the second imaging device 21) via, for example, a wireless LAN or the like.

By executing a specific program, the information processing device 100 calibrates (corrects) the relationship between the position and orientation of the first imaging device 10 and the second imaging device in a system that identifies the position of an object 22 using the first imaging device 10 and the second imaging device.

(Processing Overview)
The information processing device 100 acquires two-dimensional position information of the object 22 (hereinafter referred to as first object position information) based on the first image captured by the first imaging device 10 .

FIG. 2 is a diagram showing an image of a first image according to the first embodiment. For example, as shown in FIG. 2, a marker 210 is provided on the object 22 so as to appear in the first image 200 captured by the first imaging device 10. The information processing device 100 acquires two-dimensional position information (two-dimensional coordinates) of the object 22, for example, by taking the center position 211 of the marker 210 appearing in the first image 200 as the position of the object 22 on the first image 200. Note that the information processing device 100 can identify the center position 211 of the marker 210 by using, for example, a known AR (Augmented Reality) marker technology.

The information processing device 100 also acquires three-dimensional position information (three-dimensional coordinates) of the object 22 based on the second image captured by the second imaging device 21. For example, the information processing device 100 acquires the three-dimensional position information of the object 22 calculated by the mobile device 20 (or the second imaging device 21) and transmitted to the information processing device 100. Note that the information processing device 100 may calculate the three-dimensional position information of the object 22 from the second image transmitted by the mobile device 20.

Furthermore, the information processing device 100 calculates two-dimensional position information (hereinafter referred to as second object position information) obtained by projecting the acquired three-dimensional position information of the object 22 into two dimensions. For example, the information processing device 100 converts the three-dimensional position information of the object 22 into the camera coordinates of the first imaging device 10 using external parameters indicating the position and orientation of the first imaging device 10. Furthermore, the information processing device 100 further converts the position information converted into the camera coordinates into second object position information, which is two-dimensional position information on the first image, using internal parameters of the first imaging device 10. Note that the internal parameters of the first imaging device 10 are assumed to be known in advance through calibration, etc.

Then, the information processing device 100 executes a correction process to minimize the distance between the trajectory of the object 22 based on the first object position information (hereinafter referred to as the first trajectory) and the trajectory of the object 22 based on the second object position information (hereinafter referred to as the second trajectory). This calibrates the external parameters indicating the position and attitude of the first imaging device 10, and makes it possible to more accurately obtain, for example, two-dimensional position information on the first image from the three-dimensional position information of the object 22 acquired from the second image captured by the second imaging device 21.

By performing the above processing, the information processing system 1 can easily calibrate the relationship between the position and attitude of the first imaging device 10 and the second imaging device 21, for example, from images (first image and second image) of a moving device 20, such as a forklift, traveling.

Note that the system configuration of the information processing system 1 shown in FIG. 1 is an example. For example, the information processing system 1 may have multiple first imaging devices 10. In this case, the information processing device 100 performs the same calibration for each of the multiple first imaging devices 10.

<Hardware Configuration>
The information processing apparatus 100 includes, for example, the hardware configuration of a computer 300 as shown in Fig. 3. Alternatively, the information processing apparatus 100 may include a plurality of computers 300.

(Computer hardware configuration)
Fig. 3 is a diagram showing an example of a hardware configuration of a computer according to an embodiment. As shown in Fig. 3, the computer 300 includes, for example, a central processing unit (CPU) 301, a read only memory (ROM) 302, a random access memory (RAM) 303, a hard disk (HD) 304, a hard disk drive (HDD) controller 305, a display 306, an external device connection interface (I/F) 307, a network I/F 308, a keyboard 309, a pointing device 310, a digital versatile disk rewritable (DVD-RW) drive 312, a media I/F 314, and a bus line 315.

Of these, the CPU 301 controls the overall operation of the computer 300. The ROM 302 stores, for example, programs used to start up the computer 300, such as startup files, and data. The RAM 303 is used, for example, as a work area for the CPU 301.

The HD 304 stores, for example, programs such as an OS (Operating System), applications, and device drivers, as well as various data. The HDD controller 305 controls the reading and writing of various data from the HD 304 under the control of the CPU 301. The HD 304 and the HDD controller 305 are examples of storage devices included in the computer 300.

The display 306 displays various information such as a cursor, a menu, a window, characters, or an image. The display 306 may be provided outside the computer 300. The external device connection I/F 307 is an interface for connecting various external devices such as an external memory to the computer 300. The network I/F 308 is an interface for data communication using a communication network N, for example. The network I/F 308 is an example of a communication device provided in the computer 300.

The keyboard 309 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 310 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. Note that the keyboard 309 and the pointing device 310 may be provided outside the computer 300.

The DVD-RW drive 312 controls the reading and writing of various data from the DVD-RW 311, which is an example of a removable recording medium. Note that the DVD-RW 311 is not limited to a DVD-RW, and may be other removable recording media. The media I/F 314 controls the reading and writing (storing) of data from the media 313, such as a flash memory. The bus line 315 includes an address bus, a data bus, and various control signals for electrically connecting the above components.

The configuration of computer 300 shown in FIG. 3 is an example. Computer 300 may have any configuration as long as it has, for example, CPU 301, ROM 302, RAM 303, a storage device, a communication device, and a bus line 315.

In addition, if the mobile device 20 (or the second imaging device) is equipped with a computer, the computer has, for example, a CPU 301, a ROM 302, a RAM 303, a storage device, a communication device, and a bus line 315.

<Functional configuration>
Fig. 4 is a diagram showing an example of a functional configuration of an information processing system according to an embodiment. In the example of Fig. 4, an information processing device 100 is communicably connected to a first imaging device 10 and a mobile device 20 via a communication network N such as a LAN.

(Functional configuration of mobile device)
The mobile device 20 realizes, for example, a communication unit 411, an imaging unit 412, and a position information calculation unit 413 by executing a predetermined program by a computer included in the mobile device 20 (or the second imaging device 21). Furthermore, the mobile device 20 realizes a memory unit 414 by a storage device or the like included in the mobile device 20 (or the second imaging device 21). Note that each of the above functional configurations may be possessed by the mobile device 20 equipped with the second imaging device 21. For example, some or all of the functional configurations of the mobile device 10 shown in FIG. 4 may be possessed by the second imaging device 21.

The communication unit 411 uses a communication device (e.g., a wireless LAN device) provided in the mobile device 20 to connect the mobile device 20 to the communication network N and executes communication processing to communicate with the information processing device 100.

The imaging unit 412 performs imaging processing, for example, using the second imaging device 21 to capture a second captured image of the surroundings of the mobile device 20.

The position information calculation unit 413 executes a position information calculation process to calculate three-dimensional position information indicating the position of the object 22 based on the second image captured by the imaging unit 412. For example, the position information calculation unit 413 calculates three-dimensional position information of the object 22 for each frame based on the second image (moving image) captured by the imaging unit 412.

As a suitable example, the position information calculation unit 413 uses known VSLAM technology to measure three-dimensional position information indicating the position of the second imaging device 21 for each video frame. In addition, the position information calculation unit 413 calculates three-dimensional position information of the object 22 from the three-dimensional position information of the second imaging device 21 and the relative position between the second imaging device 21 and the object 22 acquired in advance.

The position information calculation unit 413 may measure three-dimensional position information of the second imaging device 21 using other positioning technologies, such as radio waves or ultrasound, and calculate the three-dimensional position information of the object 22 from the relative positions between the second imaging device 21 and the object 22 that have been acquired in advance.

In addition, the position information calculation unit 413 transmits the calculated three-dimensional position information of the object 22 to the information processing device 100 via the communication unit 411.

The memory unit 414 stores, for example, the programs executed by the mobile device 20 and data necessary for calculating the three-dimensional position information of the object 22.

(Functional configuration of information processing device)
The information processing device 100 realizes a communication unit 401, a first acquisition unit 402, a second acquisition unit 403, a first calculation unit 404, a second calculation unit 405, a correction unit 406, etc., by executing a predetermined program in the CPU 301, for example. Note that at least a part of the above-mentioned functional configurations may be realized by hardware.

In addition, the information processing device 100 realizes a memory unit 407, for example, by a program executed by the CPU 301, the HD 304, the HDD controller 305, etc.

The communication unit 401, for example, uses the network I/F 308 or the like to connect the information processing device 100 to the communication network N and executes communication processing to communicate with the first imaging device 10 and the mobile device 20, etc. For example, the communication unit 401 receives the first image transmitted by the first imaging device 10 and the three-dimensional position information of the object 22 transmitted by the mobile device 20, etc.

The first acquisition unit 402 executes a first acquisition process to acquire first object position information, which is two-dimensional position information of the object 22, based on the first image captured by the first imaging device 10. For example, the first acquisition unit 402 acquires, as the first object position information, the two-dimensional coordinates of the center position 211 of the marker 210 captured in the first image 200 as shown in FIG. 2.

The second acquisition unit 403 executes a second acquisition process to acquire three-dimensional position information of the object 22 by a means different from the first acquisition unit 402. As an example, the second acquisition unit 403 acquires three-dimensional position information of the object calculated based on the second image captured by the second imaging device 21. For example, the second acquisition unit 403 acquires the three-dimensional position information of the object 22 transmitted by the mobile device 20 via the communication unit 401. Specifically, as described above, the mobile device 20 measures three-dimensional position information indicating the position of the second imaging device 21 for each video frame using a known VSLAM technique. In addition, the mobile device 20 calculates the three-dimensional position information of the object 22 from the three-dimensional position information of the second imaging device 21 and the relative position between the second imaging device 21 and the object 22 acquired in advance. However, this is not limited to this, and the mobile device 20 may calculate the three-dimensional position information of the object 22 using other positioning techniques such as radio waves or ultrasound.

The first calculation unit (projection unit) 404 executes a first calculation process to calculate second object position information obtained by projecting the three-dimensional position information of the object 22 acquired by the second acquisition unit 403 into two dimensions. For example, the first calculation unit 404 converts the three-dimensional position information of the object 22 into the camera coordinates of the first imaging device 10 using external parameters indicating the position and orientation of the first imaging device 10. The first calculation unit 404 also converts the position information converted into the camera coordinates into two-dimensional position coordinates on the first image using internal parameters of the first imaging device 10 previously stored in the storage unit 407 or the like.

The second calculation unit (distance calculation unit) 405 executes a second calculation process to calculate the distance between the first trajectory of the object 22 based on the first object position information and the second trajectory of the object 22 based on the second object position information.

The correction unit 406 executes a correction process to correct the distance between the first trajectory and the second trajectory. For example, the correction unit 406 corrects external parameters indicating the position and orientation of the first imaging device 10, or the second object position, etc., so as to minimize the distance between the first trajectory and the second trajectory. Furthermore, the correction unit 406 resets the position and orientation of the first imaging device 10 based on the execution result of the correction process, as necessary. The specific processing content of the correction process will be described later.

The storage unit 407 stores various data and information, such as, for example, a first image, first object position information, three-dimensional position information of the object 22, internal parameters and external parameters of the first imaging device 10, or second object position information.

Note that the functional configuration of the information processing system 1 shown in FIG. 4 is an example. For example, the information processing system 1 may have a functional configuration as shown in FIG. 5.

FIG. 5 is a diagram showing another example of the functional configuration of an information processing system according to an embodiment. In the example of FIG. 5, the second imaging device 21 is configured to transmit the captured second image to the information processing device 100 via the communication network N. In addition, the information processing device 100 realizes a position information calculation unit 413 that calculates three-dimensional position information of the object 22 from the second image captured by the second imaging device 21, for example, by executing a predetermined program in the CPU 301. In this case, the second acquisition unit 403 acquires the three-dimensional position information of the object 22 from the position information calculation unit 413.

<Processing flow>
Next, a process flow of the information processing method according to the first embodiment will be described.

(Correction process)
Fig. 6 is a flowchart showing an example of a correction process according to an embodiment. This process shows an example of a correction process in which the information processing device 100 having the functional configuration shown in Fig. 4 corrects external parameters indicating the position and orientation of the first imaging device 10.

In step S601, the first acquisition unit 402 acquires first object position information, which is two-dimensional position information of the object 22, based on the first image captured by the first imaging device 10. For example, the first acquisition unit 402 acquires first object position information indicating the center position 211 of the marker 210 for each frame from a series of images captured by the first imaging device 10 at multiple times, and stores the acquired first object position information in the storage unit 407 or the like in association with the imaging time.

In step S602, the second acquisition unit 403 acquires three-dimensional position information of the object 22 calculated based on the second image captured by the second imaging device 21. For example, the second acquisition unit 403 stores in the storage unit 407, etc., the three-dimensional position information of the object 22 calculated by the position information calculation unit 413 of the mobile device 20 from a series of images captured at multiple times by the second imaging device 21 and the image capture time (or calculation time).

In step S603, the correction unit 406 initializes the position and orientation of the first imaging device 10 with arbitrary values. For example, if the correction unit 406 knows the approximate position and orientation of the first imaging device, it is preferable to use those values to initialize external parameters indicating the position and orientation of the first imaging device 10.

In step S604, the first calculation unit 404 calculates second object position information obtained by projecting the three-dimensional position information of the object 22 acquired by the second acquisition unit 403 into two dimensions. For example, the first calculation unit 404 converts the three-dimensional position information of the object 22 into the camera coordinates of the first imaging device 10 using external parameters indicating the position and orientation of the first imaging device 10. The first calculation unit 404 also converts the position information converted into the camera coordinates into second object position information, which is two-dimensional position information on the first image, using internal parameters of the first imaging device 10. Note that the information processing device 100 stores internal parameters such as the focal length, image center, and distortion characteristics of the first imaging device 10 in advance in the storage unit 407, etc.

In step S605, the second calculation unit 405 executes a distance calculation process to calculate the distance between the first trajectory of the object 22 based on the first object position information and the second trajectory of the object 22 based on the second object position information. The distance calculation process will be described later.

In step S606, the correction unit 406 determines whether the distance between the first trajectory and the second trajectory has been minimized. For example, if the distance between the first trajectory and the second trajectory has converged to a minimum value, the correction unit 406 determines that the distance between the first trajectory and the second trajectory has been minimized. If the distance between the first trajectory and the second trajectory has not been minimized, the correction unit 406 transitions the process to step S607. On the other hand, if the distance between the first trajectory and the second trajectory has been minimized, the correction unit 406 ends the correction process.

When the process proceeds to step S607, the correction unit 406 corrects the position and orientation (position and orientation) of the first image capture device 10. For example, the correction unit 406 resets the external parameters indicating the position and orientation of the first image capture device 10 based on a known nonlinear optimization method such as the conjugate gradient method, and returns the process to step S604.

By the processing of Figure 6, the information processing device 100 can correct the external parameters indicating the position and orientation of the first imaging device 10.

(Regarding distance calculation processing)
7 is a diagram showing an image of imaging timing according to an embodiment. This diagram shows an example of the imaging timing of the first imaging device 10 and the imaging timing of the second imaging device 21. In the example of FIG. 7, the first imaging device 10 captures a first image at times t11, t12, t13, and so on, and the second imaging device 21 captures a second image at times t21, t22, t23, and so on. Also, it shows that the information processing device 100 receives the first image from the first imaging device 10 at times T1, T2, T3, and so on.

7, the imaging timing of the first imaging device 10 and the imaging timing of the second imaging device 21 generally do not match due to differences in frame rates and time lags in image transfer. As a result, when the mobile device 20 is moving, for example, the position of the object 22 differs between when the first imaging device 10 images the mobile device 20 at time t11 and when the second imaging device 21 images the mobile device 20 at time t21.

Therefore, in conventional calibration methods, for example, the mobile device 20 is stopped at a certain point to capture a first image and a second image, and then the mobile device 20 is stopped at another point to capture the first image and the second image, and this process is repeated.

Therefore, in this embodiment, the distance between the trajectory of the object 22 based on the first object position information and the trajectory of the object 22 based on the second object position information is calculated, so that calibration can be performed using the first image and the second image captured while the mobile device 20 is traveling.

Figure 8 is a diagram showing an image of two trajectories according to one embodiment. In Figure 8, black circles 811a, 811b, 811c, ... indicate the position of object 22 according to first object position information, which is two-dimensional position information based on a first image captured by first imaging device 10. Also, white circles 821a, 821b, 821c, ... indicate the position of object 22 according to second object position information, which is two-dimensional position information based on a second image captured by second imaging device 21.

As shown in FIG. 8, when calibration is not completed, the first trajectory 810 of the object 22 based on the first object position information and the second trajectory 820 of the object 22 based on the second object position information are misaligned.

The information processing device 100 according to this embodiment corrects external parameters indicating the position and orientation of the first imaging device 10 so as to minimize the distance between a first trajectory 810 of the object 22 based on the first object position information and a second trajectory 820 of the object 22 based on the second object position information. This makes it possible to correct the second object position information so that the first trajectory 810 and the second trajectory 820 substantially overlap, for example, as shown in FIG. 9. Note that the distance between the first trajectory and the second trajectory can be calculated, for example, by the processes shown in FIGS. 10 and 11.

(Distance calculation process)
Fig. 10 is a flowchart showing an example of a distance calculation process between two trajectories according to an embodiment. This process shows an example of a distance calculation process in which the second calculation unit 405 calculates the distance between the first trajectory of the object 22 based on the first object position information and the second trajectory of the object 22 based on the second object position information in step S605 in Fig. 6, for example.

Note that here, the black circles 811a, 811b, 811c, ... which indicate the position of the object 22 according to the first object position information shown in FIG. 8 are referred to as first points, and the white circles 821a, 821b, 821c, ... which indicate the position of the object 22 according to the second object position information are referred to as second points.

In step S1001, the second calculation unit 405 performs the processes of steps S1002 to S1004 for each of the first points.

In step S1002, the second calculation unit 405 selects a second point that is imaged within a predetermined time (e.g., within ±3 seconds) from the image capture time of the first point. Here, the predetermined time is set so that multiple second points are included within the predetermined time. If the number of multiple second points is three or more, the three or more second points become a broken line with continuous line segments having each of the second points as an end point. Note that by selecting second points within the predetermined time, the information processing device 100 can avoid calculating the distance to an incorrect trajectory even if the object 22 passes the same position at different times.

In step S1003, the second calculation unit 405 calculates the distance between the first point and a broken line connecting the selected second points. For example, the second calculation unit 405 executes a process for calculating the distance between the broken line and the point, which will be described later with reference to FIG. 11.

In step S1004, the second calculation unit 405 records the calculated distance between the first point and the broken line, for example, in the memory unit 407 or the RAM 303.

In step S1005, the second calculation unit 405 executes the processes of steps S1002 to S1004 for each of the first points, and then transitions to step S1006.

When proceeding to step S1006, the second calculation unit 405 determines the sum of the distances between the recorded first points and the broken line as the distance between the two trajectories.

(Calculation of distance between broken line and point)
11 is a flowchart showing an example of a process for calculating a distance between a polygonal line and a point according to an embodiment. This process shows an example of a process for calculating a distance between a polygonal line connecting selected second points and a first point, which is executed by the second calculation unit 405 in step S1003 in FIG.

In step S1101, the second calculation unit 405 performs the processes of steps S1102 to S1106 for each line segment connecting the second points selected in step S1002 of FIG. 10, for example.

In step S1102, the second calculation unit 405 branches the process depending on the position of the first point relative to the line segment. For example, as shown in FIG. 12, the second calculation unit 405 divides the plane into a first region, a second region, and a third region with two lines 1202 and 1203 that pass through two adjacent second points (white circles 821-1 and 821-2) and are perpendicular to the line segment 1201. If the first point is in the first region, the second calculation unit 405 shifts the process to step S1103. If the first point is in the second region, the second calculation unit 405 shifts the process to step S1104. If the first point is in the third region, the second calculation unit 405 shifts the process to step S1105.

When the process proceeds to step S1103, the second calculation unit 405 calculates the distance 1204 between the first point (black circle 811) and the second point (white circle 821-1), as shown in FIG. 12.

When the process proceeds to step S1104, the second calculation unit 405 calculates the length 1205 of the perpendicular line drawn from the first point (black circle 811) to the line segment 1201, as shown in FIG. 12.

When the process proceeds to step S1105, the second calculation unit 405 calculates the distance 1206 between the first point (black circle 811) and the second point (white circle 821-2), as shown in FIG. 12.

In step S1106, the second calculation unit 405 records the calculated distance, for example, in the memory unit 407 or the RAM 303.

In step S1107, when the second calculation unit 405 completes the processing of steps S1102 to S1106 for each of the line segments connecting the second points selected in step S1002 of FIG. 10, the second calculation unit 405 transitions the processing to step S1108.

In step S1108, the second calculation unit 405 determines the minimum value of the distances recorded in step S1106 as the distance between the line segment connecting the second point and the first point.

By the processing of Figures 10 and 11, the second calculation unit 405 can calculate the distance between the first trajectory of the object 22 based on the first object position information and the second trajectory of the object 22 based on the second object position information.

(Modification)
In the explanation of Fig. 10 and 11, the black circles 811a, 811b, 811c, ... indicating the position of the object 22 according to the first object position information shown in Fig. 8 are set as first points, and the white circles 821a, 821b, 821c, ... indicating the position of the object 22 according to the second object position information are set as second points. However, this is not limited thereto, and the second calculation unit 405 may execute the processing of Fig. 10 and 11 by setting the black circles indicating the position of the object 22 according to the first object position information as second points and the white circles indicating the position of the object 22 according to the second object position information as first points. For example, when the first imaging device 10 has a higher frame rate, it is desirable that the second calculation unit 405 calculates the distance between the first trajectory 810 and the second trajectory 820 by setting the white circles 821 as first points and the black circles 811 as second points, as shown in Fig. 13.

(Regarding the positional relationship between the second imaging device and the object)
As described above, the position information calculation unit 413 calculates three-dimensional position information of the object 22 from three-dimensional position information of the second imaging device 21 calculated by the VSLAM technique or the like and a previously acquired relative position between the second imaging device 21 and the object 22. Therefore, in order for the position information calculation unit 413 to calculate the three-dimensional position information of the object 22, it is necessary to acquire the relative position between the second imaging device 21 and the object 22 in advance.

As a preferred example, the position information calculation unit 413 calculates the relative position between the second imaging device 21 and the object 22 from an image of a marker 210 provided on the object 22 captured by the second imaging device 21, as shown in FIG. 14. For example, the position information calculation unit 413 captures an image of the marker 210 provided on the object 22 by the second imaging device 21, and obtains the coordinates of the outer diameter of the marker 210. In addition, the position information calculation unit 413 calculates the center position 211 of the marker 210 in the camera coordinate system of the second imaging device 21 from the obtained coordinates of the outer diameter of the marker 210 and the actual size of the marker 210.

In addition, if the marker 210 provided on the object 22 cannot be imaged from the position of the second imaging device 21, the position information calculation unit 413 can use an auxiliary marker to calculate the center position 211 of the marker 210 in the camera coordinate system of the second imaging device 21.

15(B), the auxiliary marker 1501 is prepared as an AR marker fixed to the top of a rod 1502, and the distance L from the center 1503 of the auxiliary marker 1501 to the bottom end of the rod 1502 is measured in advance. Also, for example, as shown in FIG. 15(A), the bottom end of the rod 1502 is brought into contact with the center of the marker 210, and the auxiliary marker 1501 is imaged by the second imaging device 21 so that the entire auxiliary marker 1501 is included in the imaging range of the second imaging device 21.

The position information calculation unit 413 determines, from the captured image, the center 1503 of the auxiliary marker 1501 in a three-dimensional coordinate system based on the second imaging device 21, and a three-dimensional vector indicating the vertical and horizontal directions of the marker on the plane of the auxiliary marker 1501. The position information calculation unit 413 can also calculate, as the position of the center of the marker 210, a point that has moved downward from the position of the center 1503 of the auxiliary marker 1501 by the length L of the rod 1502 that was measured in advance.

As described above, according to the first embodiment, the calibration process can be facilitated in a system that identifies the position of an object using a first imaging device 10 that captures an image of an object 22 moved by a moving device 20 and a second imaging device 21 mounted on the moving device.

Second Embodiment
In the first embodiment, the moving device 20 that moves the object 22 has been described as being a moving body such as a forklift, etc. However, the present embodiment is not limited to this, and can also be applied to, for example, a robot system that images a workpiece supplied to a work area from above with the first imaging device 10 and recognizes the position of the workpiece to perform work.

Fig. 16 is a diagram showing an example of the system configuration of an information processing system according to the second embodiment. In the example of Fig. 16, the information processing system 1600 includes a first imaging device 10, a moving device 1611, a second imaging device 21 mounted on the moving device 1611, and an information processing device 100.

The first imaging device 10 is installed, for example, on the ceiling 1601 of a factory or the like, and captures an object 1602 such as a workpiece moved by a moving device 1611, and transmits the captured first image to the information processing device 100.

16, the moving device 1611 is, for example, a five-axis vertical articulated robot that is installed on a floor 1612 of a factory or the like and moves an object 1602. The moving device 1611 has, for example, a fixed base 1613 installed on the floor 1612, and a first arm 1615 rotatably mounted on the fixed base 1613 around a first joint shaft 1614. The moving device 1611 also has a second arm 1617 rotatably mounted on the tip of the first arm 1615 by a second joint shaft 1616, and a third arm 1619 rotatably mounted on the tip of the second arm 1617 by a third joint shaft 1618. Furthermore, the moving device 1611 has a wrist 1621 that is rotatably attached to the tip of the third arm 1619 by a fourth joint shaft 1620, and an end effector 1623 that is rotatable and replaceably attached to the wrist 1621 around a fifth joint shaft 1622. As a result, the end effector 1623 attached to the wrist 1621 is configured to rotate by the fourth joint shaft 1620, which is the joint shaft of the wrist 1621.

The end effector 1623 holds the object 1602, for example, with a suction nozzle, a hand, or a gripper. The first to fifth joint axes 1614, 1616, 1618, 1620, and 1622 are each driven by a servo motor. This allows the moving device 1611 to move the object 1602 with five degrees of freedom. The wrist 1621 of the moving device 1611 is provided with a second imaging device 21 that captures an image of the object 1602, which is a workpiece supplied to the working area.

With the above configuration, the moving device 1611 moves the object 1602, on which the marker 210 is printed, with the object 1602 attached to the end effector 1623. In addition, the second imaging device 21 transmits a second image captured around the moving device 1611 to the information processing device 100.

The information processing device 100 can calibrate the relationship between the position and orientation of the first imaging device 10 and the second imaging device 21 based on the first image received from the first imaging device 10 and the second image received from the second imaging device 21 in a manner similar to the first embodiment.

In this way, the mobile device 20 only needs to be a device that moves the object 1602, and the mobile device 20 itself does not necessarily need to be capable of moving.

Third Embodiment
The first and second embodiments have been described on the premise that there is a difference between the imaging time of the first image captured by the first imaging device 10 and the imaging time of the second image captured by the second imaging device 21. In the third embodiment, as another means for solving the problem, a method for eliminating the difference between the imaging time of the first image captured by the first imaging device 10 and the imaging time of the second image captured by the second imaging device 21 will be described.

Fig. 17 is a diagram showing an example of the system configuration of an information processing system according to the third embodiment. In the example of Fig. 17, the information processing system 1700 has a system configuration similar to that of the information processing system 1 according to the first embodiment described in Fig. 1. Furthermore, the information processing device 100 according to the third embodiment has a function of simultaneously transmitting a trigger signal that instructs the first imaging device 10 and the second imaging device 21 to capture images. As a specific example, the information processing device 100 can transmit a trigger signal via the communication network N by using a software trigger function of a camera interface standard such as GigE Vision.

With the above configuration, in the information processing system 1 according to the third embodiment, the first imaging device 10 and the second imaging device 21 perform imaging almost simultaneously. Note that even if a slight delay difference occurs due to, for example, differences in network configuration, etc., it may be ignored if the error is sufficiently small relative to the moving speed of the moving device 20 (for example, the error in the position of the moving device 20 due to the delay difference is about 1 mm).

According to the third embodiment, it is possible to calibrate the relationship between the position and orientation of the first imaging device 10 and the second imaging device 21 using a conventional calibration method without stopping the moving device 20.

As described above, according to each embodiment of the present invention, the calibration process is facilitated in a system that identifies the position of an object using a first imaging device 10 that captures an image of an object moved by a moving device and a second imaging device 21 mounted on the moving device.

<Additional Information>
Each function of each embodiment described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), and a conventional circuit module designed to execute each function described above.

Furthermore, the group of devices described in the examples represents only one of multiple computing environments for implementing the embodiments disclosed herein. In one embodiment, the information processing device 100 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processing disclosed herein.

Furthermore, each functional configuration of the information processing device 100 may be integrated into one device, or may be divided into multiple devices. Furthermore, at least a part of each functional configuration of the information processing device 100 may be possessed by the mobile device 20 (or the second imaging device 21).

<Additional Notes>
This specification discloses the following information processing device, information processing system, information processing method, and program.
(Section 1)
An information processing device capable of communicating with a first imaging device that captures an image of an object moved by a moving device,
a first acquisition unit that acquires first object position information, which is two-dimensional position information of the object, from a first image captured by the first imaging device;
A second acquisition unit that acquires three-dimensional position information of the object;
a first calculation unit that calculates second object position information, which is two-dimensional position information on the first image, from the three-dimensional position information of the object acquired by the second acquisition unit;
a second calculation unit that calculates a distance between a first trajectory of the object based on the first object position information and a second trajectory of the object based on the second object position information;
a correction unit that executes a correction process to correct a distance between the first trajectory and the second trajectory;
having
Information processing device.
(Section 2)
The information processing device according to claim 1 , wherein the correction unit resets a position and an orientation of the first imaging device based on a result of the execution of the correction process.
(Section 3)
The information processing device described in paragraph 1, wherein the second calculation unit calculates the distance between the first trajectory and the second trajectory based on the distance between a line segment connecting the position of the object according to the first object position information and a point indicating the position of the object according to the second object position information.
(Section 4)
The information processing device described in paragraph 1, wherein the second calculation unit calculates the distance between the first trajectory and the second trajectory based on the distance between a line segment connecting the position of the object according to the second object position information and a point indicating the position of the object according to the first object position information.
(Section 5)
5. The information processing device according to any one of claims 1 to 4, wherein the correction unit corrects the second object position information so as to minimize a distance between the first trajectory and the second trajectory.
(Section 6)
the first calculation unit calculates the second object position information by projecting three-dimensional position information of the object onto the first image using external parameters indicating a position and an orientation of the first imaging device;
5. The information processing device according to any one of claims 1 to 4, wherein the correction unit corrects the external parameters so as to minimize a distance between the first trajectory and the second trajectory.
(Section 7)
the information processing device is capable of communicating with a second imaging device mounted on the mobile device;
7. The information processing device according to any one of claims 1 to 6, wherein the second acquisition unit acquires three-dimensional position information of the object based on a second image captured by the second imaging device.
(Section 8)
8. An information processing system including the information processing device according to claim 7 and a second imaging device mounted on a mobile device,
The information processing system includes:
a position information calculation unit that calculates three-dimensional position information of the object from three-dimensional position information of the mobile device and a relative position between the mobile device and the object, the three-dimensional position information being acquired based on the second image captured by the second imaging device;
Information processing system.
(Section 9)
the second imaging device captures an image of a marker provided on the object;
The calculation unit calculates a relative position between the moving device and the object from an image of the marker.
9. An information processing system according to claim 8.
(Article 10)
the second imaging device captures an image of an auxiliary marker that is disposed so that a relative position from a center position of a marker provided on the object is constant;
The calculation unit calculates a relative position between the moving device and the object from an image of the auxiliary marker.
9. An information processing system according to claim 8.
(Article 11)
an information processing device capable of communicating with a first imaging device that captures an image of an object moved by a moving device,
a first acquisition process for acquiring first object position information, which is two-dimensional position information of the object, from a first image captured by the first imaging device;
A second acquisition process for acquiring three-dimensional position information of the object;
a first calculation process for calculating second object position information, which is two-dimensional position information on the first image, from three-dimensional position information of the object acquired in the second acquisition process;
a second calculation process for calculating a distance between a first trajectory of the object based on the first object position information and a second trajectory of the object based on the second object position information;
a correction process for correcting a distance between the first trajectory and the second trajectory;
Execute
Information processing methods.
(Article 12)
A computer capable of communicating with a first imaging device that images an object moved by a moving device,
a first acquisition process for acquiring first object position information, which is two-dimensional position information of the object, from a first image captured by the first imaging device;
A second acquisition process for acquiring three-dimensional position information of the object;
a first calculation process for calculating second object position information, which is two-dimensional position information on the first image, from three-dimensional position information of the object acquired in the second acquisition process;
a second calculation process for calculating a distance between a first trajectory of the object based on the first object position information and a second trajectory of the object based on the second object position information;
a correction process for correcting a distance between the first trajectory and the second trajectory;
Execute the
program.

Although the embodiment of the present invention has been described above, the present invention is not limited to such a specific embodiment, and various modifications and applications are possible within the scope of the gist of the present invention described in the claims.

1, 1600, 1700 Information processing system 10 First imaging device 20, 1611 Moving device 21 Second imaging device 22, 1602 Object 100 Information processing device 402 First acquisition unit 403 Second acquisition unit 404 First calculation unit 405 Second calculation unit 406 Correction unit 407 Storage unit 413 Position information calculation unit 414 Storage unit

JP 2022-130032 A JP 2019-125354 A

Claims (12)

An information processing device capable of communicating with a first imaging device that captures an image of an object moved by a moving device,
a first acquisition unit that acquires first object position information, which is two-dimensional position information of the object, from a first image captured by the first imaging device;
A second acquisition unit that acquires three-dimensional position information of the object;
a first calculation unit that calculates second object position information, which is two-dimensional position information on the first image, from the three-dimensional position information of the object acquired by the second acquisition unit;
a second calculation unit that calculates a distance between a first trajectory of the object based on the first object position information and a second trajectory of the object based on the second object position information;
a correction unit that executes a correction process to correct a distance between the first trajectory and the second trajectory;
having
Information processing device. The information processing device according to claim 1, wherein the correction unit resets the position and orientation of the first imaging device based on the result of the correction process. The information processing device according to claim 1, wherein the second calculation unit calculates the distance between the first trajectory and the second trajectory based on the distance between a line segment connecting the positions of the object according to the first object position information and a point indicating the position of the object according to the second object position information. The information processing device according to claim 1, wherein the second calculation unit calculates the distance between the first trajectory and the second trajectory based on the distance between a line segment connecting the positions of the object according to the second object position information and a point indicating the position of the object according to the first object position information. The information processing device according to claim 1, wherein the correction unit corrects the second object position information so as to minimize the distance between the first trajectory and the second trajectory. the first calculation unit calculates the second object position information by projecting three-dimensional position information of the object onto the first image using external parameters indicating a position and an orientation of the first imaging device;
The information processing apparatus according to claim 1 , wherein the correction unit corrects the external parameters so as to minimize a distance between the first trajectory and the second trajectory. the information processing device is capable of communicating with a second imaging device mounted on the mobile device;
The information processing apparatus according to claim 1 , wherein the second acquisition unit acquires three-dimensional position information of the object based on a second image captured by the second imaging device. An information processing system including the information processing device according to claim 7 and a second imaging device mounted on a mobile device,
The information processing system includes:
a position information calculation unit that calculates three-dimensional position information of the object from three-dimensional position information of the mobile device and a relative position between the mobile device and the object, the three-dimensional position information being acquired based on the second image captured by the second imaging device;
Information processing system. the second imaging device captures an image of a marker provided on the object;
The calculation unit calculates a relative position between the moving device and the object from an image of the marker.
The information processing system according to claim 8. the second imaging device captures an image of an auxiliary marker that is disposed so that a relative position from a center position of a marker provided on the object is constant;
The calculation unit calculates a relative position between the moving device and the object from an image of the auxiliary marker.
The information processing system according to claim 8. an information processing device capable of communicating with a first imaging device that captures an image of an object moved by a moving device,
a first acquisition process for acquiring first object position information, which is two-dimensional position information of the object, from a first image captured by the first imaging device;
A second acquisition process for acquiring three-dimensional position information of the object;
a first calculation process for calculating second object position information, which is two-dimensional position information on the first image, from three-dimensional position information of the object acquired in the second acquisition process;
a second calculation process for calculating a distance between a first trajectory of the object based on the first object position information and a second trajectory of the object based on the second object position information;
a correction process for correcting a distance between the first trajectory and the second trajectory;
Execute
Information processing methods. A computer capable of communicating with a first imaging device that images an object moved by a moving device,
a first acquisition process for acquiring first object position information, which is two-dimensional position information of the object, from a first image captured by the first imaging device;
A second acquisition process for acquiring three-dimensional position information of the object;
a first calculation process for calculating second object position information, which is two-dimensional position information on the first image, from three-dimensional position information of the object acquired in the second acquisition process;
a second calculation process for calculating a distance between a first trajectory of the object based on the first object position information and a second trajectory of the object based on the second object position information;
a correction process for correcting a distance between the first trajectory and the second trajectory;
Execute the
program.

Images