JP2024102426A - Generation method of image, generation method of free viewpoint image and provision method - Google Patents

Abstract

To generate a free viewpoint image without using a large number of cameras.SOLUTION: A generation method of an image of an object for generating a free viewpoint image includes the steps of: (a) preparing a multi-articulated robot attached with an imaging device; (b) acquiring a photographed image by imaging the object by the imaging device; (c) controlling the multi-articulated robot and changing the relative position of the imaging device relative to the object from the relative position in step (b) that has been executed just before; and (d) repeating step (b) and step (c).SELECTED DRAWING: Figure 4

Description

This disclosure relates to a method for generating an image, a method for generating a free viewpoint image, and a method for providing the image.

Patent document 1 describes a technology that generates free viewpoint images using multiple cameras placed at a sports facility.

International Publication No. 2019/021375

The technology described in Patent Document 1 requires the use of multiple cameras. This requires the adjustment and maintenance of the posture of each camera, which is time-consuming. Therefore, there has been a demand for technology that can generate free viewpoint images without using multiple cameras.

This disclosure can be realized in the following forms:

According to a first aspect of the present disclosure, a method for generating an image of an object for generating a free viewpoint image is provided. This image generating method includes the steps of (a) preparing an articulated robot to which an imaging device is attached, (b) acquiring an image of an object by the imaging device capturing an image of the object, (c) controlling the articulated robot to change the relative position of the imaging device with respect to the object from the relative position in the immediately preceding step (b), and (d) repeating steps (b) and (c).

According to a second aspect of the present disclosure, there is provided a method for generating a free viewpoint image using an image generated by the image generation method of the above aspect. This method for generating a free viewpoint image includes the step of (e) generating data representing a free viewpoint image capable of showing an image of the object as seen from a desired virtual viewpoint, using a plurality of the captured images.

According to a third aspect of the present disclosure, there is provided a method for providing an image using a free viewpoint image generated by the method for generating a free viewpoint image of the above aspect. This method includes the steps of (f) presenting the free viewpoint image generated by executing step (e) to a user, (g) receiving from the user an instruction regarding a desired part and a desired viewpoint in the object, (h) controlling the articulated robot to change the relative position of the imaging device to a relative position that allows the designated desired part to be imaged from the designated desired viewpoint, (i) obtaining an enlarged captured image by the imaging device enlarging and capturing an image of the desired part, and (j) displaying the enlarged captured image.

1 is a schematic diagram showing an overall configuration of a remote operation support system. 1 is a block diagram showing a schematic configuration of a main part of an autonomous mobile robot and a monitoring device. 11 is a flowchart showing a process executed by the monitoring device. 4 is a detailed flowchart of step S10 shown in FIG. 3. FIG. 13 is a diagram showing an example of the positional relationship between an imaging device placed at a preset point and a manipulator to be monitored. 4 is a detailed flowchart of step S20 shown in FIG. 3. FIG. 11 is an explanatory diagram of an instruction to switch the viewpoint. FIG. 13 is an explanatory diagram of an instruction for enlarged display.

A. Embodiments:
1 is a schematic diagram showing the overall configuration of a remote operation support system 1 according to this embodiment. The remote operation support system 1 enables a user at a remote location to monitor and control a robot system RB1.

The robot system RB1 to be monitored includes a manipulator 91, which is a six-axis robot, an end effector 92 attached to the manipulator 91, a robot controller 93 that controls the manipulator 91, and a work table 94. The end effector 92 is, for example, a gripper that holds the workpiece W1.

In the embodiment, the manipulator 91 to be monitored performs precise work. For this reason, a user at a remote location uses the remote operation support system 1 to control the operation of the robot system RB1 while observing the robot system RB1. The robot system RB1 to be monitored is also called the target object. The manipulator 91 operates in either an automatic operation mode or a manual operation mode. In the automatic operation mode, the manipulator 91 operates automatically without receiving instructions from the user during the operation. In the manual operation mode, the manipulator 91 operates according to instructions received successively from the user.

The remote operation support system 1 includes an autonomous mobile robot 10 and a monitoring device 80.

The autonomous mobile robot 10 is an autonomous traveling and transporting robot (Autonomous Mobile Robot: AMR). The autonomous mobile robot 10 moves within a factory in which the robot system RB1 is installed. The autonomous mobile robot 10 includes a manipulator 20, an imaging device 30, a vehicle 40, and a robot controller 50. The vehicle 40 is also referred to as a mobile body.

The manipulator 20 is a six-axis robot. The manipulator 20 includes a base 21 and an arm 22. The base 21 supports the arm 22. The base 21 is fixed to the vehicle 40. Each of the six joints J1 to J6 of the arm 22 is provided with a servo motor, a reducer, and an angle sensor. The servo motor is supplied with current from the robot controller 50 and generates a rotation output for driving each joint. The reducer reduces the speed of the rotation input provided by the corresponding servo motor. The angle sensor is, for example, an encoder, a potentiometer, or a resolver. The angle sensor detects the rotation angle (axis position) of the output shaft of the corresponding servo motor as the rotation angle of that joint. The detected rotation angle is output to the robot controller 50.

The position of the control point of the manipulator 20 in the robot coordinate system RC is controlled by the robot controller 50. The control point of the manipulator 20 is set, for example, at a determined position at the tip of the arm 22. The control point is sometimes called the TCP (Tool Center Point). The robot coordinate system RC is a coordinate system that is fixed with respect to the position of the base 21. The robot coordinate system RC is a three-dimensional orthogonal coordinate system defined by the X-axis and Y-axis that are mutually orthogonal on a horizontal plane, and the Z-axis whose positive direction is vertically upward. The position in the robot coordinate system RC can be expressed by the position in the X-axis direction, the position in the Y-axis direction, and the position in the Z-axis direction. In addition, the posture in the robot coordinate system RC can be expressed by the angular position of rotation around the X-axis, the angular position of rotation around the Y-axis, and the angular position of rotation around the Z-axis.

The imaging device 30 is a camera capable of capturing full-color still images or full-color moving images. The imaging device 30 is fixed to the tip of the arm 22 of the manipulator 20.

Figure 2 is a block diagram showing the general configuration of the main parts of the autonomous mobile robot 10 and the monitoring device 80. Note that the manipulator 20 is not shown in Figure 2.

The imaging device 30 includes a memory 31, a communication unit 32, and an imaging unit 33. The memory 31 stores various data and programs for controlling the imaging device 30. The communication unit 32 communicates with the monitoring device 80 via wireless communication or wired communication. The imaging unit 33 captures an image of a monitoring target and outputs data of the captured image. The captured image data is transmitted to the monitoring device 80 by the communication unit 32.

As shown in FIG. 1, the vehicle 40 supports a base 21. The vehicle 40 can move the manipulator 20 to any position on the floor surface. The vehicle 40 is equipped with a set of drive wheels DW, two sets of driven wheels FW1, FW2, a drive unit 41, and a vehicle control unit 42. The set of drive wheels DW is rotationally driven by the drive unit 41. The driven wheels FW1, FW2 rotate when an external force is applied.

The drive unit 41 drives a set of drive wheels DW. The drive unit 41 includes a set of servo motors, a set of reducers, and a set of angle sensors to drive each of the set of drive wheels DW. Each of the set of servo motors rotates its output shaft under the control of the vehicle control unit 42. Each of the set of reducers reduces the rotation of the output shaft of the corresponding servo motor and transmits it to the corresponding drive wheel DW. Each of the set of angle sensors is, for example, an encoder, a potentiometer, or a resolver. The angle sensor detects the rotation angle of the output shaft of the corresponding servo motor. The detected rotation angle is output to the vehicle control unit 42.

2, the vehicle control unit 42 includes a memory 45, a communication unit 46, a detection unit 47, and a processor 48. The memory 45 stores various data and programs for controlling the vehicle 40. For example, the memory 45 stores a program for controlling the vehicle 40, map data of the inside of the factory where the autonomous mobile robot 10 moves, and speed data indicating a designated value of the traveling speed of the vehicle 40. The communication unit 46 communicates with the robot controller 50 and the monitoring device 80 by wireless communication or wired communication. The detection unit 47 detects the surrounding conditions of the vehicle 40. The detection unit 47 is, for example, LiDAR (Light Detection and Ranging), a camera, or a sonar. The detection unit 47 may include two or more of these sensors. The processor 48 realizes various functions by executing the program stored in the memory 45. For example, the processor 48 controls the driving of the drive unit 41 so that the vehicle 40 reaches the designated destination by executing the program stored in the memory 45.

The robot controller 50 changes the position and posture of the manipulator 20 by driving each joint of the manipulator 20. In the embodiment, the robot controller 50 drives the manipulator 20 to place the imaging device 30 at a specified position in a specified posture. As shown in FIG. 1, the robot controller 50 is disposed inside the housing 21a of the base 21.

As shown in FIG. 2, the robot controller 50 includes a memory 51, a communication unit 52, and a processor 53. The memory 51 stores various data and programs for controlling the robot controller 50. For example, the memory 51 stores a control program for operating the manipulator 20. The communication unit 52 communicates with the vehicle control unit 42 and the monitoring device 80 via wireless communication or wired communication. The processor 53 realizes various functions by executing the various programs stored in the memory 51.

The processor 53 calculates the position and orientation of the manipulator 20. The orientation of the manipulator 20 means the position and orientation of the entire manipulator 20 in three-dimensional space. Specifically, the processor 53 calculates the rotation angle of each joint required to move the control point of the manipulator 20 to the position represented by the given coordinates, and controls the rotation angle of each joint. This causes the control point of the manipulator 20 to be moved to the position represented by the given coordinates.

The monitoring device 80 is a computer that uses the autonomous mobile robot 10 to remotely monitor the robot system RB1. The monitoring device 80 is installed in a facility other than the factory where the robot system RB1 and the autonomous mobile robot 10 are located.

The monitoring device 80 includes a memory 81, a communication unit 82, an input device 83, a display device 84, and a processor 85. The memory 81 stores various programs and data used for various processes executed by the monitoring device 80. The communication unit 82 communicates with the imaging device 30, the vehicle control unit 42, and the robot controller 50 by wireless communication or wired communication via the network NW. The network NW shown in FIG. 1 is, for example, a LAN (Local Area Network), the Internet, or a dedicated line. The input device 83 shown in FIG. 2 is, for example, a keyboard or a mouse. The display device 84 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

The processor 85 realizes various functions by executing the programs stored in the memory 81. The processor 85 functions as an image generating unit 810 and a viewpoint switching unit 820 by executing the programs stored in the memory 81.

The image generation unit 810 uses multiple images of the robot system RB1 captured from multiple viewpoints to generate data representing a free viewpoint image that can represent an image of the robot system RB1 as viewed from a desired virtual viewpoint. The image generation unit 810 displays the free viewpoint image on the display device 84 based on the data representing the free viewpoint image.

The viewpoint switching unit 820 accepts instructions regarding the viewpoint desired by the user. In an embodiment, the user determines a desired part and a desired viewpoint based on the displayed specified image and indicates the desired part and the desired viewpoint in order to check the details of the robot system RB1 to be monitored. The viewpoint switching unit 820 switches the display of the free viewpoint image so that the user can view the indicated part from the indicated viewpoint. The viewpoint switching unit 820 also controls the manipulator 20 to capture an image of the robot system RB1 to be monitored so that the user can view the indicated part from the indicated viewpoint.

Figure 3 is a flowchart showing the processing executed by the monitoring device 80. As shown in Figure 3, in step S10, an image for generating a free viewpoint image is generated. In step S20, a free viewpoint image is generated. The processing shown in Figure 3 is executed by the processor 85 functioning as an image generation unit 810 and a viewpoint switching unit 820. For example, when a user instructs the start of processing via the input device 83, the processor 85 starts the processing shown in Figure 3. Before starting execution of the processing shown in Figure 3, the autonomous mobile robot 10 is prepared in advance.

Figure 4 is a detailed flowchart of step S10 shown in Figure 3. Below, a method for generating an image of a monitoring target for generating a free viewpoint image will be described. In step S101, the processor 85 determines whether or not an instruction to specify a monitoring target has been received. If an instruction to specify a monitoring target has been received (step S101; YES), the processor 85 executes the processing of step S102. Below, an example will be described in which the user specifies the robot system RB1 as the monitoring target.

The user also separately issues sequential commands regarding the operation of the manipulator 91 to the robot controller 93 of the robot system RB1 to be monitored. A camera (not shown) is provided at the tip of the manipulator 91, and the user issues commands while looking at an image showing the situation around the manipulator 91 captured by the camera of the manipulator 91. Thus, the manipulator 91 performs operations according to the commands.

In step S101, if no instruction to specify a monitoring target has been received (step S101; NO), the processor 85 waits.

In step S102, the processor 85 designates information indicating the location of the robot system RB1 within the factory as the destination, and then transmits a movement instruction to the vehicle control unit 42 of the vehicle 40.

The autonomous mobile robot 10 that has received the instruction moves to the specified destination using information indicating the location of the robot system RB1 within the factory and map data stored in memory 45. For example, the vehicle 40 moves to the destination while estimating its own position using a SLAM (Simultaneous Localization and Mapping) algorithm, using the rotation angle detected by an encoder provided in the drive unit 41. When the autonomous mobile robot 10 has completed its movement, it notifies the processor 85 that the movement has been completed.

In step S103, the processor 85 determines whether or not a start condition for starting generation of an image of the monitored object in order to generate a free viewpoint image has been satisfied. In the embodiment, the start condition includes that the movement of the autonomous mobile robot 10 has been completed, and that the manipulator 91, which is the monitored object, has operated according to successive commands issued by the user and has assumed a predetermined position and posture.

As shown in FIG. 1, the manipulator 91, for example, performs an operation of inserting the grasped workpiece WK1 into a hole formed in another workpiece WK2. The predetermined position and posture are the position and posture immediately before the manipulator 91 inserts the workpiece WK1 into the hole formed in the workpiece WK2. The manipulator 91 needs to accurately insert the workpiece WK1 into the center of the hole formed in the workpiece WK2. For this reason, the user stops the operation of the manipulator 91 immediately before inserting the workpiece WK1 into the hole in the workpiece WK2.

In step S103 shown in FIG. 4, if the processor 85 determines that the start condition is satisfied (step S103; YES), it executes step S104. On the other hand, if the start condition is not satisfied (step S103; NO), the processor 85 waits.

In step S104, the processor 85 drives the manipulator 20 via the robot controller 50 so that the manipulator 20 moves along the imaging motion path PI.

Figure 5 shows an example of the positional relationship between the imaging device 30, which is placed at each of multiple pre-set points as the manipulator 20 moves along the imaging motion path PI, and the manipulator 91 of the robot system RB1 to be monitored. Note that the manipulator 20 is not shown in Figure 4. Points at which the manipulator 91 can be imaged from multiple viewpoints are set on the motion path PI. As the manipulator 20 moves, the imaging device 30 is sequentially positioned at multiple pre-set points for imaging. The manipulator 20 is also programmed to stop each time it reaches one of the multiple points set on the imaging motion path PI.

In step S105 shown in FIG. 4, the processor 85 determines whether or not the manipulator 20 has reached the next point. If the processor 85 determines that the manipulator 20 has reached the next point (step S105; YES), it executes step S106. If the manipulator 20 has not reached the next point (step S105; NO), the processor 85 waits.

In step S106, the processor 85 acquires an image IM10 of the monitored object at the current point. Specifically, the processor 85 causes the imaging device 30 to capture an image of the monitored object. The image IM10 is, for example, an image of the hand portion of the manipulator 91, which is the monitored object, the workpiece W1, and the workpiece W2. Furthermore, the processor 85 acquires, as sensor information, the detection values of the angle sensors provided at each of the joints J1 to J6 of the manipulator 20 via the robot controller 50. The processor 85 associates the information representing the current point, the image IM10, and the sensor information, and stores them in the memory 81.

In step S107, the processor 85 determines whether imaging has been performed at all points set on the motion path PI. If imaging has been performed at all points (step S107; YES), the process shown in FIG. 4 is terminated. On the other hand, if imaging has not been performed at all points (step S107; NO), the processor 85 executes the process of step S104 again. Thus, the manipulator 20 moves along the motion path PI for imaging. In this way, the processor 85 changes the relative position of the imaging device 30 with respect to the manipulator 91 from the relative position of the imaging device 30 with respect to the manipulator 91 at the time of the imaging performed immediately before. When no other imaging has been performed between this imaging and the previous imaging, the previous imaging is referred to as the imaging performed immediately before.

By executing the process shown in Figure 4, multiple images IM10 are collected to generate a free viewpoint image of the robot system RB1.

Figure 6 is a detailed flowchart of step S20 shown in Figure 3. Below, a method for generating a free viewpoint image using an image generated by the image generation method shown in Figure 4, and a method for providing an image using the generated free viewpoint image will be described. It is assumed that the process shown in Figure 4 has been executed before the process shown in Figure 6 is executed.

In step S201, the processor 85 reads out the images IM10 and the corresponding sensor information S1 captured at each of the multiple points from the memory 81. The processor 85 calculates coordinate values in the robot coordinate system RC that represent the position and orientation of the manipulator 20's hand at each point based on forward kinematics using the displacement amount of each joint at each point where the image was captured.

In step S202, the processor 85 generates data representing a free viewpoint image. Specifically, the processor 85 uses the multiple images IM10 and the coordinate values calculated for each of them to generate multiple viewpoint images when the robot system RB1 is viewed from each of multiple virtual viewpoints. The generated multiple viewpoint images constitute a free viewpoint image. The processor 85 stores the data representing the free viewpoint image, including the generated viewpoint images and the multiple images IM10, in the memory 81.

In step S203, the processor 85 uses the data representing the free viewpoint image to display on the display device 84 a viewpoint image of the robot system RB1 as seen from a preset initial viewpoint. In this manner, the generated free viewpoint image is presented to the user.

In step S204, the processor 85 determines whether or not an instruction to switch the viewpoint has been received from the user. The user can use the input device 83 to perform a viewpoint switching operation to instruct the display device 84 to display an image of a desired part of the robot system RB1 as seen from a desired viewpoint.

Figure 7 is an explanatory diagram of an instruction to switch the viewpoint. For example, in step S203, the processor 85 displays on the display device 84 a viewpoint image of the tip of the manipulator 91 and images of buttons for adjusting the rotations of roll, pitch, and yaw, as shown in the upper part of Figure 7. The user can operate the buttons using the input device 83 to display on the display device 84 the robot system RB1 as viewed from the desired viewpoint. For example, roll is rotation around the Y axis in the robot coordinate system set in the robot system RB1. Pitch is rotation around the X axis in the robot coordinate system set in the robot system RB1. Yaw is rotation around the Z axis in the robot coordinate system set in the robot system RB1.

As shown in FIG. 6, in step S204, if the processor 85 receives an instruction from the user to switch the viewpoint (step S204; YES), the processor 85 executes the process of step S205. If the processor 85 does not receive an instruction from the user to switch the viewpoint (step S204; NO), the processor 85 executes the process of step S206.

In step S205, the processor 85 displays on the display device 84 a viewpoint image in which the user views a desired part from a desired viewpoint. For example, in response to an instruction from the user to switch the viewpoint, the processor 85 displays on the display device 84 an image such as that shown in the lower part of FIG. 7.

In step S206 shown in FIG. 6, the processor 85 determines whether or not an instruction for enlarged display has been received from the user. The user can use the input device 83 to instruct the display device 84 to enlarge and display the captured image of the robot system RB1.

Figure 8 is an explanatory diagram of an instruction for enlarged display. For example, in step S203, the processor 85 displays on the display device 84 an image in which an image of the tip of the manipulator 91 is superimposed on a number indicating a selectable viewpoint direction, as shown in the upper part of Figure 8. The user can select the number indicating the desired viewpoint direction using the input device 83.

As shown in FIG. 6, in step S206, if the processor 85 receives an instruction from the user regarding enlarged display (step S206; YES), the processor 85 executes the process of step S207. If the processor 85 does not receive an instruction from the user regarding enlarged display (step S206; NO), the processor 85 executes the process of step S208.

In step S207, the processor 85 displays on the display device 84 an image of a desired part of the manipulator 91 that is enlarged and captured at a desired viewpoint. First, the processor 85 drives the manipulator 20 via the robot controller 50 to place the imaging device 30 in a position and orientation that allows the manipulator 91 to capture a designated part at a designated viewpoint. Then, the processor 85 causes the imaging device 30 to capture the manipulator 91. At this time, the processor 85 controls the imaging device 30 so that the zoom magnification of the imaging device 30 is higher than when the image IM10 used to generate the free viewpoint image was captured. Alternatively, the processor 85 may control the manipulator 20 so that the distance between the imaging device 30 and the manipulator 91 is closer than when the image IM10 used to generate the free viewpoint image was captured. Then, the processor 85 displays on the display device 84 the image captured by the imaging device 30. The image captured by enlarging it is also called a magnified captured image. For example, when an image such as that shown in the upper part of FIG. 8 is displayed on the display device 84, assume that the user selects "08" as the desired viewpoint direction. In this case, an image such as that shown in the lower part of FIG. 8 is captured by the imaging device 30 and displayed on the display device 84.

Therefore, the user can determine the desired part of the monitored object and the desired viewpoint based on the free viewpoint image, and then view an image of the desired part of the monitored object, the manipulator 91, enlarged and viewed from the desired viewpoint. The user resumes the operation of the manipulator 91 via the robot controller 93. While checking the manipulator 91 imaged in an enlarged state, the user issues commands to the robot controller 93 one by one regarding the operation of the manipulator 91, thereby causing the manipulator 91 to insert the workpiece WK2 into the hole formed in the workpiece WK1. The user issues instructions to the monitoring device 80 regarding switching of the viewpoint in the free viewpoint image and instructions regarding the enlarged display as many times as necessary. This changes the relative position of the imaging device 30 with respect to the manipulator 91 from the relative position of the imaging device 30 with respect to the manipulator 91 at the time of the immediately preceding imaging.

In step S208, the processor 85 determines whether or not to continue the processing. For example, the processor 85 determines to continue the processing shown in FIG. 6 until information indicating an instruction to stop the processing is input by a user operation. The processor 85 determines to end the processing shown in FIG. 6 when information indicating an instruction to stop the processing is input by a user operation. If the processing is to be continued (step S208; YES), the processor 85 executes the processing of step S204 again. If the processing is not to be continued (step S208; NO), the processor 85 ends the processing shown in FIG. 6.

As described above, in the embodiment, multiple captured images of an object taken from different viewpoints can be obtained to generate a free viewpoint image without using multiple cameras. Therefore, a free viewpoint image can be generated without using multiple cameras. According to an aspect in which multiple cameras are prepared, in addition to adjusting the attitude of each camera, maintenance is also required, but in the embodiment, the effort of adjusting, maintaining, etc., the multiple cameras is not required.

Furthermore, the autonomous mobile robot 10 equipped with the manipulator 20, which is a six-axis robot, can move around the target of monitoring and capture the target, making it easy to capture the target of monitoring from a desired viewpoint. Even if the user is in a remote location, the user can determine the desired part of the target of monitoring and the desired viewpoint based on the free viewpoint image, and then view the desired part of the target of monitoring in an enlarged state from the desired viewpoint. Even if multiple robot systems are being monitored, the autonomous mobile robot 10 can move to the location of each robot system, allowing the user to monitor each robot system from a remote location.

B. Other embodiments:
B1. Other embodiment 1
Also, the range in which the imaging device 30 can capture images may be limited. Specifically, the imaging device 30 captures only a limited range including the manipulator 91. For example, the range in which the autonomous mobile robot 10 can travel may be limited to a preset range, so that the imaging device 30 captures only the limited range including the manipulator 91. Alternatively, the imaging device 30 may be controlled to enable the imaging function only when the autonomous mobile robot 10 is located within the preset range. Thus, the imaging device 30 does not capture images of a range other than the limited range. This prevents the user from viewing equipment, information, etc. that is not desirable to be viewed in a factory, facility, etc. in which the robot system RB1 to be monitored is located. Thus, for example, it is possible to prevent equipment, information, etc. related to trade secrets from being viewed from a remote location.

B2. Other embodiment 2
In the embodiment, an example has been described in which the manipulator 20 to which the imaging device 30 is attached can move around the periphery of the monitoring target by the autonomous mobile robot 10. However, the manipulator 20 is not necessarily supported by the vehicle 40, and may not be able to move around the periphery of the monitoring target. Even in this case, by disposing the manipulator 20 in a position near the robot system RB1 where the manipulator 91 can be imaged, it is possible to acquire an image for generating a free viewpoint image.

In the embodiment, an example has been described in which the monitoring device 80 is installed in a facility other than the factory in which the robot system RB1 and the autonomous mobile robot 10 are located, and the user remotely monitors and controls the robot system RB1. However, the robot system RB1 does not have to be controlled remotely. In this case, the monitoring device 80 is installed in the factory in which the robot system RB1 and the autonomous mobile robot 10 are located. For example, when multiple robot systems are to be monitored, the user can use the monitoring device 80 to monitor and control the multiple robot systems.

In the embodiment, an example in which one imaging device 30 is attached to the manipulator 20 has been described, but two or more imaging devices may be attached to the manipulator 20.

The number of joints of the manipulator 20 of the autonomous mobile robot 10 is not limited to six. In addition, the manipulator 20 in the embodiment is a vertical multi-joint robot, but the manipulator 20 may be a horizontal multi-joint robot.

The means for realizing the functions of the monitoring device 80 is not limited to software, and some or all of the functions may be realized by dedicated hardware. For example, circuits such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC) may be used as the dedicated hardware.

C. Other Forms:
The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each form described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

(1) According to a first aspect of the present disclosure, there is provided a method for generating an image of an object for generating a free viewpoint image, the method for generating an image of the object includes the steps of: (a) preparing an articulated robot having an imaging device attached thereto; (b) acquiring an image of the object by the imaging device; (c) controlling the articulated robot to change a relative position of the imaging device with respect to the object from the relative position in the immediately preceding step (b); and (d) repeating steps (b) and (c).
According to the above aspect, a plurality of captured images of an object can be obtained from different viewpoints to generate a free viewpoint image without using a large number of cameras.

(2) In the above embodiment, the step (a) may include (a1) a step of preparing an autonomous mobile transport robot, the autonomous mobile transport robot being a multi-joint robot mounted on a moving body and capable of moving around the object, and the step (c) may include (c1) a step of changing the relative position of the imaging device to the object from the relative position in the step (b) executed immediately before by moving the autonomous mobile transport robot.
According to this aspect, since the articulated robot can move around the object and capture the image of the object, it is easy to capture the image of the object from a desired viewpoint.

(3) According to a second aspect of the present disclosure, there is provided a method for generating a free-viewpoint image by using an image generated by the image generating method of the above aspect, the method including: (e) generating data representing a free-viewpoint image capable of showing an image of the object as seen from a desired virtual viewpoint by using a plurality of the captured images.
According to this aspect, a free viewpoint image can be generated without using a large number of cameras.

(4) According to a third aspect of the present disclosure, there is provided a method for providing an image using a free viewpoint image generated by the method for generating a free viewpoint image according to the above aspect, the method for providing an image including: (f) presenting the free viewpoint image generated by executing the step (e) to a user; (g) receiving, from the user, an instruction regarding a desired part and a desired viewpoint in the object; (h) controlling the articulated robot to change the relative position of the imaging device with respect to the object to a relative position that allows the designated desired part to be imaged from the designated desired viewpoint; (i) acquiring an enlarged captured image by the imaging device enlarging and imaging the desired part; and (j) displaying the enlarged captured image.
According to the above aspect, the user can determine a desired portion of an object and a desired viewpoint based on the free viewpoint image, and then visually recognize the desired portion of the object in an enlarged state from the desired viewpoint.

(5) In the above embodiment, step (a) may include (a2) a step of preparing the articulated robot to which the imaging device is attached and capable of communicating with a computer at a remote location, and step (g) may include (g1) a step of receiving the instruction regarding the desired part and the desired viewpoint of the object from the user at the remote location via the computer.
According to this embodiment, a user at a remote location can view a desired portion of the object in an enlarged state from a desired viewpoint.

(6) In the above embodiment, the step (b) includes a step (b1) of the imaging device acquiring the captured image by imaging a limited range including the object without imaging a range other than the limited range including the object, and the providing method may not include the step of imaging a range other than the limited range.
According to this embodiment, the range in which the imaging device can capture an image is limited. For example, in a factory, facility, etc. in which an articulated robot is installed, the range in which the imaging device can capture an image is set so as not to include equipment, information, etc. that is undesirable to be viewed. This prevents equipment, information, etc. that is undesirable to be viewed from being viewed by a user in a remote location. Therefore, for example, equipment, information, etc. related to trade secrets can be prevented from being viewed from a remote location.

1...Remote operation support system, 10...Autonomous mobile robot, 20...Manipulator, 21...Base, 21a...Housing, 22...Arm, 30...Imaging device, 31...Memory, 32...Communication unit, 33...Imaging unit, 40...Vehicle, 41...Drive unit, 42...Vehicle control unit, 45...Memory, 46...Communication unit, 47...Detection unit, 48...Processor, 50...Robot controller, 51...Memory, 52...Communication unit, 53...Processor, 80...Monitoring device, 81...Memory Molly, 82... communication unit, 83... input device, 84... display device, 85... processor, 91... manipulator, 92... end effector, 93... robot controller, 94... work table, 810... image generation unit, 820... viewpoint switching unit, DW... driving wheel, FW1, FW2... driven wheel, IM10... image, NW... network, PI... motion path, RB1... robot system, RC... robot coordinate system, S1... sensor information, WK1, WK2... work

Claims (6)

A method for generating an image of an object for generating a free viewpoint image, comprising the steps of:
(a) preparing an articulated robot having an imaging device attached thereto;
(b) acquiring a captured image by capturing an image of an object with the imaging device;
(c) controlling the articulated robot to change a relative position of the imaging device with respect to the object from the relative position in the immediately preceding step (b);
(d) repeating steps (b) and (c);
including,
How the image is generated. The method of claim 1 , further comprising:
The step (a) comprises:
(a1) preparing an autonomous traveling transport robot that is the articulated robot mounted on a moving body and capable of moving around the object;
Including,
The step (c)
(c1) moving the autonomous traveling transfer robot to change the relative position of the imaging device with respect to the object from the relative position in the immediately preceding step (b);
How the image is generated. A method for generating a free viewpoint image by using an image generated by the generation method according to claim 2, comprising the steps of:
(e) generating data representing a free viewpoint image that can represent an image of the object as viewed from a desired virtual viewpoint using a plurality of the captured images;
A method for generating a free viewpoint image, comprising: A method for providing an image using a free viewpoint image generated by the method for generating a free viewpoint image according to claim 3,
(f) presenting the free viewpoint image generated by executing the step (e) to a user;
(g) receiving from the user an indication of a desired portion and a desired viewpoint on the object;
(h) controlling the articulated robot to change the relative position of the imaging device with respect to the target object to a relative position where the designated desired portion can be imaged from the designated desired viewpoint;
(i) acquiring an enlarged captured image by the imaging device enlarging and imaging the desired portion;
(j) displaying the enlarged captured image;
The method of provision includes: The method of claim 4, further comprising:
The step (a) comprises:
(a2) preparing the articulated robot to which the imaging device is attached and which is capable of communicating with a computer at a remote location;
Including,
The step (g)
(g1) receiving the instruction regarding the desired portion and the desired viewpoint of the object from the user at the remote location via the computer;
including,
How it's provided. The method of claim 5, further comprising:
The step (b)
(b1) acquiring the captured image by the imaging device capturing an image of a limited range including the object without capturing an image of a range other than the limited range including the object;
Including,
The providing method does not include a step of imaging a range other than the limited range.
How it's provided.

Images