JP7536312B2 - Image interface device, image operation device, operation object operation device, operation object operation system, operation object presentation method, and operation object presentation program - Google Patents

Description

The present invention relates to an image interface device, an image operation device, an operation object operation device, an operation object operation system, an operation object presentation method, and an operation object presentation program.

Telexistence is a technology that allows a user to control a remote robot as if it were in the user's hands (see, for example, non-patent document 1).

"Telecommunication, Teleimmersion and Telexistence II" Susumu Tachi, Ohmsha, IOS Press, ISBN 4-274-90638-8 (Ohmsha) ISBN 1-58603- 519-3

Telexistence is a technology that controls and steers a robot based on information obtained from sensors installed in a remote robot. With telexistence, the robot's movements are mapped to the user's physical movements, giving the user the sensation of being "possessed" by the robot and controlling it. This allows the user to operate a robot while feeling as if a remote object were close by.

However, in conventional telexistence, the size of the user relative to the size of the robot is fixed and predetermined. Therefore, if the size, movement, or position of the robot changes from robot to robot, the user cannot operate the robot in a natural way.

The present invention has been made in consideration of these circumstances, and its purpose is to provide technology that enables users to seamlessly operate objects, such as robots, that have a variety of sizes, positions, movements or movement characteristics.

In order to solve the above problems, an image interface device according to one aspect of the present invention includes an image display unit that displays an image of an object to be operated by a user, a surrogate body drawing unit that draws a virtual surrogate body of the user on the image display unit, and an operation signal input unit to which an operation signal from the user for the object to be operated is input. The size of the surrogate body is variable according to the size of the object to be operated. The image of the object to be operated displayed on the image display unit is a subjective image as seen from the surrogate body.

The surrogate body in the embodiment may have multiple sizes, and these multiple sizes may be dynamically switchable for the object to be operated.

In an embodiment, the surrogate body has multiple positions relative to the object to be manipulated, and these multiple positions may be dynamically switchable.

The position of the proxy body in the embodiment may be variable depending on the position of the object to be operated.

In the embodiment, the surrogate body may move on the image display unit in conjunction with the operation performed on the image of the operation target object. In this case, the speed of movement of the surrogate body on the image display unit may be variable according to the speed of movement of the operation target object.

In an embodiment, the surrogate body may have a variable position or orientation relative to the object being operated.

In the embodiment, the operation object may include a plurality of operation objects having different sizes. In this case, when the plurality of operation objects are operated by successively switching between them, the size of the surrogate body may be continuously variable.

The surrogate body of the embodiment may include two surrogate bodies of different sizes. In this case, the image of the operation object displayed on the image display unit may be a mixture of image components of the operation object as seen from the two surrogate bodies of different sizes.

In the embodiment, the image of the object to be operated may be an image captured by a camera installed in the vicinity of the object to be operated.

Another aspect of the present invention is an image manipulation device. This image manipulation device includes an image display unit that displays an image of an object to be manipulated by a user, a surrogate body drawing unit that draws a virtual surrogate body of the user on the image display unit, an operation signal input unit to which an operation signal from the user for the object to be manipulated is input, and an operation unit that generates an operation signal through operation by the user. The size of the surrogate body is variable according to the size of the object to be manipulated. The image of the object to be manipulated displayed on the image display unit is a subjective image as seen from the surrogate body.

The surrogate body in the embodiment may have multiple sizes, and these multiple sizes may be dynamically switchable for the object to be operated.

In an embodiment, the surrogate body has multiple positions relative to the object to be manipulated, and these multiple positions may be dynamically switchable.

Yet another aspect of the present invention is an operation object operation device. This operation object operation device includes an image display unit that displays an image of an operation object operated by a user, a surrogate body drawing unit that draws a virtual surrogate body of the user on the image display unit, an operation signal input unit to which an operation signal from the user for the operation object is input, an operation unit that generates an operation signal through operation by the user, and an operation signal output unit that outputs the user's operation signal to the operation object. The size of the surrogate body is variable depending on the size of the operation object. The image of the operation object displayed on the image display unit is a subjective image seen from the surrogate body.

Yet another aspect of the present invention is a system for manipulating an object to be operated. This system for manipulating an object to be operated includes an image display unit that displays an image of the object to be operated by the user, a surrogate body drawing unit that draws a virtual surrogate body of the user on the image display unit, an operation signal input unit to which an operation signal from the user to the object to be operated is input, an operation unit that generates an operation signal by the user's operation, an operation signal output unit that outputs the user's operation signal to the object to be operated, and a camera that photographs the object to be operated. The size of the surrogate body is variable depending on the size of the object to be operated. The image of the object to be operated displayed on the image display unit is a subjective image seen from the surrogate body.

The surrogate body in the embodiment may have multiple sizes, and these multiple sizes may be dynamically switchable for the object to be operated.

In an embodiment, the surrogate body has multiple positions relative to the object to be manipulated, and these multiple positions may be dynamically switchable.

The camera of the embodiment may be provided with a movement mechanism for changing its position.

The camera of the embodiment may be equipped with a rotation mechanism to change the shooting direction.

The camera movement mechanism and rotation mechanism of the embodiment move the camera in the direction opposite to the user's operation direction, and the image displayed on the image display unit may be a left-right inverted version of the image captured by the camera.

The camera of the embodiment may be a stereo camera having two cameras.

The interocular distance of the stereo camera in the embodiment may be variable depending on the size of the surrogate body.

The camera in the embodiment may be a depth camera that detects the distance to the object to be operated in real time.

The camera in the embodiment may include multiple cameras, each capturing a different field of view.

Yet another aspect of the present invention is a method for presenting an object to be operated. This method includes the steps of acquiring information on the size of the object to be operated, drawing a virtual surrogate body of the user, and displaying an image of the object to be operated. The size of the surrogate body is variable depending on the size of the object to be operated. The image of the object to be operated is a subjective image seen from the surrogate body.

The surrogate body in the embodiment may have multiple sizes, and these multiple sizes may be dynamically switchable for the object to be operated.

In an embodiment, the surrogate body has multiple positions relative to the object to be manipulated, and these multiple positions may be dynamically switchable.

Yet another aspect of the present invention is a program for presenting an object to be operated. This program causes a computer to execute the steps of acquiring information on the size of the object to be operated, drawing a virtual surrogate body of the user, and displaying an image of the object to be operated. The size of the surrogate body is variable depending on the size of the object to be operated. The image of the object to be operated is a subjective image seen from the surrogate body.

The surrogate body in the embodiment may have multiple sizes, and these multiple sizes may be dynamically switchable for the object to be operated.

In an embodiment, the surrogate body has multiple positions relative to the object to be manipulated, and these multiple positions may be dynamically switchable.

In addition, any combination of the above components, and conversions of the present invention between devices, methods, systems, recording media, computer programs, etc., are also valid aspects of the present invention.

According to the present invention, it is possible to present on a screen images that allow the user to seamlessly control objects, such as robots, that have a variety of sizes and movement characteristics.

1 is a functional block diagram of an image interface device according to a first embodiment. FIG. 11 is a functional block diagram of an image interface device according to a second embodiment. FIG. 13 is a functional block diagram of an image manipulation device according to a third embodiment. FIG. 13 is a functional block diagram of an operation target object operating device according to a fourth embodiment. FIG. 13 is a functional block diagram of an operation target object operating system according to a fifth embodiment. 23 is a flowchart showing a processing procedure of an operation target object presenting method according to a sixth embodiment.

The present invention will be described below with reference to the drawings based on preferred embodiments. The embodiments are illustrative and do not limit the invention. All features and combinations described in the embodiments are not necessarily essential to the invention. The same or equivalent components, members, and processes shown in each drawing are given the same reference numerals, and duplicated descriptions are omitted as appropriate. In addition, the scale and shape of each part shown in each drawing are set for convenience to facilitate explanation, and are not to be interpreted as being limiting unless otherwise specified. In addition, when terms such as "first" and "second" are used in this specification or claims, unless otherwise specified, these terms do not represent any order or importance, but are intended only to distinguish one configuration from another. In addition, some of the members that are not important in explaining the embodiment in each drawing are omitted.

[First embodiment]
1 shows a functional block diagram of an image interface device 1 according to the first embodiment. The image interface device 1 includes an image display unit 11, a proxy body drawing unit 14, and an operation signal input unit 15.

The image display unit 11 is a display that displays moving images, such as a liquid crystal display, a video projector, or a head-mounted display. The image display unit 11 displays an image 10 of an object to be operated by the user 12. Furthermore, the virtual surrogate body 13 of the user 12 is drawn on the image display unit 11 by the surrogate body drawing unit 14. The image 10 of the object to be operated displayed on the image display unit 11 is a subjective image seen from the surrogate body 13. That is, the image 10 of the object to be operated is displayed as an image seen from the eyes of the surrogate body 13 drawn on the image display unit 11. The image 10 of the object to be operated may be a real image of a remote object to be operated, photographed by a camera or the like, or may be a virtual image of a simulator or game. Below, an industrial robot such as a construction machine will be mainly described as an example of an object to be operated. However, the object to be operated is not limited to this, and may be anything that can be operated by a user, such as a humanoid robot, a vehicle, or a medical instrument.

The proxy body drawing unit 14 draws a virtual proxy body 13 of the user 12 on the image display unit 11.

An operation signal from the user 12 for the operation object is input to the operation signal input unit 15. The operation signal input to the operation signal input unit 15 is transmitted to the image display unit 11. The image 10 of the operation object displayed on the image display unit 11 is operated by this operation signal.

The surrogate body 13 virtually represents the body of the user 12, and is drawn on the image display unit 11 by the surrogate body drawing unit 14 as described above. That is, when viewed from the perspective of the user 12 operating the image interface device 1, the surrogate body 13 behaves like an alter ego on the image display unit 11. The surrogate body 13 may be a concrete image similar to the body of the user 12, or may be an abstract image such as a silhouette, a line drawing, or a semi-transparent image.

The size of the surrogate body 13 varies depending on the size of the object to be operated. Typically, the larger the object to be operated, the larger the surrogate body 13 is drawn, and the smaller the object to be operated, the smaller the body is drawn. For example, the surrogate body 13 is drawn as being substantially the same size as the object to be operated. For example, if the object to be operated is heavy machinery, the surrogate body 13 is drawn as a "giant" that is approximately the same size as the heavy machinery. Conversely, if the object to be operated is a micromachine, the surrogate body 13 is drawn as a "dwarf" that is approximately the same size as the micromachine. As described above, the image 10 of the object to be operated displayed on the image display unit 11 is a subjective image seen from the surrogate body 13. This allows the user to operate the image 10 of the object to be operated displayed on the image display unit 11 from the standpoint of the surrogate body 13, whose size corresponds to the size of the object to be operated.

In general, when a user operates an object displayed on a screen, if the size of the object on the screen differs from the user's own size, the user feels uncomfortable with the operation. For example, when moving the arm of a heavy machine on the screen with his/her hand, the user's size is too small compared to the heavy machine, so the range of motion of the arm and the range of motion of the user's arm are significantly different, and the user feels that it is difficult to operate as intended. In contrast, by placing a surrogate body of the same size as the heavy machine on the screen and operating the heavy machine from the position of this surrogate body, the user feels that the range of motion of the arm and the range of motion of the user's arm match, and operation without discomfort is possible. Conversely, for example, when operating a micromachine on the screen with his/her hand, the user's size is too large compared to the micromachine, so the fine movement of the micromachine and the movement of the hand are significantly different, and the user feels that it is difficult to operate as intended. In contrast, by placing a surrogate body of the same size as the micromachine on the screen and operating the micromachine from the position of this surrogate body, the user feels that the fine movement of the micromachine and the movement of the user's hand match, and operation without discomfort is possible.

In one embodiment, the surrogate body 13 may have a plurality of sizes. In this case, the size of the surrogate body 13 may be dynamically switchable with respect to the object to be operated. In the above description, a typical example of the surrogate body 13 has been shown in which the larger the object to be operated, the larger the body is drawn, and the smaller the object to be operated, the smaller the body is drawn. However, this is not limited to this, and the surrogate body 13 may have various sizes for one object to be operated depending on the purpose and situation. For example, if the object to be operated is a heavy machine with a height of 6 m, the surrogate body 13 may be drawn as having four heights, such as a "very large giant" with a height of 10 m, a "giant" with a height of approximately the same size as the heavy machine, a "small giant" with a height of 4 m, and a "human" with a height of about 1 m.

In the above example, when the surrogate body 13 is drawn as a "giant", the scale of the surrogate body 13 and the object to be operated are almost the same, so the user does not feel any discomfort. On the other hand, the object to be operated is operated by a "giant", resulting in a coarse operation accuracy. Next, when the surrogate body 13 is drawn as a "small giant", the scale of the surrogate body 13 and the object to be operated are not significantly different, so a feeling of operation in which the sense of discomfort is improved to a certain extent while the decrease in accuracy is suppressed is obtained. Next, when the surrogate body 13 is drawn as a "human", a feeling of high precision operation is obtained, although there is a sense of discomfort due to the difference in scale between the surrogate body 13 and the object to be operated. Finally, when the surrogate body 13 is drawn as a "very large giant", a feeling of operation on a large scale is obtained, although there is a sense of discomfort due to the surrogate body 13 being larger than the object to be operated. In this way, by giving the surrogate body 13 multiple sizes and dynamically switching between these multiple sizes, the user can accurately select the size of the surrogate body 13 according to the purpose and purpose of the operation, the situation in which the object to be operated is placed, and his/her own preferences.

When the surrogate body 13 has multiple sizes and the sizes can be dynamically switched relative to the object to be operated, the surrogate body 13 may have multiple positions relative to the object to be operated. In this case, the position of the surrogate body 13 may be dynamically switched depending on the size of the surrogate body 13.

In general, where the surrogate body 13 should be positioned relative to the operation object depends on the size of the surrogate body 13. In particular, the more the size of the surrogate body 13 and the operation object differs, the more important the position of the surrogate body 13 relative to the operation object becomes. For example, in the above example, when the surrogate body 13 has the size of a "very large giant", a "giant", a "small giant", or a "human", the surrogate body 13 may have a plurality of height positions in the range from the top to the bottom of the operation object at each of the sizes. Here, when the surrogate body 13 is drawn as a "giant", the surrogate body 13 and the operation object are almost the same size, so the height position of the surrogate body 13 may be roughly the same as the operation object. Here, when the size of the surrogate body 13 is switched to a "human", a more accurate operation can be realized by matching the center position of the surrogate body 13 with the operation position of the operation object (the tip of an arm, the end effector of a robot, etc.).
In this way, by giving the surrogate body 13 multiple positions relative to the object to be operated and dynamically switching between these multiple positions, the user can accurately position the surrogate body 13 according to the size of the surrogate body.

In one embodiment, the position of the surrogate body 13 may be variable depending on the position of the object to be operated. Generally, when a user operates an object to be operated displayed on a screen, the user feels uncomfortable if he or she is positioned in an inappropriate location relative to the object to be operated on the screen. For example, when the object to be operated is positioned next to a wall, the user will not operate the object from a narrow location between the object to be operated and the wall. In such a case, by positioning the surrogate body 13 in a position that is easy for the user to operate depending on the position of the object to be operated, it becomes possible for the user to perform an operation that does not feel strange.

In one embodiment, the surrogate body 13 may move on the image display unit together with the operation performed on the image 10 of the operation object. In this case, the speed of movement of the surrogate body 13 may be variable according to the speed of movement of the operation object. Typically, the surrogate body 13 is drawn to move slower the larger the operation object, and to move faster the smaller the operation object. As described above, when the user operates the arm of a heavy machine by hand on the image display unit, the user feels that the movement speed of the hand is too fast compared to the movement speed of the arm. Similarly, when the user operates the micromachine by hand on the image display unit, the user feels that the movement speed of the hand is too slow compared to the movement speed of the micromachine. In contrast, if the hand of the surrogate body 13 drawn on the image display unit is made to move quickly in response to the movement of the micromachine, the user can operate the micromachine without any discomfort.

In one embodiment, the surrogate body 13 may have a variable position and orientation relative to the object to be operated. For example, the surrogate body 13 may be drawn to be located in front, behind, left, right, above, or below the object to be operated. In this case, the user feels as if he or she is operating the object to be operated from the front, back, left, right, above, or below, respectively. For example, when a user operates the arm of a heavy machine by hand on an image display unit, the natural position and orientation of the heavy machine to operate the heavy machine differs depending on the posture and direction of movement of the heavy machine. Therefore, if the user's position and orientation relative to the heavy machine are fixed, the operation may be felt to be unnatural. In contrast, if the surrogate body 13 is drawn to be in an appropriate position and orientation relative to the heavy machine, the user can operate the heavy machine without any discomfort.

In one embodiment, when there are a plurality of operation objects of different sizes and the user switches between them continuously to operate them, the surrogate body 13 may change its size continuously. For example, consider a case where there is a huge first operation object such as a heavy machine and a small second operation object such as a robot that performs fine electrical wiring. The user operates the first operation object to lift and move a car, and then quickly operates the second operation object to perform electrical wiring work in the engine room of the car. At this time, while the user is operating the first operation object, the surrogate body 13 is a "giant", and when the user starts operating the second operation object, the surrogate body 13 is continuously transformed into a "dwarf". As a result, when operating operation objects of different sizes continuously, the user can naturally operate them as if he or she had continuously transferred to a body of the size optimal for operating these operation objects.

In one embodiment, when the object to be operated has multiple configurations or parts and the user operates them by continuously switching between them, the surrogate body 13 may continuously change its size. For example, consider a robot having a first arm that is large in size and has low operating accuracy, and a second arm that is small in size and has high operating accuracy. That is, the robot has two parts with different sizes and operating accuracy. The user operates the first arm of the robot to move a precision device, and then quickly operates the second arm to repair the precision device. At this time, the surrogate body 13 is a "human" while the user is operating the first arm, and when the user starts operating the second arm, the surrogate body 13 is continuously transformed into a "dwarf". As a result, when the object to be operated has multiple parts with different sizes and operating accuracy that are continuously switched between and operated, the user can naturally operate the parts as if he or she had continuously moved into a body of the size optimal for operating these parts.

In one embodiment, the image 10 of the object to be operated displayed on the image display unit 11 may be a mixture of image components of the object to be operated as seen from two different sizes of surrogate bodies. For example, it may be a mixture of image components of a large object to be operated as seen from a large surrogate body and image components of a small object to be operated as seen from a small surrogate body. According to this embodiment, the user can instantly switch between the images by switching their awareness of which image to view.

As described above, according to these embodiments, an image of the object to be operated can be displayed on the image display unit. Since the image displayed at this time is a subjective image seen from the virtual body, the user can operate it without feeling uncomfortable.

[Second embodiment]
2 shows a functional block diagram of an image interface device 1 according to the second embodiment. The image interface device 1 includes an image display unit 11 that displays an image 10 of an operation object 110 operated by a user 12, a surrogate body drawing unit 14 that draws a virtual surrogate body 13 of the user 12 on the image display unit 11, and an operation signal input unit 15 to which an operation signal from the user 12 to the operation object 110 is input. The image 10 of the operation object 110 is an image captured by a camera 100 installed near the operation object 110. That is, this embodiment differs from the more general first embodiment in that the image of the operation object 110 is an image captured by the camera 100. Other configurations and operations are common to the first embodiment.

Camera 100 is, for example, a video camera. Camera 100 is placed near the object to be operated 110. The image captured by camera 100 is almost the same as the view seen from the surrogate body 13. In this sense, camera 100 shares a viewpoint with surrogate body 13. Camera 100 transmits the captured image to image display unit 11. Image display unit 11 displays the image received from camera 100.

According to this embodiment, a real object to be operated can be displayed on the image display unit. The image displayed at this time is a subjective image seen from the virtual body, so the user can operate it without feeling uncomfortable.

[Third embodiment]
3 shows a functional block diagram of an image manipulation device 2 according to a third embodiment. The image manipulation device 2 includes an image display unit 11 that displays an image 10 of an operation object 110 to be operated by a user 12, a surrogate body drawing unit 14 that draws a virtual surrogate body 13 of the user 12 on the image display unit 11, an operation signal input unit 15 to which an operation signal from the user 12 to the operation object 110 is input, and an operation unit 20. That is, in this embodiment, the operation unit 20 is added to the image interface device of FIG. 1.

The operation unit 20 generates an operation signal by being operated by the user 12. The operation unit 20 is, for example, a motion controller held in the hand of the user 12. The operation unit 20 is not limited to this, and may be any suitable controller such as a mouse, a joystick, or a game pad.

While viewing an image 10 of an object to be operated displayed on the image display unit 11, the user 12 operates the operation unit 20 to operate the image 10. This causes the operation unit 20 to generate an operation signal that determines the behavior of the image 10. The operation unit 20 transmits the generated operation signal to the operation signal input unit 15. The operation signal input unit 15 transmits the operation signal to the image display unit 11.

The image 10 of the object to be operated may be an image of a remote object to be operated captured by a camera or the like, or it may be a virtual image such as a simulator or game.

According to this embodiment, the user can view the image of the object to be operated displayed on the image display unit and operate the image on the screen without feeling uncomfortable.

[Fourth embodiment]
Fig. 4 shows a functional block diagram of an operation object manipulation device 3 according to a fourth embodiment. The operation object manipulation device 3 includes an image display unit 11 that displays an image 10 of an operation object 110 manipulated by a user 12, a surrogate body drawing unit 14 that draws a virtual surrogate body 13 of the user 12 on the image display unit 11, an operation signal input unit 15 to which an operation signal from the user 12 to the operation object 110 is input, an operation unit 20, and an operation signal output unit 30. That is, in this embodiment, the operation signal output unit 30 is added to the image manipulation device 2 in Fig. 3.

The operation signal output unit 30 receives an operation signal from the user 12 via the image display unit 11 .
The operation signal output section 30 outputs this operation signal to the operation target 110 .

While viewing the image 10 of the operation object 110 displayed on the image display unit 11, the user 12 operates the operation unit 20 to operate the remote operation object 110. This causes the operation unit 20 to generate an operation signal that determines the behavior of the operation object 110. The operation unit 20 transmits the generated operation signal to the operation signal input unit 15. The operation signal input unit 15 transmits the operation signal to the image display unit 11. The image display unit 11 transmits the operation signal to the operation signal output unit 30. The operation signal output unit 30 outputs the operation signal to the operation object 110.

According to this embodiment, the user can operate the object remotely and seamlessly while viewing the image of the object displayed on the image display unit.

[Fifth embodiment]
5 shows a functional block diagram of an operation object manipulation system 4 according to a fifth embodiment. The operation object manipulation system 4 includes an image display unit 11 that displays an image 10 of an operation object 110 manipulated by a user 12, a surrogate body drawing unit 14 that draws a virtual surrogate body 13 of the user 12 on the image display unit 11, an operation signal input unit 15 to which an operation signal from the user 12 to the operation object 110 is input, an operation unit 20, an operation signal output unit 30, and a camera 100. That is, in this embodiment, the camera 100 is added to the operation object manipulation device 3 of FIG. 4.

The camera 100 is, for example, a video camera, and is placed near the object to be operated 110. The image captured by the camera 100 is almost the same as the view seen from the surrogate body 13. In this sense, the camera 100 shares a viewpoint with the surrogate body 13. The camera 100 transmits the captured image to the image display unit 11. The image display unit 11 displays the image received from the camera 100.

According to this embodiment, the user can operate the object remotely and seamlessly while viewing an image of the real object captured by a camera provided in the system.

In one embodiment, the camera 100 may be provided with a movement mechanism for changing its position. The movement mechanism moves the position of the camera 100 relative to the object to be operated 110, for example, by a motor. This allows the camera 100 to capture the object to be operated 110, for example, from the front, back, left, right, above, or below. Therefore, the subjective image seen by the surrogate body 13 also shows the object to be operated 110 seen from the front, back, left, right, above, or below.

In one embodiment, the camera 100 may be equipped with a rotation mechanism for changing the shooting direction. The rotation mechanism rotates the camera 100 around a predetermined axis, for example, using a motor. Rotation around the axis corresponds to, for example, rolling, pitching, yawing, etc. of the camera. This allows the camera 100 to shoot the object to be operated 110 from various angles. Therefore, the subjective images seen by the surrogate body 13 also show the object to be operated 110 seen from various angles.

In one embodiment, the moving mechanism or rotating mechanism of the camera 100 may move the camera in the opposite direction to the user's operation direction. Furthermore, the image displayed on the image display unit may be an image captured by the camera 100 with the left and right reversed. In this case, the user sees a view from the surrogate body on the image display unit 11, which is a view with the left and right reversed from the actual view. Furthermore, the user operates the object to be operated in the opposite direction to the direction of the operation performed by the user. In other words, the user feels as if he or she is operating the object to be operated in a world with the left and right reversed from the actual world. According to this embodiment, for example, if a right-handed user wants to approach the object from the right side with his or her right hand according to his or her physical sensation, but can only approach the object to be operated from the left side due to the environment in which it is placed, the user can operate the object using his or her right hand without feeling uncomfortable.

In one embodiment, the camera 100 may be a stereo camera equipped with two cameras. In this case, the image of the operation target object 110 captured by the camera 100 is three-dimensional and has a sense of depth for the user.

In one embodiment, the interocular distance of the stereo camera may be variable depending on the size of the surrogate body 13. In this case, the camera 100 may be provided with an interocular distance adjustment mechanism for dynamically adjusting the interocular distance using a motor or the like. Typically, the interocular distance of the stereo camera is adjusted to be longer the larger the surrogate body 13 is, and is adjusted to be shorter the smaller the surrogate body 13 is. In this embodiment, the interocular distance of the stereo camera is adjusted according to the size of the surrogate body 13, so that the user can see an image with a more natural sense of depth for each operation target object 110.

In one embodiment, the camera 100 may be a depth camera that detects the distance to the object of operation 110 in real time. In this case, the image of the object of operation 110 captured by the camera 100 provides the user with a more realistic sense of depth and immersion.

In one embodiment, the camera 100 may include multiple cameras each capturing a different field of view. In this case, the images of the multiple fields of view captured by the multiple cameras may be displayed simultaneously, or may be displayed separately by switching between them as the user wishes. In this embodiment, the user can view the object 110 to be operated from multiple viewpoints of the surrogate body 13.

Sixth embodiment
FIG. 6 is a flowchart showing a processing procedure of the operation target object presenting method according to the sixth embodiment.

Step S1 is a process for acquiring information on the size of the operation object to be presented. The size information may be acquired in real time by a camera or the like, or may be acquired from data previously stored in a database or the like. The size of the operation object is arbitrary, but for example, it may be several tens of meters for construction machinery or several mm for a micromachine. Once the size information of the operation object has been acquired, the process proceeds to step S2.

Step S2 is a process of drawing a surrogate body on the image display unit according to the size of the object to be operated. This surrogate body virtually represents the user's body. It may be a concrete image similar to the user's body, or an abstract image such as a silhouette, line drawing, or semi-transparent image. The size of the surrogate body is variable according to the size of the object to be operated. Typically, the larger the object to be operated, the larger the surrogate body is drawn, and the smaller the object to be operated, the smaller the body is drawn. For example, the surrogate body is drawn as being substantially the same size as the object to be operated. Once the surrogate body is drawn on the image display unit, the process proceeds to step S3.

Step S3 is a process of displaying an image of the operation object on the image display unit as a subjective image of the surrogate body. That is, the image of the operation object is displayed as an image seen from the eyes of the surrogate body drawn on the image display unit. The image of the operation object may be a real image of a remote operation object captured by a camera or the like, or may be a virtual image of a simulator, game, or the like.

According to this embodiment, an object to be operated can be presented to the user. The image presented at this time is a subjective image seen from the virtual body, so the user can operate it without feeling uncomfortable.

[Seventh embodiment]
The seventh embodiment is an operation object presentation program. This program causes a computer to execute step S1 of displaying an image of an operation object operated by a user on an image display unit, step S2 of drawing a virtual surrogate body of the user on the image display unit, and step S3 of operating the image of the operation object by an operation signal from the user. The processing procedure of each step is the same as that explained in the sixth embodiment, so a detailed explanation will be omitted.

According to this embodiment, a computer can be used to present an object to be operated to a user. The image presented at this time is a subjective image seen from the virtual body, so the user can operate it without feeling uncomfortable.

(Modification)
The present invention has been described above based on the embodiments. These embodiments are merely illustrative, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[Modification 1]
The moving mechanism and the rotating mechanism of the camera in the embodiment may be operated using a head mounted display. In this case, the head mounted display transmits a signal for operating the moving mechanism and the rotating mechanism of the camera to the moving mechanism and the rotating mechanism. This signal is for moving the shooting position and the direction of the camera in conjunction with the movement of the user's head.
According to this modification, the user can freely control the shooting position and direction of the operation object by moving his/her head.

[Modification 2]
The operation target object operating device according to the embodiment may include a microphone in the vicinity of the operation target object. Sound collected by the microphone is transmitted to the user as sound heard by the surrogate body. The user operates the operation target object while viewing the image displayed on the image display unit and listening to the sound collected by the microphone.
According to this modification, the user can share hearing as well as the viewpoint with the surrogate body.
This allows the user to operate the operation object in a more natural way.

Each modified example has the same effect and functionality as the embodiment.

Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present invention. The new embodiment resulting from the combination has the combined effects of each of the combined embodiments and modifications.

The present invention has a wide range of applications, including remote control of construction machinery, surgical assistance, operation of humanoid robots, operation of vehicles, simulators, and games. The method according to the present invention can be used for such a variety of applications.

DESCRIPTION OF SYMBOLS 1. Image interface device 2. Image operation device 3. Operation object operation device 4. Operation object operation system 10. Image 11. Image display unit 12. User 13. Surrogate body 14. Surrogate body drawing unit 15. Operation signal input unit 20. Operation unit 30. Operation signal output unit 100. Camera 110. Operation object S1. Step of acquiring information on the size of the operation object S2. Step of drawing a surrogate body according to the size of the operation object on the display unit S3. Step of displaying an image of the operation object on the display unit as a subjective image of the surrogate body.

Claims (19)

an image display unit that displays an image of an object to be operated by a user;
a surrogate body drawing unit that draws a virtual surrogate body of the user on the image display unit;
an operation signal input unit to which an operation signal from the user for the operation object is input,
The size of the surrogate body is variable depending on the size of the operation target object;
the image of the operation object displayed on the image display unit is a subjective image seen from the surrogate body,
The surrogate body has a plurality of sizes, and the plurality of sizes are dynamically switchable with respect to the operation object;
An image interface device, characterized in that the surrogate body has a plurality of positions relative to the object to be operated, and the plurality of positions are dynamically switchable. 2. The image interface device according to claim 1 , wherein the position of the representative body is variable according to the position of the object to be operated. the surrogate body moves on the image display unit in accordance with an operation performed on the image of the operation target object;
3. The image interface device according to claim 1, wherein the moving speed of said representative body on the image display unit is variable according to the moving speed of said operation target object. 4. The image interface device according to claim 1 , wherein the position or orientation of the representative body relative to the operation object is variable. The operation object includes a plurality of operation objects having different sizes,
5. The image interface device according to claim 1 , wherein the size of the representative body is continuously variable when the plurality of operation objects are successively switched and operated. The surrogate body includes two different sized surrogate bodies;
An image interface device according to any one of claims 1 to 5, characterized in that the image of the object to be operated displayed on the image display unit is a mixture of image components of the object to be operated as seen from the two differently sized surrogate bodies. 7. The image interface device according to claim 1 , wherein the image of the operation object is an image taken by a camera installed in the vicinity of the operation object. an image display unit that displays an image of an object to be operated by a user;
a surrogate body drawing unit that draws a virtual surrogate body of the user on the image display unit;
an operation signal input unit to which an operation signal from the user for the operation object is input;
an operation unit that generates an operation signal by being operated by the user;
The size of the surrogate body is variable depending on the size of the operation target object;
the image of the operation object displayed on the image display unit is a subjective image seen from the surrogate body,
The surrogate body has a plurality of sizes, and the plurality of sizes are dynamically switchable with respect to the operation object;
An image manipulation device, characterized in that the surrogate body has a plurality of positions relative to the object to be manipulated, and the plurality of positions are dynamically switchable . an image display unit that displays an image of an object to be operated by a user;
a surrogate body drawing unit that draws a virtual surrogate body of the user on the image display unit;
an operation signal input unit to which an operation signal from the user for the operation object is input;
an operation unit that generates an operation signal when operated by the user;
an operation signal output unit that outputs an operation signal of the user to the operation target,
The size of the surrogate body is variable depending on the size of the operation target object;
the image of the operation object displayed on the image display unit is a subjective image seen from the surrogate body,
The surrogate body has a plurality of sizes, and the plurality of sizes are dynamically switchable with respect to the operation object;
An object manipulation device, characterized in that the surrogate body has a plurality of positions relative to the object to be manipulated, and the plurality of positions are dynamically switchable . an image display unit that displays an image of an object to be operated by a user;
a surrogate body drawing unit that draws a virtual surrogate body of the user on the image display unit;
an operation signal input unit to which an operation signal from the user for the operation object is input;
an operation unit that generates an operation signal when operated by the user;
an operation signal output unit that outputs an operation signal of the user to the operation target;
a camera for photographing the operation object,
The size of the surrogate body is variable depending on the size of the operation target object;
the image of the operation object displayed on the image display unit is a subjective image seen from the surrogate body,
The surrogate body has a plurality of sizes, and the plurality of sizes are dynamically switchable with respect to the operation object;
An object manipulation system, characterized in that the surrogate body has a plurality of positions relative to the object to be manipulated, and the plurality of positions are dynamically switchable . The object manipulation system according to claim 10 , wherein the camera includes a movement mechanism for changing a position. The object manipulation system according to claim 11 , wherein the camera includes a rotation mechanism for changing a shooting direction. the moving mechanism and the rotating mechanism move the camera in a direction opposite to a user's operation direction;
13. The object manipulation system according to claim 12 , wherein the image displayed on the image display unit is a left-right inverted version of the image captured by the camera. 14. The object manipulation system according to claim 10 , wherein the camera is a stereo camera having two cameras. 15. The object manipulation system according to claim 14 , wherein an interocular distance of the stereo cameras is variable according to a size of the representative body. The object manipulation system according to claim 10 , wherein the camera is a depth camera that detects the distance to the manipulation object in real time. The object manipulation system according to claim 10 , wherein the camera comprises a plurality of cameras each capturing a different field of view. acquiring information on the size of an object to be operated;
A step of drawing a surrogate body on an image display unit according to a size of the operation target object;
and displaying an image of the operation object on an image display unit as a subjective image of the surrogate body,
The size of the surrogate body is variable according to the size of the operation target object;
the image of the operation object is a subjective image seen from the surrogate body,
The surrogate body has a plurality of sizes, and the plurality of sizes are dynamically switchable with respect to the operation object;
A method for presenting an object to be operated, characterized in that the surrogate body has a plurality of positions relative to the object to be operated, and the plurality of positions are dynamically switchable . acquiring information on the size of an object to be operated;
A step of drawing a surrogate body on an image display unit according to a size of the operation target object;
displaying an image of the operation object on the image display unit as a subjective image of the surrogate body;
The size of the surrogate body is variable according to the size of the operation target object;
the image of the operation object is a subjective image seen from the representative body,
The surrogate body has a plurality of sizes, and the plurality of sizes are dynamically switchable with respect to the operation object;
The surrogate body has a plurality of positions relative to the object to be operated, and the plurality of positions are dynamically switchable.

Images